How the Layer Fails: Throttles, Retries, and Loops

The layer is built, its identity is settled both ways, and it spans accounts. Now the honest part: how it fails. Putting a gateway between every internal call does more than add a per-call price. It changes what failure looks like, and it quietly removes two things the direct invoke gave you for nothing: Lambda’s recursive-loop circuit breaker, and any retry at all. The layer from part 103 is still worth building. The point of this part is that you now own what it took away, and the missing pieces are not the ones most write-ups warn about.

This is the failure-mode inventory: the account throttle bucket your east-west traffic now shares with your public front door, the status codes the gateway hands back to a caller, the loop detector that no longer watches this path, why idempotency stops being optional, and the quotas that end the design. The POC is not deployed, so nothing here is a measurement. Every number below is a documented quota, with a source.

Your internal traffic shares a bucket with your front door

The first change is not a latency change. It is a blast-radius change.

One row on the API Gateway quota page decides the shape of this whole layer. It names the “Throttle quota per account, per Region across HTTP APIs, REST APIs, WebSocket APIs, and WebSocket callback APIs”, and sets it at “10,000 requests per second (RPS) with an additional burst capacity provided by the token bucket algorithm, using a maximum bucket capacity of 5,000 requests” (API Gateway quotas). Read the row header, not the number. The bucket is per account and per Region, and it is shared across every API type you run. The same row closes the door on the burst half: “It is not a quota that a customer can control or request changes to.”

The throttling page says it again in the active voice: “API Gateway examines the rate and a burst of request submissions against all APIs in your account, per Region” (request throttling). The same page tells you not to treat the number as a wall: “Both throttles and quotas are applied on a best-effort basis and should be thought of as targets rather than guaranteed request ceilings.”

Now the arithmetic. Before this layer, one external request drew one token from that bucket, and the internal fan-out drew none, because a direct invoke never touches API Gateway. After this layer, an external request that fans out to N internal hops draws N+1. Picture a checkout: it calls orders, orders calls inventory, inventory calls pricing. That one request now costs four tokens, and your public APIs drink from the same bucket.

That is fine while things work. It is not fine during an incident. A retry storm inside your internal layer throttles your own front door, and the customer-facing 429s arrive from an API that nobody touched. The blast radius of an internal failure is the account and the Region, not the service.

Check your Region before you assume 10,000. A documented list of Regions defaults to 2,500 RPS with a 1,250 burst: Cape Town, Milan, Jakarta, UAE, Hyderabad, Melbourne, Spain, Zurich, Tel Aviv, Calgary, Malaysia, Thailand, and Mexico Central. A wide estate in one of those Regions runs out of headroom four times sooner.

The timeout dial makes it worse, not better. Private REST APIs can raise the integration timeout, and the quota table says so. The row is “Integration timeout for private APIs”, its default reads “50 milliseconds - 29 seconds for all integration types”, and it is marked increasable. The footnote is the part people miss: “You can raise the integration timeout to greater than 29 seconds, but this might require a reduction in your Region-level throttle quota for your account” (API Gateway quotas table). Edge-optimized APIs cannot raise it at all; private ones can, which is precisely the trap.

Read the dial as a price, not a feature. Buying headroom for one slow chained call costs RPS ceiling for every API in the account, public ones included. If an east-west call genuinely needs more than 29 seconds, it is not a request/response call. It is a workflow, and the last section is about where workflows belong.

Nothing retries, so the caller must

Part 101 complained that direct invoke has confusing retry semantics, driven by a flag the caller picked rather than a contract the callee published. It does not follow that the gateway fixes this. The gateway removes retries entirely.

The Lambda developer guide is unambiguous: “API Gateway does not retry any Lambda invocations. If Lambda returns an error, API Gateway returns an error response to the client” (API Gateway errors). The retries page states it from the other side: “API Gateway always relays the error response back to the requester”, in contrast with an asynchronous invoke, where Lambda “retries function errors twice” on its own (invocation retries).

So retry policy is entirely the caller’s problem now, and the caller has to decide from a status code. That map is worth learning, because it does not say what you would expect: “API Gateway treats all invocation and function errors as internal errors. If the Lambda API rejects the invocation request, API Gateway returns a 500 error code. If the function runs but returns an error, or returns a response in the wrong format, API Gateway returns a 502. In both cases, the body of the response from API Gateway is {“message”: “Internal server error”}.”

Status	What it means	What the caller should do
504	INTEGRATION_TIMEOUT or INTEGRATION_FAILURE. The gateway stopped waiting.	Retry with backoff, but the callee may have committed. Safe only with an idempotency key.
502	The function ran and returned an error, or returned a malformed proxy response.	A malformed response will repeat. Retry once at most, then fail.
500	The Lambda API rejected the invocation. Body is {"message": "Internal server error"}.	Opaque. It may be a throttled callee. Back off as if it were.
429	THROTTLED or QUOTA_EXCEEDED, applied “when usage plan-, method-, stage-, or account-level throttling limits exceeded”.	Back off. This is the only throttle signal you can actually see.
403	INVALID_SIGNATURE or EXPIRED_TOKEN.	Your signature, not their fault. Re-sign; never blind-retry.
413	REQUEST_TOO_LARGE.	Never retry. Fix the payload.

The status-to-response mapping above comes from the gateway response types reference.

Now join two documented facts. A callee whose reserved concurrency is exhausted is throttled by Lambda, and the Lambda API answers the invoke with a TooManyRequestsException carrying a retryAfterSeconds field, documented as “The number of seconds the caller should wait before retrying” (Invoke). API Gateway’s rule is that the Lambda API rejecting an invocation request becomes a 500. Which means a throttled callee is indistinguishable from a broken one, and the retryAfterSeconds hint that would let the caller back off intelligently never reaches it. Both halves are AWS’s; joining them is this post’s reading, not a sentence AWS writes. Design for it regardless, because the operational consequence is concrete: the one throttle signal the caller can see is the gateway’s own 429, not the callee’s.

Signatures expire, and that failure hides inside your retry loop. AWS states the window plainly: “In most cases, a request must reach AWS within five minutes of the time stamp in the request. Otherwise, AWS denies the request” (signing AWS requests). An expired signature surfaces as INVALID_SIGNATURE, which is a 403 and not a 5xx. Therefore a retry wrapper that only retries 5xx drops these on the floor. Worse, a wrapper that holds a pre-signed request and re-sends it after a long backoff gets 403 forever, and every log line points at IAM. The rule is one line: re-sign on every attempt, and never retry a stale signature.

The loop breaker you gave away

This is the regression, and it is the one to take to your architecture review.

Lambda ships a recursive-loop detector. It is on by default, it costs nothing, and it works from X-Ray tracing headers plus SDK metadata. The trip wire is documented: “If your function is invoked approximately 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you” (recursive loop detection).

Then read what it covers. AWS scopes it explicitly: “Lambda can detect only recursive loops that include certain supported AWS services.” The list follows: “Lambda currently detects recursive loops between your functions, Amazon SQS, Amazon S3, and Amazon SNS. Lambda also detects loops comprised only of Lambda functions, which may invoke each other synchronously or asynchronously.” And the exclusion is spelled out for a service that is not on the list: “When another AWS service such as Amazon DynamoDB forms part of the loop, Lambda can’t currently detect and stop it.”

API Gateway is not on that list, and the list reads as exhaustive rather than illustrative. A call from A to the gateway to B, back through the gateway to A, is a loop with an unsupported service in it, travelling as SigV4-signed HTTPS rather than as an SDK write to a supported service. It is not detected and it is not stopped.

Notice exactly what changed. Loops made only of Lambda functions are what direct invoke produces, and those were covered. You had a free 16-invocation circuit breaker on that path. You traded it away the moment you put the gateway in the middle. The gateway bought you a contract boundary and charged you a safety net.

The replacement is hand-rolled, and it belongs in the shared signing client from part 103 rather than in each service. Propagate a hop count, and refuse above a fixed budget.

const MAX_HOPS = 8;

// Callee side: every internal route runs this before it does any work.
// Header lookups assume the handler normalised event.headers to lower case.
export function assertHopBudget(headers: Record<string, string | undefined>): number {
  const hops = Number(headers['x-hop-count']);
  if (!Number.isInteger(hops) || hops < 1) {
    throw new Error('missing or invalid x-hop-count');
  }
  if (hops >= MAX_HOPS) {
    // The circuit breaker Lambda used to provide on the direct-invoke path.
    throw new Error(`hop budget exhausted at ${hops}`);
  }
  return hops;
}

// Caller side: the shared SigV4 client, plus two propagated headers.
// An entry-point function has no inbound hop count, so the chain starts at 0.
export async function callService(
  url: URL,
  body: string,
  inbound: Record<string, string | undefined>,
) {
  const hops = Number(inbound['x-hop-count'] ?? '0');

  const request = new HttpRequest({
    protocol: url.protocol,
    hostname: url.hostname,
    path: url.pathname,
    method: 'POST',
    headers: {
      host: url.hostname,
      'content-type': 'application/json',
      'x-hop-count': String(hops + 1),
      'x-correlation-id': inbound['x-correlation-id'] ?? crypto.randomUUID(),
    },
    body,
  });

  // Re-sign on every attempt. A signature is only good for five minutes.
  const signed = await signer.sign(request);
  return fetch(url, { method: signed.method, headers: signed.headers, body: signed.body });
}

Be honest about what this is. It is a cooperative control, not a security control. It works because every caller in the estate uses the same client, and it stops working the day one team writes its own. Enforcement therefore has to sit on the receiving side: a route that sees no hop count at all should refuse rather than assume zero, which is why assertHopBudget rejects a missing header instead of defaulting it. Carry the correlation id in the same place, because the first thing you will want when a loop does escape is the ability to pull the whole chain out of the logs with one query.

Idempotency is no longer optional

The logic chain here is short and it has no escape hatch.

API Gateway never retries. Therefore the caller must retry, or the call simply fails. But a 504 is ambiguous: it tells you the gateway stopped waiting, not that the callee stopped working. The callee may have committed the write and merely answered past 29 seconds. Therefore every non-GET east-west route needs an idempotency key. There is no version of this layer in which that step is optional.

AWS says the same thing in one sentence, immediately after telling you to retry: “If you retry, ensure that your function’s code can handle the same event multiple times without causing duplicate transactions or other unwanted side effects” (invocation retries).

The key comes from the caller and it has to be stable across attempts. A key derived from the payload plus a timestamp is not a key, because it changes on every retry. Generate it once when the caller forms its intent, send it as x-idempotency-key, and reuse it for every retry of that intent. The pattern is not exotic: it is what AWS’s own APIs do with client tokens on EC2, ECS, and DynamoDB TransactWriteItems.

On the callee side, do not write the store yourself. Powertools for AWS Lambda (TypeScript) ships an Idempotency utility whose first listed feature is to “Prevent Lambda handler from executing more than once on the same event payload during a time window”, persisted to DynamoDB, Valkey, or Redis (Powertools Idempotency). It is exposed as a decorator, as Middy middleware, or as a makeIdempotent wrapper, and the Lambda developer guide points at it by name.

import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';

const persistenceStore = new DynamoDBPersistenceLayer({ tableName: 'orders-idempotency' });

// Key on the caller's header, not on the payload. The same business intent
// must produce the same key even if a timestamp field moved.
const config = new IdempotencyConfig({
  eventKeyJmesPath: 'headers."x-idempotency-key"',
  throwOnNoIdempotencyKey: true,
});

const reserve = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  // The commit. Runs at most once per key inside the expiry window.
  return { statusCode: 200, body: JSON.stringify({ reserved: true }) };
};

export const handler = makeIdempotent(reserve, { persistenceStore, config });

The trade-off is real and worth naming. Idempotency puts writes on the hot path of every mutating call: a conditional put before the work, an update after it. That is added latency, plus a table to own. It also brings a failure mode of its own, because a request that dies between the reservation and the completion leaves a record in progress, and the next attempt has to decide whether to wait or to take over. Powertools resolves that with an expiry window, which is one more timeout you now have to tune. Pay the cost anyway. The alternative is a duplicate order every time a callee answers slowly, and you cannot even count how often it happens, because a 504 does not tell you whether the write landed.

The walls you will hit

Concurrency compounds the throttle problem. The Regional default is 1,000 concurrent executions per account, shared by every function, and each function can add 1,000 execution environments every 10 seconds (Lambda quotas). Now look at a synchronous chain. The caller holds an execution environment, idle, for the whole duration of the callee. AWS states the consequence on the same anti-pattern page that started this series: “The concurrency of all three functions must be equal. In a busy system, this uses more concurrency than would otherwise be needed” (event-driven architectures). A two-hop chain therefore triples concurrency consumption per user request, out of one shared pool. It is the same multiplier as the throttle bucket, applied to a different quota.

Reserved concurrency on the callee protects the account pool, and it converts backpressure into the opaque 500 from the table above. You are choosing which failure you prefer, not avoiding one.

The IAM wall arrives at deploy time, which is why nobody sees it coming. The aggregate size of all inline policies on one role cannot exceed 10,240 characters, and IAM lists that limit among the ones you cannot request an increase for. One concession: “IAM doesn’t count white space when calculating the size of a policy against these limits” (IAM quotas). CDK’s grant helpers write inline role policy by default.

Now the arithmetic, which is this post’s rather than AWS’s. One execute-api:Invoke statement naming a fully-qualified route ARN runs roughly 150 to 200 characters once minified. That puts a caller role somewhere around 50 to 70 per-route grants before it stops deploying. The failure surfaces as a LimitExceeded error at cdk deploy, on the day someone adds one route too many. Two escapes exist, and both cost something. Move the grants into customer managed policies, capped at 6,144 characters each, 10 attachable per role by default and 25 at the raised quota. Or coarsen the grant to a path prefix, which trades away the per-route precision that was the reason to build the layer.

Wall	Limit	Increasable
Inline policy size per IAM role, aggregate	10,240 characters	No
Customer managed policy size	6,144 characters each	No
Managed policies attached per role	10 by default	Yes, to 25
Resources per REST API	300	Yes
API Gateway resource policy length	8,192 characters	Yes
Private APIs per account per Region	600	No
VPC endpoint policy size	20,480 characters	No
Concurrent executions per account per Region	1,000	Yes

Two of those walls push against each other. AWS’s own escape from the 300-resource limit is to “Use {proxy+} paths to reduce the number of resources”. That works, and in the same motion it collapses your ability to grant per route, because a {proxy+} route is one ARN. The 8,192-character resource policy squeezes from the other direction, since in a multi-team estate every allowed principal and endpoint is written into that one document.

One piece of good news, because the endpoint is not the wall people expect. AWS documents that “each VPC endpoint can support a bandwidth of up to 10 Gbps per Availability Zone, and automatically scales up to 100 Gbps” (PrivateLink quotas). The shared interface endpoint is not a throughput bottleneck for an internal service layer. The same page carries two details worth reading together, because taken alone the first one sounds alarming. A VPC endpoint supports an MTU of 8,500 bytes and does not support path MTU discovery. That is the kind of line that starts a hunt for silent hangs on large payloads. The next line calls the hunt off: “VPC endpoints enforce Maximum Segment Size (MSS) clamping for all packets.” Ordinary TCP, which is all the HTTPS traffic in this design, gets clamped rather than dropped. Read the pair, not the first half.

When you do not need this layer at all

Everything above is the cost of putting a gateway on a request/response call. The honest question to ask before paying it is whether the call is request/response at all.

AWS’s anti-pattern page, the one that told you not to have Lambda functions call each other, carves out an exception to its own rule: “this doesn’t apply to durable functions, which are specifically designed to orchestrate multi-step workflows by invoking other functions”. That carve-out deserves to be taken seriously rather than skipped.

Durable functions are AWS’s answer to the multi-step chain. In AWS’s words, “With AWS Lambda durable functions, you can build resilient multi-step applications and AI workflows that can execute for up to one year while maintaining reliable progress despite interruptions”, and the mechanism “uses checkpoints to track progress and automatically recover from failures through replay”. The programming model splits work into steps and waits: “Steps execute business logic with built-in retries and progress tracking, while waits suspend execution without incurring compute charges” (durable functions).

Read that against the list of things this post just told you to rebuild by hand. Built-in retries. Progress tracking. Recovery through replay. If your east-west call is really a step in a workflow that one team owns, durable functions hand you all of it, and the gateway hands you none of it.

Bound it honestly, though, because it is not an east-west answer. The chaining primitive stops at the account line, and AWS states it directly: “Cross-account chained invocations are not supported. The invoked function must be in the same AWS account as the calling function” (invoking durable functions). One account, one team, workflow-shaped work. That is the domain. A multi-team estate spread across accounts is exactly the shape it does not cover, and that estate is what this series is built for.

So treat it as a filter rather than a competition, and apply it before you build anything. If nobody is waiting on the reply, use an event bus. If the caller is orchestrating steps it owns inside one account, use durable functions or Step Functions. What is left, a genuine request/response call across a team boundary, is what this layer is for. It is a smaller set than most teams assume, and every call you remove from it is a call that does not draw from the throttle bucket.

Common pitfalls

1. Reading a 500 as a bug in the callee. It may be a throttled callee instead. The Lambda API rejecting an invocation becomes a 500 with the body {"message": "Internal server error"}, and the retryAfterSeconds hint does not survive the trip. Back off on 500s as if they were 429s.
2. Retrying only on 5xx, and retrying a signature you already built. An expired signature is a 403, so a 5xx-only policy drops it silently and sends you to debug IAM. A pre-signed request replayed after a long backoff fails forever. Re-sign inside the retry, not outside it.
3. Retrying a non-idempotent POST on a 504. The callee may have committed and simply answered late. Without an idempotency key sent by the caller, that is how one 504 becomes two orders.
4. Assuming Lambda still stops your loops. It stops loops through Lambda, SQS, SNS, and S3. A loop through API Gateway is not on the supported list. Without a propagated hop count you have no breaker at all.
5. Raising the integration timeout to make a slow call fit. It may reduce the Region-level throttle quota for the whole account, public APIs included. A call that needs more than 29 seconds is a workflow wearing a request’s clothes.
6. Granting per route with inline policies and never counting the characters. The 10,240-character inline limit cannot be raised, and the bill arrives at cdk deploy rather than at runtime.

The layer is worth building, and part 103 stands. What changes is the work you own once it lands: a retry policy in the caller, an idempotency key on every mutating route, a hop count in place of the loop detector Lambda no longer provides, and a running count of how much of the account throttle bucket your east-west traffic now draws. None of that is exotic, and all of it is invisible until the first incident, which is the argument for writing it into the shared client before the second service ships. The POC is not deployed, so read the failure modes above as documented behaviour to design against rather than behaviour observed.

You now know how the layer fails. The last question is what you can see while it runs, and what you must not log. Part 107 is the observability half: the access log that already names the calling service and the route, and the user token the identity design forced into a header that you have to keep out of your log groups.

References

• Amazon API Gateway quotas - The 10,000 RPS throttle quota “per account, per Region across HTTP APIs, REST APIs, WebSocket APIs, and WebSocket callback APIs”, plus the Regions that default to 2,500 RPS.
• Throttle requests to your REST APIs - “API Gateway examines the rate and a burst of request submissions against all APIs in your account, per Region”, and the best-effort caveat on every throttle.
• API Gateway quotas table - The 29-second integration timeout, and the footnote that raising it “might require a reduction in your Region-level throttle quota for your account”.
• Error handling with API Gateway and Lambda - “API Gateway does not retry any Lambda invocations”, and the 500 versus 502 mapping that makes a throttled callee look broken.
• Gateway response types - The full status-code map: INTEGRATION_TIMEOUT to 504, THROTTLED to 429, INVALID_SIGNATURE to 403, REQUEST_TOO_LARGE to 413.
• Lambda Invoke API reference - The TooManyRequestsException retryAfterSeconds field, documented as “The number of seconds the caller should wait before retrying”, which API Gateway collapses into an opaque 500.
• Error handling and automatic retries in Lambda - “API Gateway always relays the error response back to the requester”, against asynchronous invokes where Lambda “retries function errors twice”, plus the idempotency instruction.
• Recursive loop detection - The supported-service list (Lambda, SQS, S3, SNS) and the 16-invocation trip wire. API Gateway is not on the list.
• Signing AWS API requests - The five-minute SigV4 window: “a request must reach AWS within five minutes of the time stamp in the request”.
• AWS Lambda quotas - 1,000 concurrent executions per account per Region, shared by every function, and the per-function scaling limit.
• Lambda event-driven architectures - The anti-pattern page on synchronous chains: “The concurrency of all three functions must be equal. In a busy system, this uses more concurrency than would otherwise be needed.”
• IAM and STS quotas - The 10,240-character aggregate inline policy limit per role, which cannot be increased, and the 6,144-character cap on customer managed policies.
• AWS PrivateLink quotas - “each VPC endpoint can support a bandwidth of up to 10 Gbps per Availability Zone, and automatically scales up to 100 Gbps”, plus the 8,500-byte MTU and the lack of path MTU discovery.
• AWS Lambda durable functions - Checkpoints, replay, and built-in step retries: everything this post just told you to rebuild by hand, for workflows that live inside one account.
• Invoking durable functions - The boundary that matters here, under Chained invocations: “Cross-account chained invocations are not supported. The invoked function must be in the same AWS account as the calling function.”
• Powertools for AWS Lambda: Idempotency - The utility’s first listed feature: “Prevent Lambda handler from executing more than once on the same event payload during a time window”, persisted to DynamoDB, Valkey, or Redis.