Building the Layer with a Private API Gateway

You have chosen the layer (part 101) and settled what it carries (part 102). This part is the build: the private REST API, the resource policy that makes it reachable, AWS_IAM grants scoped per route, the two CDK stacks split on the ownership boundary, the SigV4 call from a Node 22 Lambda, and the operational facts about VPC-attached callers that decide whether any of it works. The substitution at its heart stays small: the caller’s execution role holds execute-api:Invoke on a route ARN instead of lambda:InvokeFunction on a function ARN, and the callee’s function ARN never leaves the callee’s stack. The private endpoint constraint comes first, because it settles the API type before anything else.

Private endpoints are REST-only

The API Gateway developer guide is direct about this in Considerations for private APIs: “Only REST APIs are supported” (Private REST APIs).

That single line kills the most common cost objection. HTTP APIs are cheaper per request, and community threads reach for them constantly, but there is no private endpoint type for an HTTP API. What HTTP APIs have is VPC Link V2, and it runs in the opposite direction. It lets a public API Gateway reach into your VPC to hit an ALB, an NLB, or a Cloud Map service (VPC links V2). That is a private integration, not a private endpoint. If the API itself must be unreachable from the internet, REST is the only flavour that does it, and you pay the REST rate.

Three other considerations from the same page belong in your design notes: private APIs enforce a TLS 1.2 minimum, HTTP/2 requests are enforced to HTTP/1.1, and you cannot restrict a private API to IPv4 only, since only dualstack is supported. One more comes from the endpoint types page: private API endpoints pass all header names through as-is (API endpoint types). Edge-optimized endpoints capitalize them, so a service that quietly depended on that capitalization can break when it moves behind a private API.

The resource policy is the on switch

A freshly created private API is not reachable. AWS says so in step 3 of Create a private API: “Your current private API is inaccessible to all VPCs” (Create a private API). Until a resource policy names your VPC or endpoint, every call fails. The failure looks enough like a DNS problem that people go debugging Route 53 for an hour.

The canonical shape is allow-then-deny, keyed on aws:SourceVpce (resource policy examples):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"]
    },
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"],
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "vpce-1a2b3c4d" }
      }
    }
  ]
}

The execute-api:/* shorthand expands to a full ARN when API Gateway saves the policy. The route-level form is execute-api:/{stage}/{METHOD}/{path}, which is how you scope a policy statement to one route rather than the whole API.

Per-route grants with AWS_IAM

With AWS_IAM authorization on a method, the caller signs the request with SigV4 (service name execute-api), and API Gateway evaluates the caller’s identity policy. The grant looks like this:

execute-api:Invoke  on  arn:aws:execute-api:{region}:{account}:{apiId}/{stage}/POST/orders

That is the whole authorization story, and it needs no new policy language. The callee can rewrite, rename, or replace the backing Lambda and not one consumer policy changes, because no consumer policy ever mentioned the function.

There is a trap here worth stating plainly, because the error message points somewhere else. If the resource policy names an AWS principal such as a role ARN, and any method is still on NONE authorization, API Gateway returns User: anonymous is not authorized to perform: execute-api:Invoke. The troubleshooting section of the Create a private API page states the fix: every method must use AWS_IAM authorization. Set it once as a default in CDK rather than per method, so a new route cannot be added without it.

The CDK stacks

Two stacks, because the ownership boundary is the point. The platform stack owns the VPC and one shared interface endpoint. AWS’s own best practice is to use a single VPC endpoint for multiple private APIs. That is what amortises the fixed cost across an entire estate.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export class PlatformNetworkStack extends cdk.Stack {
  public readonly vpc: ec2.Vpc;
  public readonly apiEndpoint: ec2.InterfaceVpcEndpoint;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Isolated subnets only: east-west traffic never needs a NAT gateway.
    // Calls to other AWS APIs (DynamoDB, SQS) need their own endpoints.
    this.vpc = new ec2.Vpc(this, 'Vpc', {
      maxAzs: 2,
      natGateways: 0,
      subnetConfiguration: [
        { name: 'isolated', subnetType: ec2.SubnetType.PRIVATE_ISOLATED, cidrMask: 24 },
      ],
    });

    // The endpoint ENI must accept 443 from the caller Lambdas.
    // Omit securityGroups and CDK defaults to open: true, which opens
    // 443 to the whole VPC CIDR. This narrows it to the caller SG.
    const endpointSg = new ec2.SecurityGroup(this, 'ApiEndpointSg', {
      vpc: this.vpc,
      description: 'execute-api interface endpoint',
      allowAllOutbound: false,
    });
    endpointSg.addIngressRule(
      ec2.Peer.ipv4(this.vpc.vpcCidrBlock),
      ec2.Port.tcp(443),
      'HTTPS from in-VPC callers',
    );

    // One endpoint, many private APIs. This is the fixed cost of the layer.
    this.apiEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ExecuteApiEndpoint', {
      vpc: this.vpc,
      service: ec2.InterfaceVpcEndpointAwsService.APIGATEWAY,
      privateDnsEnabled: true,
      securityGroups: [endpointSg],
      subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
    });
  }
}

The service stack owns the API and the handler. This is where the single most useful CDK detail lives: endpointConfiguration.vpcEndpoints does not write the resource policy. The CDK reference describes that prop as “A list of VPC Endpoints against which to create Route53 ALIASes” (EndpointConfiguration). The policy comes from either the policy prop or from grantInvokeFromVpcEndpointsOnly(), documented as adding “a resource policy that only allows API execution from a VPC Endpoint” (RestApi). Two props, two jobs. Setting only the first is why an API that looks correctly wired returns 403 on every call.

What vpcEndpoints does do is narrower. It sets vpcEndpointIds on the underlying resource, which associates the endpoint with the API, and API Gateway creates the alias from there. Pin aws-cdk-lib at 2.179.0 or later, which is where grantInvokeFromVpcEndpointsOnly() first appears. On an older version you get a bare TypeScript error with no hint of the cause.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as nodejs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as logs from 'aws-cdk-lib/aws-logs';

interface OrdersServiceStackProps extends cdk.StackProps {
  readonly apiEndpoint: ec2.IInterfaceVpcEndpoint;
}

export class OrdersServiceStack extends cdk.Stack {
  public readonly api: apigateway.RestApi;

  constructor(scope: Construct, id: string, props: OrdersServiceStackProps) {
    super(scope, id, props);

    const handler = new nodejs.NodejsFunction(this, 'OrdersHandler', {
      entry: 'src/orders/handler.ts',
      runtime: lambda.Runtime.NODEJS_22_X,
      timeout: cdk.Duration.seconds(10),
    });

    this.api = new apigateway.RestApi(this, 'OrdersApi', {
      restApiName: 'orders',
      endpointConfiguration: {
        types: [apigateway.EndpointType.PRIVATE],
        // Route53 ALIAS records only. This does NOT create the resource policy.
        vpcEndpoints: [props.apiEndpoint],
      },
      // Applied to every method, so a new route cannot ship without auth.
      defaultMethodOptions: {
        authorizationType: apigateway.AuthorizationType.IAM,
      },
      deployOptions: {
        stageName: 'v1',
        tracingEnabled: true,
        metricsEnabled: true,
        // The throttling contract direct invoke cannot express.
        throttlingRateLimit: 200,
        throttlingBurstLimit: 400,
        accessLogDestination: new apigateway.LogGroupLogDestination(
          new logs.LogGroup(this, 'OrdersAccessLogs', {
            retention: logs.RetentionDays.ONE_MONTH,
          }),
        ),
        accessLogFormat: apigateway.AccessLogFormat.jsonWithStandardFields(),
      },
    });

    // This is what makes the API reachable. Without it, every VPC is denied.
    this.api.grantInvokeFromVpcEndpointsOnly([props.apiEndpoint]);

    this.api.root
      .addResource('orders')
      .addMethod('POST', new apigateway.LambdaIntegration(handler));
  }
}

The caller’s stack grants on the route, and arnForExecuteApi builds the ARN for you:

import * as iam from 'aws-cdk-lib/aws-iam';

const checkout = new nodejs.NodejsFunction(this, 'CheckoutHandler', {
  entry: 'src/checkout/handler.ts',
  runtime: lambda.Runtime.NODEJS_22_X,
  vpc: props.vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
  environment: {
    ORDERS_URL: props.ordersApi.urlForPath('/orders'),
  },
});

// A grant on a route, not on a function ARN.
checkout.addToRolePolicy(new iam.PolicyStatement({
  actions: ['execute-api:Invoke'],
  resources: [props.ordersApi.arnForExecuteApi('POST', '/orders', 'v1')],
}));

One deployment detail to plan around: associating or dissociating a VPC endpoint with a private API requires redeploying the API, and DNS propagation takes minutes. Order the platform stack before the service stacks.

Signing the call from a Node 22 Lambda

The caller signs with SigV4 against the service name execute-api. Not lambda, not apigateway. Wrong service name is a 403 with an unhelpful message.

import { SignatureV4 } from '@smithy/signature-v4';
import { HttpRequest } from '@smithy/protocol-http';
import { Sha256 } from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';

const ordersUrl = new URL(process.env.ORDERS_URL!);

// Built once per execution environment, reused across invocations.
const signer = new SignatureV4({
  service: 'execute-api',
  region: process.env.AWS_REGION!,
  credentials: defaultProvider(),
  sha256: Sha256,
});

export const handler = async (event: { cartId: string }) => {
  const body = JSON.stringify({ cartId: event.cartId });

  const request = new HttpRequest({
    protocol: ordersUrl.protocol,
    hostname: ordersUrl.hostname,
    path: ordersUrl.pathname,
    method: 'POST',
    headers: {
      host: ordersUrl.hostname, // required in the signed headers
      'content-type': 'application/json',
    },
    body,
  });

  const signed = await signer.sign(request);

  // Global fetch; no HTTP client dependency needed on Node 22.
  const response = await fetch(ordersUrl, {
    method: signed.method,
    headers: signed.headers,
    body: signed.body,
  });

  if (!response.ok) {
    throw new Error(`orders returned ${response.status}: ${await response.text()}`);
  }

  return response.json();
};

The body you sign and the body you send must be byte-identical. Serialising twice, or letting a client library re-encode the payload, produces a payload hash mismatch and another 403.

Callers must be VPC-attached

The caller resolves and reaches the interface endpoint from inside the VPC, so it has to be attached to one. Do not file this under the cost of choosing API Gateway. A Lambda caller must reach VPC Lattice from a VPC too: either one associated with the service network, or one holding a service-network VPC endpoint. Neither endpoint is a wall around the VPC. Traffic can arrive from outside it over peering, Transit Gateway, Direct Connect, or VPN, and a private API Gateway takes on-premises callers over Direct Connect the same way. The rule that survives is narrower than it looks: a Lambda caller is VPC-attached whichever transport you pick. That is a property of doing private east-west on AWS, not a tax this design charges.

The reflex objection is that VPC Lambdas cold-start slowly. That has not been true since the Hyperplane rework. The current documentation states that the ENI is created when the function is created or its VPC settings are updated, not at invoke time. Each Hyperplane ENI supports up to 65,000 connections, shared across functions that use the same subnet and security group combination (Lambda VPC networking). The 2019 announcement recorded a cold start dropping from 14.8 seconds to 933 milliseconds in an X-Ray trace (improved VPC networking). That is historical context for why the myth died, not a benchmark to plan against.

Three operational consequences follow. A newly created VPC function stays in Pending until its ENI is ready, which can take minutes, so budget for it in a cold pipeline. Reusing one subnet plus security group combination across your functions lets Lambda share ENIs instead of minting new ones.

The third one bites a sparse internal service layer specifically. If a VPC function sits idle for 14 days, Lambda reclaims the unused Hyperplane ENI and moves the function to Inactive. The next invocation fails and the function re-enters Pending. A rarely-called internal endpoint is exactly that traffic shape, so treat the first call after a quiet fortnight as a failure the caller has to retry.

Common pitfalls

1. Reaching for an HTTP API because it is cheaper. There is no private endpoint type for HTTP APIs. Use a REST API with EndpointType.PRIVATE and budget for $3.50 per million.
2. Setting vpcEndpoints in endpointConfiguration and assuming the resource policy came with it. It only creates Route 53 aliases. Call grantInvokeFromVpcEndpointsOnly() or set policy.
3. Naming an IAM role as Principal in the resource policy while a method still uses NONE authorization. The result is User: anonymous is not authorized to perform: execute-api:Invoke. Put AWS_IAM in defaultMethodOptions.
4. Leaving the interface endpoint on the default security group. Its ENI needs an explicit rule accepting 443 from the callers.
5. Signing with the wrong service name. It is execute-api.
6. Building one VPC endpoint per API. AWS’s best practice is one endpoint, many private APIs; doing it per API multiplies the fixed cost by N.
7. Quoting the 256 KB async payload limit. The current quota is 1 MB async and 6 MB each way sync.
8. Assuming VPC-attached Lambdas cold-start slowly. Cite the current ENI docs, not 2018 blog posts.

The layer is running, and it proves which service is calling. Which user it is calling for is a separate question. Part 104 answers it, and states what the transport actually encrypts.

References

• Private REST APIs in API Gateway - The load-bearing page: “Only REST APIs are supported”, one endpoint for many APIs, and the TLS 1.2 and HTTP/1.1 considerations.
• Set up VPC links V2 in API Gateway - Shows that the HTTP API private story is a private integration, gateway into VPC, not a private endpoint.
• API endpoint types - Private API endpoints pass all header names through as-is, where edge-optimized endpoints capitalize them.
• Create a private API - “Your current private API is inaccessible to all VPCs”, and the anonymous-principal troubleshooting entry that forces AWS_IAM on every method.
• API Gateway resource policy examples - The canonical allow-then-deny aws:SourceVpce policy and the route-level shorthand.
• CDK: interface EndpointConfiguration - vpcEndpoints is “A list of VPC Endpoints against which to create Route53 ALIASes”; it does not create the resource policy.
• CDK: class RestApi - grantInvokeFromVpcEndpointsOnly() and arnForExecuteApi(), the two methods that carry this pattern.
• Lambda VPC networking - Hyperplane ENI created at function create or update, 65,000 connections per ENI, and the 14-day idle reclaim that bites a sparse internal service.
• Improved VPC networking for AWS Lambda - The 2019 announcement that recorded a VPC cold start dropping from 14.8 seconds to 933 milliseconds. Historical context for why the slow-cold-start myth died.