Observability, and Where Service Meshes Are Heading

A service mesh sells three things: identity, resilience, and observability. This series composed the first two out of primitives that were already in the account: identity in part 104, resilience in part 106. Observability is the third, and it is the one people assume arrives for free. Half of it does. Without a line of application code, every call writes one JSON line naming the calling service, the route, the status, and the latency. No sidecar anywhere.

The other half is billed. Per-route metrics are an opt-in with a price on it. And the pipeline that shows you all of this is the pipeline that can store the user token part 104 forced into a custom header. Both halves are here: the telemetry, and the control that stops it becoming a leak. This part is also where the series ends, so its last section steps back to the industry question the whole thing sits inside. The POC behind this series is not deployed, so what follows is the configuration I plan to run, not a set of measurements.

The access log is the telemetry

Access logs are field-selected, and that property is the whole safety story. AWS describes the mechanism plainly: “To specify the access details, you select $context variables, a log format, and a log group destination” (set up logging). The $context list is a closed set of names. There is no $context.request.header.* variable for REST access logs, so an arbitrary custom header cannot land in an access log unless you deliberately name a field that carries it. One narrow exception exists and it is worth knowing: $context.requestOverride.header.header_name logs an integration-request header override. Do not name the user assertion in an override.

The fields that turn this into mesh telemetry are the identity ones, and AWS_IAM fills them in for you (access-log variables). $context.identity.caller is “The principal identifier of the caller that signed the request. Supported for resources that use IAM authorization.” $context.identity.userArn is “The Amazon Resource Name (ARN) of the effective user identified after authentication.” $context.identity.principalOrgId is “The AWS organization ID.” And $context.identity.vpceId is “The VPC endpoint ID of the VPC endpoint making the request… Present only when you have a private API.” Pair those with $context.resourcePath, $context.httpMethod, $context.status, $context.responseLatency, and $context.integrationLatency. One field is mandatory: “The access log format must include at least $context.requestId or $context.extendedRequestId.”

Part 103 shipped AccessLogFormat.jsonWithStandardFields() on the stage, which is a reasonable start and stops short of the interesting part. The standard fields carry requestId, caller, resourcePath, and status; they leave out userArn, principalOrgId, vpceId, and both latency fields. Name them explicitly instead:

import * as cdk from 'aws-cdk-lib';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as logs from 'aws-cdk-lib/aws-logs';

const accessLogs = new logs.LogGroup(this, 'OrdersAccessLogs', {
  retention: logs.RetentionDays.ONE_MONTH,
});

// Field-selected by construction. Only the $context variables named here are
// written, and there is no $context.request.header.* variable for a REST access
// log, so the user assertion cannot arrive in this line by accident.
const accessLogFormat = apigateway.AccessLogFormat.custom(
  JSON.stringify({
    requestId: '$context.requestId', // required: this one or extendedRequestId
    resourcePath: '$context.resourcePath',
    httpMethod: '$context.httpMethod',
    status: '$context.status',
    responseLatency: '$context.responseLatency',
    integrationLatency: '$context.integrationLatency',
    // Who called. AWS_IAM fills these in; nothing in the handler has to.
    callerPrincipal: '$context.identity.caller',
    callerArn: '$context.identity.userArn',
    callerOrgId: '$context.identity.principalOrgId',
    vpceId: '$context.identity.vpceId', // private APIs only
  }),
);

this.api = new apigateway.RestApi(this, 'OrdersApi', {
  restApiName: 'orders',
  // endpointConfiguration and defaultMethodOptions as in part 103.
  deployOptions: {
    stageName: 'v1',
    accessLogDestination: new apigateway.LogGroupLogDestination(accessLogs),
    accessLogFormat,
  },
});

Read what that produces. One JSON line per call, telling you which IAM principal called which route, through which VPC endpoint, in which organization, with what status and what latency. That is the per-call observability a mesh sells, and you just wrote it in a stage config. No sidecar, no collector, no agent. And no user token in the line, because the format is a list of names you chose.

Per-route metrics are not free

Metrics are where the free story stops, and it is the correction most write-ups skip. API Gateway publishes to the AWS/ApiGateway namespace: Count, Latency (“The time between when API Gateway receives a request from a client and when it returns a response to the client”), IntegrationLatency (“The time between when API Gateway relays a request to the backend and when it receives a response from the backend”), 4XXError, and 5XXError (metrics and dimensions). The difference between the first two latencies is the gateway overhead per call, which is the number part 101 says to watch.

The catch is in the dimensions. You get ApiName and ApiName, Stage by default. The one you actually want, ApiName, Method, Resource, Stage, carries two sentences in the documentation: “API Gateway will not send these metrics unless you have explicitly enabled detailed CloudWatch metrics.” and “Enabling these metrics will incur additional charges to your account.” So per-route metrics are an opt-in with a bill attached. In CloudFormation the switch is MethodSettings[].MetricsEnabled on AWS::ApiGateway::Stage; in CDK it is metricsEnabled in deployOptions, which part 103 already set and which applies to every method on the stage.

The cost driver is cardinality: method times resource times stage. One service with twelve routes across two stages is fine. Thirty services with twelve routes each is a different conversation, and it arrives as a custom-metrics line on the bill rather than as an API Gateway line, which is why nobody sees it coming. The cheaper path is already in your hands. You are writing $context.resourcePath and $context.responseLatency into the access log on every call, so per-route latency and per-route error rate are a Logs Insights query away, at log-ingestion prices you are paying anyway. The trade-off is real and worth stating: a metric is alarmable in seconds, a log query is not. Turn on detailed metrics for the routes you page someone about, and read the rest out of the log.

Tracing and the service map

The other mesh-grade feature is the call graph, and it is a stage setting. AWS is explicit that private APIs are not excluded: “API Gateway supports X-Ray tracing for all API Gateway REST API endpoint types: Regional, edge-optimized, and private” (X-Ray for REST APIs). Better, the propagation works even where you have not switched it on: “If you call an API Gateway API from a service that’s already being traced, API Gateway passes the trace through, even if X-Ray tracing isn’t enabled on the API.” A caller Lambda with tracing on therefore keeps its trace across the gateway hop.

What you get out of it is the picture people buy a mesh for. “For APIs integrating with AWS services such as AWS Lambda and Amazon DynamoDB, you will see more nodes providing performance metrics related to those services. There will be a service map for each API stage” (service maps). Caller, gateway, callee, and the callee’s DynamoDB table, in one graph, per stage. In CDK it is tracingEnabled: true in deployOptions, which part 103 set.

OpenTelemetry is the emerging industry standard for tracing across service boundaries, and the closing section of this part returns to why, so be straight about the choice on the Lambda side. ADOT is “A secure, production-ready, AWS-supported distribution of the OpenTelemetry (OTel) SDK”, shipped as “fully managed Lambda layers that package everything you need to collect telemetry data using the OTel SDK. By consuming this layer, you can instrument your Lambda functions without having to modify any function code” (Lambda tracing). AWS then states the trade-off itself, and it is unusually honest: “The X-Ray and Powertools for AWS Lambda SDKs are part of a tightly integrated instrumentation solution offered by AWS. The ADOT Lambda Layers are part of an industry-wide standard for tracing instrumentation that collect more data in general, but may not be suited for all use cases.” Read that as a cost and cold-start warning, not as a discouragement.

The default for this layer is X-Ray plus Powertools, because the estate is already all-AWS and the service map arrives with a stage flag. Take ADOT when portability is a stated goal, which is the portability-versus-operator-free trade this part’s closing section describes: you pay in data volume and layer weight, and you buy an exit. There is a third option worth one paragraph. CloudWatch Application Signals “is an application performance monitoring (APM) solution that enables developers and operators to monitor the health and performance of their serverless applications built using Lambda”. You switch it on from the Lambda console in one click, with no instrumentation code, on enhanced ADOT layers, and it supports Node.js 22.x (Application Signals). Two caveats travel with it. “Using Application Signals for your Lambda functions incurs costs.” And it does not co-exist with hand-rolled tracing: “remove any existing X-Ray SDK instrumentation code from your function. Custom X-Ray SDK instrumentation code can interfere with the layer-provided instrumentation.”

The header you were forced to use

Now the negative half, and it follows directly from the identity design. Under AWS_IAM, SigV4 owns the Authorization header, so a forwarded user assertion has to ride in a custom header such as x-user-assertion. API Gateway’s redaction promise reads: “API Gateway redacts authorization headers, API key values, and similar sensitive request parameters from the logged data” (set up logging).

Be precise about what that sentence commits to, because it is easy to overclaim in both directions. AWS names authorization headers and API keys, then leaves “similar sensitive request parameters” deliberately open. It nowhere documents that an arbitrary custom header is redacted, and it nowhere documents that it is not. So the honest statement is not “your JWT is being logged”. It is that AWS gives you no promise that it is not. The one header the authorization model forced you into is the one header AWS declines to make a commitment about.

Part 104 answered that with config discipline: keep data tracing off on any stage that carries a user assertion. That is correct advice and it is not a control. Data tracing is a stage setting, and a stage setting is something a human can turn on. An engineer debugging a failing route in the middle of the night turns it on, because that is what it is for. The same is true of the second spill path, which is quieter and more common: a console.log(event) in the callee prints the whole event, headers included, into a Lambda log group nobody is auditing. Neither hole is closed by a rule. Both are closed by a policy that runs at ingestion, on log groups you did not have to name.

Masking a token you cannot anchor on

CloudWatch Logs data protection is that policy, and no managed identifier covers a JWT. The Credentials category is exactly five types: AwsSecretKey, OpenSSHPrivateKey, PgpPrivateKey, PkcsPrivateKey, and PuttyPrivateKey (credentials data types). There is no bearer-token or JWT identifier. That leaves one path: “Custom data identifiers (CDIs) let you define your own custom regular expressions that can be used in your data protection policy” (custom data identifiers).

The obvious regex is the one you cannot write. You would like to anchor on the JSON key, something like the literal text "x-user-assertion":", and the supported character set says no. AWS lists exactly three groups: Alphanumeric: (a-zA-Z0-9), Symbols: ( '_' | '#' | '=' | '@' |'/' | ';' | ',' | '-' | ' ' ), and RegEx reserved characters: ( '^' | '$' | '?' | '[' | ']' | '{' | '}' | '|' | '\' | '*' | '+' | '.' ). Read the list for what is missing. The colon is not there. The double quote is not there. Every JSON key anchor you would reach for needs both.

Anchor on the token instead. A JWT header segment is the base64url encoding of a JSON object, so it always begins with the two characters {", which always encode to eyJ. That gives a shape made entirely of characters the CDI grammar allows:

eyJ[A-Za-z0-9_-]*\.eyJ[A-Za-z0-9_-]*\.[A-Za-z0-9_-]*

Three base64url segments, the first two starting eyJ, separated by literal dots. It does not care which header carried the token, which log group it landed in, or whether it arrived through data tracing or through somebody’s console.log. The whole policy:

{
  "Name": "user-assertion-protection",
  "Version": "2021-06-01",
  "Configuration": {
    "CustomDataIdentifier": [
      {
        "Name": "JwtLikeToken",
        "Regex": "eyJ[A-Za-z0-9_-]*\\.eyJ[A-Za-z0-9_-]*\\.[A-Za-z0-9_-]*"
      }
    ]
  },
  "Statement": [
    {
      "Sid": "audit-findings",
      "DataIdentifier": ["JwtLikeToken"],
      "Operation": {
        "Audit": {
          "FindingsDestination": {
            "CloudWatchLogs": { "LogGroup": "security/data-protection-findings" }
          }
        }
      }
    },
    {
      "Sid": "mask-in-place",
      "DataIdentifier": ["JwtLikeToken"],
      "Operation": { "Deidentify": { "MaskConfig": {} } }
    }
  ]
}

Two details that cost people an afternoon. The backslashes are doubled because the policy document is JSON, so the regex escape survives parsing. And the limits are tight: “A maximum of 10 custom data identifiers are supported for each data protection policy”, each with a maximum length of 200 characters. This is a control for a handful of shapes, not a data-loss-prevention product.

The account-level policy and the free alarm

Attach it at the account, not at the log group. AWS: “You can create a data protection policy for all log groups in your account, and you can also create a data protection policies for individual log groups. When you create a policy for your entire account, it applies to both existing log groups and log groups that are created in the future.” (masking sensitive log data) The two levels stack rather than override each other: “custom data identifiers defined within an account-level policy work in combination with custom data identifiers defined in a log group-level policy” (custom data identifiers).

The reason to go account-level here is specific, not general. API Gateway creates its execution log group itself, named API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}, at deploy time. You do not know that name when you write the stack for a new service, and a per-log-group policy therefore always trails your deployments. The same argument covers every Lambda log group in the estate, which is where the console.log(event) path lands. An account-level policy protects the log group a team creates next week, which is exactly the log group you are worried about.

import * as logs from 'aws-cdk-lib/aws-logs';

// One per account. It covers existing log groups and every log group created
// later, including the ones API Gateway names for itself at deploy time.
new logs.CfnAccountPolicy(this, 'UserAssertionProtection', {
  policyName: 'user-assertion-protection',
  policyType: 'DATA_PROTECTION_POLICY',
  scope: 'ALL',
  policyDocument: JSON.stringify(dataProtectionPolicy),
});

Now be honest about what this bought, because masking is not deletion. AWS is clear that “Sensitive data is detected and masked when it is ingested into the log group. When you set a data protection policy, log events ingested to the log group before that time are not masked.” Everything already in your log groups stays as it is. And the token is still stored. “Only users who have the logs:Unmask IAM permission can view unmasked data”, and CloudWatchLogsFullAccess includes logs:Unmask, which is a policy plenty of engineers already hold. Masking hides the token from a casual reader. It does not destroy it, and it is not a substitute for not logging it. There is a bill too. Data protection is charged per GB scanned, $0.12 per GB in us-east-1 (CloudWatch pricing, read 2026-07-14), and an account-wide policy scans everything you ingest.

Which is why the Audit statement, not the mask, is the part that pays for itself. “A metric is emitted to CloudWatch when sensitive data is detected… This is the LogEventsWithFindings metric and it is emitted in the AWS/Logs namespace… Metrics emitted by data protection are vended metrics and are free of charge.” Alarm on it at a threshold of zero. Masking is the corrective control that limits the damage; the alarm is the detective control that tells you a user token reached a log group at all, which is the fact you actually need. A finding means some service on this layer is still forwarding an assertion somewhere it should not, and no amount of masking fixes that. It only hides it.

Common pitfalls

1. Naming the assertion in a $context.requestOverride.header.* field. The access log is safe by construction only because no $context variable exposes an arbitrary request header. An integration-request override is the one way to put the token back into the line.
2. Reading Count and Latency in the console and concluding that per-route metrics are on. Those are the API-level and stage-level dimensions. ApiName, Method, Resource, Stage requires detailed metrics, and AWS says it “will incur additional charges to your account”.
3. Treating “data tracing is off” as a control. It is a stage setting with an on switch, and incidents are when people reach for it. The control is a data protection policy at ingestion.
4. Anchoring the custom data identifier on the JSON key. The colon and the double quote are not in the supported character set, so "x-user-assertion":" is not expressible. Anchor on the eyJ shape of the token itself.
5. Calling masking a deletion. The event is stored, logs:Unmask reveals it, CloudWatchLogsFullAccess grants that permission, and events ingested before the policy existed were never masked at all.
6. Switching on Application Signals while X-Ray SDK instrumentation is still in the function. AWS says to remove it: “Custom X-Ray SDK instrumentation code can interfere with the layer-provided instrumentation.”

Where this is all heading

It would be easy to read this series as a serverless workaround: no mesh available, so improvise. That reading is backwards. The mesh world is walking toward the same conclusion from the other side, and AWS has already arrived.

AWS is retiring its own service mesh. The App Mesh user guide carries the notice on every page: “On September 30, 2026, AWS will discontinue support for AWS App Mesh. After September 30, 2026, you will no longer be able to access the AWS App Mesh console or AWS App Mesh resources” (App Mesh). New customers were shut out back in September 2024. AWS built a sidecar mesh, ran it for years, and is switching it off. That is not a small signal about where managed east-west connectivity is going.

Be precise about what AWS replaced it with, because the obvious summary is wrong. AWS did not abolish the proxy. For ECS it points customers at Service Connect, where, in AWS’s words, “the Envoy proxy, known as the Service Connect Proxy, is fully managed by AWS” (App Mesh to Service Connect), against App Mesh’s self-managed sidecar. Envoy is still in the task. It is simply no longer your problem. The retired idea is not the sidecar, it is you operating the sidecar.

The open-source mesh is moving the same way. Istio’s ambient mode reached GA in November 2024, and its pitch is a mesh without per-pod proxies: L4 handled by a per-node ztunnel, L7 proxying opt-in and separately scaled (ambient mode). Istio’s own motivation reads like a bill of indictment against the sidecar: proxies must be injected by rewriting the pod spec, upgrades mean restarting application pods, and “the CPU and memory resources must be provisioned for worst case usage of each individual pod” (introducing ambient mesh). Cilium reaches the same place through eBPF with a per-node Envoy rather than a per-pod one.

Do not overstate it, though, because it is not unanimous. Istio still supports sidecars and says they are not going away. Linkerd, the other CNCF graduated mesh, argues the opposite outright and is fixing sidecar lifecycle problems rather than removing the proxy. The honest claim is not that the industry abandoned sidecars. It is that the centre of gravity is moving off the per-workload proxy and into the platform.

Which is the whole point. Both worlds have concluded the same thing: identity, authorization, encryption, and telemetry for east-west calls belong to the platform, not bolted to the workload. Istio ambient is Kubernetes working to acquire a property serverless had by construction. Lambda never had a sidecar to remove. The gateway, IAM, and the managed network are the platform layer, and this series is just wiring them together.

The convergence is real, but the implementations are diverging, and that is the trade you are signing. Kubernetes is standardising on portable, vendor-neutral specifications that you operate yourself. SPIFFE and SPIRE carry workload identity (SPIFFE), a CNCF graduated project since 2022. OpenTelemetry, graduated in May 2026, carries tracing across service boundaries (OpenTelemetry). Serverless standardises on cloud primitives that you do not operate and cannot move: IAM SigV4, a private API Gateway, EventBridge. SigV4 and SPIFFE answer the same question, which workload is calling, with the same shape of answer, cryptographic and short-lived. The difference is who owns the trust root. You are buying an operator-free platform and paying in portability. That is a defensible trade, and it should be a conscious one.

One gap is worth admitting rather than papering over. The industry is settling on a two-part answer to identity on an internal call, and only half of it has landed. SPIFFE covers which workload is calling. For the other half, on whose behalf, the IETF’s Transaction Tokens draft is explicit that the two are not interchangeable: “A workload MUST NOT use a transaction token to authenticate itself to another workload, service or the TTS. Transaction tokens represents information relevant to authorization decisions and are not workload identity credentials” (Transaction Tokens). That is exactly the two-check rule part 104 arrived at from the AWS side. The difference is that the standard is still a working-group draft, so for now you build the second check yourself.

Where this series lands

The argument is short. Events first: if nobody is waiting on the reply, put a bus in the middle and none of this applies. If the answer has to come back inside the same call, do not deploy a mesh. Compose one out of primitives you already own. Part 101 chose the layer against VPC Lattice, where on identity Lattice is a tie at worst and ahead once you need to authorize on a request header, so the arguments that actually decide this are cost shape and API management. Part 102 settled the wire format: gRPC is a compute decision, not a gateway decision, and the format was never the contract. Part 103 built it, a private REST API behind a shared execute-api endpoint with AWS_IAM per route. Part 104 added the two identity checks in the callee, because SigV4 proves the service and never the user. Part 105 and part 106 are where the design meets a real estate: the cross-account perimeter, and what breaks under real traffic. And observability, the third thing a mesh sells, turns out to be a stage config and a log-group policy.

The default holds wherever the layer itself does: a multi-team, all-Lambda estate on private REST APIs with AWS_IAM. Log the identity fields. Read per-route latency out of the log, and turn on detailed metrics only for the routes you page someone about. Put one account-level data protection policy in front of every log group in the account. None of this is deployed yet, so treat the numbers throughout as list prices and the configuration as a plan. The first step is the cheapest thing in the series and it needs no migration: count the cross-team lambda:InvokeFunction grants in your account today. That number is the size of the coupling you are carrying, and driving it to zero is what the whole layer is for.

References

• API Gateway variables for access logging - The closed set of loggable fields, the identity variables that AWS_IAM fills in, and the rule that a format must carry $context.requestId or $context.extendedRequestId.
• Set up CloudWatch logging for a REST API - The redaction sentence that names authorization headers and API keys, and stops short of promising anything about a custom header.
• API Gateway metrics and dimensions - Definitions of Latency and IntegrationLatency, and the per-route dimension that “will incur additional charges to your account”.
• Tracing REST APIs with AWS X-Ray - X-Ray works on all REST endpoint types including private, and passes an inbound trace through even when tracing is off on the API.
• Using X-Ray service maps - One service map per API stage, with nodes for the integrated AWS services behind the route.
• Tracing Lambda functions in Node.js - The ADOT managed layers, and AWS’s own comparison against the X-Ray and Powertools SDKs.
• CloudWatch Application Signals for Lambda - One-click APM on enhanced ADOT layers, the cost note, and the requirement to remove existing X-Ray SDK instrumentation first.
• Help protect sensitive log data with masking - Account-level policies covering future log groups, masking at ingestion, the logs:Unmask permission, and the free LogEventsWithFindings metric.
• Custom data identifiers - The supported regex character set, which contains neither the colon nor the double quote, plus the 10-identifier and 200-character limits.
• Managed data identifiers for credentials - The five credential types AWS ships. No JWT, no bearer token, which is why a custom identifier is the only path.
• Amazon CloudWatch pricing - Data protection billed per GB scanned. Prices read 2026-07-14, us-east-1.
• AWS App Mesh - “On September 30, 2026, AWS will discontinue support for AWS App Mesh.” AWS is switching off its own sidecar mesh.
• Migrating from App Mesh to ECS Service Connect - The replacement is a managed sidecar, not no sidecar: “the Envoy proxy, known as the Service Connect Proxy, is fully managed by AWS”. An AWS blog, not a doc.
• Introducing ambient mesh - Istio’s own indictment of the sidecar: injection by pod-spec rewrite, restarts on upgrade, and worst-case resource provisioning per pod.
• IETF: OAuth 2.0 Transaction Tokens - The other half of the identity answer: “A workload MUST NOT use a transaction token to authenticate itself… are not workload identity credentials.” Still a draft.