Seeing Through Serverless Observability for AWS Lambda with ADOT and CloudWatch Application Signals SHIRO SEIKE (@SEIKE460),@seike460 FUSIC | 22 AUG 2026 2025-2026 Japan AWS Top Engineers 

Serverless ships fast — until it breaks Then production becomes a black box, and the clock is ticking 

Agenda 1. The Black Box 2. Observability for Lambda 3. Zero-Code with ADOT 4. Traces Across Async 5. RED Metrics and SLOs 6. Portable Instrumentation 

1 | The Black Box A real serverless implementation, unseen 

Fully serverless, async by design → API Gateway → → api fn custom bus → queue + DLQ ← 202 Accepted, right away + scheduled batch DynamoDB audit trail (S3) The caller got 202 before any real work happened worker fn 

Request 7f3a41 failed. Where? API gateway access log only api fn → its own log group bus + queue → no request logs Each hop keeps its own diary. Nobody keeps the plot. worker fn → another log group 

NO DIRECT metric for events that match no rule PutEvents can succeed while no target is invoked; EventBridge has no UnmatchedEvents metric 

Cold starts hide behind your p50 p50 looks healthy. The user who hit a cold start at 9am disagrees. 

2 | Observability for Lambda Three signals, one question 

Structured logs are still islands // one line from the api function's log const line = { level: "info", requestId: "7f3a41", path: "/secrets/sync", status: 202, latencyMs: 84, }; // well-structured, greppable — and alone: // nothing links it to the worker's story 

RED is the question that pages you Rate — how many requests Errors — how many failed Duration — how long they took Application Signals collects Latency, Error, and Fault; volume and availability are derived 

Instrument the platform, not the code Lambda is ephemeral, async, and managed — telemetry belongs to the runtime 

3 | Zero-Code with ADOT CJS bundle, one layer, one wrapper 

Three runtime settings. Zero handler code. // CDK — the core zero-code instrumentation fn.addLayers(LayerVersion.fromLayerVersionArn(fn, "Otel", "arn:aws:lambda:" + region + ":615299751070" + ":layer:AWSOpenTelemetryDistroJs:14")); fn.addEnvironment("AWS_LAMBDA_EXEC_WRAPPER", "/opt/otel-instrument"); fn.addEnvironment("OTEL_SERVICE_NAME", "api"); // + permissions equivalent to the AWS managed policy 

The handler stays unchanged // the entire Lambda entry point, before and after import { handle } from "hono/aws-lambda"; import { app } from "./app.js"; export const handler = handle(app); // traces, AWS SDK + HTTP instrumentation, context // propagation — all injected by the layer at init 

One helper enforces the complete setup // condensed from the real helper export function instrumentLambda(fn, serviceName) { assertActiveTracing(fn); fn.addLayers(fromArn(otelLayerArn(region))); fn.addEnvironment("AWS_LAMBDA_EXEC_WRAPPER", "/opt/otel-instrument"); fn.addEnvironment("OTEL_SERVICE_NAME", serviceName); role.attachInlinePolicy(applicationSignalsPolicy); } // stack tests verify layer, env, policy, and tracing 

4 | Traces Across Async One request, linked stories 

A trace shows what logs never could api: POST /secrets/sync (Lambda invocation) DynamoDB: PutItem EventBridge: PutEvents Initialization — present only on a cold start (trace anatomy; timings intentionally omitted) 

You don't get one trace — you get links The wish The reality one trace for request 7f3a41 trace A api api bus queue bus (PutEvents) ⇠ linked ⇢ trace B queue worker One trace, end to end worker X-Ray: trace linking / OTel: span links 

Prove the link with a probe // a tiny emitter Lambda, deployed to verify (it stays) export const handler = async () => { await events.send(new PutEventsCommand({ Entries: [{ EventBusName: bus, Source: "probe", DetailType: "trace-probe", Detail: "{}" }], })); }; // pairs with a probe consumer behind the rule // CLI put-events is NOT instrumented: no trace // context would enter the bus. The emitter's does. 

Capture first. Index what you need. new xray.CfnTransactionSearchConfig(this, "Tx", { indexingPercentage: stage === "preview" ? 100 : 1, }); // Lambda invocation spans are captured without sampling // child spans follow the configured head-sampling decision // indexingPercentage controls searchable trace summaries // preview indexes 100%; production starts at 1% 

5 | RED Metrics and SLOs Application Signals does the boring parts 

Discovery is one resource away new appsignals.CfnDiscovery(this, "Signals", {}); Region, in CDK // once per account and RED metrics → spans → functions wearing the layer Application Map Application Signals SLOs 

Generated map. Generated health views. 5 3 0 services metrics lines instrumented Latency, Error, Fault of custom metric code 

SLOs live in the repo, not a wiki new appsignals.CfnServiceLevelObjective(this, "Avail", { name: "api-availability", sli: { /* availability, threshold 99.5 */ }, goal: { attainmentGoal: 99.5, interval: { rollingInterval: { duration: 7, durationUnit: "DAY" } } }, }); // second SLO on the same service: p99 <= 500 ms // alarms: 14.4x / 1h and 6x / 6h for each SLO 

14.4 × error-budget burn rate The classic fast-burn rate: at this pace our 7-day error budget is gone in half a day — the fast alarm pages, a slow 6× alarm nags 

ADOT spans end where our process ends no Lambda process runs here — ADOT cannot emit an application span → DDB Streams → Pipes → Firehose S3 WORM audit This managed pipeline is monitored by a daily DynamoDB-to-S3 count reconciliation, not by Lambda autoinstrumentation 

6 | Portable Instrumentation OpenTelemetry data, AWS-specific operations 

Portable telemetry. Rebuild operations. today: CloudWatch + X-Ray OTel spans → AWS trace propagation today Lambda + ADOT layer migration: add a collector + OTLP exporter keep — OTel instrumentation and semantic conventions rebuild — exporter/auth/network, SLOs, alarms, and map 

AWS recommends OpenTelemetry 2026-02 today end date X-Ray SDKs enter maintenance mode Official migration path is OTel / ADOT N/A security fixes only, no new features the layer you saw is the recommended route the current AWS timeline lists no end date 

See, measure, stay portable One layer makes Lambda speak OTel Signals turns spans into SLOs OTel underneath — portable instrumentation Start with one function — CJS bundle, layer, wrapper, IAM, service name. 

Thank You Everything will be Serverless — make it a world you can see through