Repurposing OpenTelemetry Traces as Test Data: Breaking the Cost Barrier in System Migration

Talked at KubeCon + CloudNativeCon Japan 2026: https://kubecon-cloudnativecon-japan-2026.sessionize.com/session/1194861

Proving a new system matches the old one is the hardest part of legacy migration. Writing equivalence tests from scratch is slow and expensive.

The fix is to repurpose what the live system already emits: capture request/response pairs as OTel trace spans and treat them as ground truth for testing the replacement.

A PoC on a Java-to-Go migration confirmed this works. Four CRUD endpoints were instrumented; the spans drove E2E tests covering all four.

OTel's standard conventions don't capture HTTP response bodies, so observability traces aren't automatically useful for testing. Custom attributes are required.

The PoC also tested OBI (opentelemetry-ebpf-instrumentation), which traces without code changes. Basic coverage works. But adding custom attributes via eBPF is impractical, so OBI alone isn't enough for test-quality traces.

You'll leave knowing when OTel traces are sufficient for migration testing, when they're not, and whether OBI closes the gap.

Transcript

1. #KubeCon #CloudNativeCon Repurposing OpenTelemetry Traces as Test Data: Breaking the Cost Barrier in System Migration Yoshiki Fujikane (@ffjlabo), CyberAgent, inc.
2. About me Yoshiki Fujikane • CyberAgent, Inc. (Japan) • AmebaLIFE Business Division • Platform Engineer • nickname : fujiwo (pronounced: fujio)
3. Today’s Talk In system migration, How do we prove that the new system still behaves like the old one?
4. System migration Building a new system while preserving the behavior of the old one
5. The reality around a migration 😎 🤔 🤯 How hard can it be? The Reality Hits Subjective Chaos
6. 😎 How hard can it be? • ✅ It works on the production • ✅ There is a source code for it Running System
7. 🤔 The Reality Hits • ✅ It works on the production • ✅ There is a source code Running System • ❌ Hard to understand the code • ❌ No latest spec documents • ❌ No tests • ❌ No people who knows it well
8. 🤯 Subjective Chaos Running System New System 🤯 Created similar one! It might be OK Let’s release!
9. 🔥🔥🔥
10. We rely on guesswork • It behaves the same as before – probably. • The spec document said so – probably. • Someone who knows the system said it was fine – probably. No facts. Nothing recorded that could support the claim.
11. The real cost barrier: prepare the proof based on the fact • Nobody has facts – stale docs, departed authors, code without test… • Need spec re-discovery, but do it based on the guesswork
12. Where is the fact?
13. 💡In the running system!
14. Running system as source of truth Running System New System A running system already knows what it does. Let it tell us!
15. Capture the input/output of the system System input output { “input”: …, “output”: …, Test data for the new system } Capture the pair – then reuse it as the test: same input, expect the same output.
16. But How to record it?
17. Observability
18. “Observability” as measurement proof Observability lets you understand a system from the outside by letting you ask questions about that system without knowing its inner workings. Ref: https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability • Emitted by the running system itself – exactly the source we said should “tell us” • Standardized end to end (OpenTelemetry) – instrumentation, Collector pipeline
19. The signal that fits: Trace • Log ◦ what happened at a point, but not how it connects across the entire system • Trace ✅ ◦ can be tied related detail of the execution as attributes in a single span • Metrics ◦ less information about the execution
20. From proof by guesswork measurement
21. Three conditions the capture must meet • Evidence: Behavior recorded as the input/output of a system • Containment: Limited impact on the application – with an off switch • Coverage: Deployable on the system you actually need to prove
22. Three conditions the capture must meet • Evidence: Behavior recorded as the input/output of a system • Containment: Limited impact on the application – with an off switch • Coverage: Deployable on the system you actually need to prove So far, all of this is on paper. Do all three actually hold?
23. Let’s find out with PoC!
24. PoC: use traces as test data 1. Record 2. Generate 3. Run Locally previous system { “input”: …, “output”: …, } Test fixtures E2E test for new system
25. PoC setup: REST API with Go • Goal: prove the pipeline – trace in, runnable E2E tests out • System: REST API implemented by Go ◦ net/http, instrumented with otelhttp ◦ with an empty database for determinism • Environment: docker compose App (REST API) Jaeger (trace) Test data (e2e test)
26. Problem: no request/response body in trace • No request/response body from otelhttp -> outside the OTel semantic conventions for now • Observability traces != test data, out of the box
27. Solution: custom instrumentation Attach request/response body as custom attributes
28. Learning 1 A trace can hold the behavior. By default, it does not.
29. Create test fixture from traces
30. It worked. But can we take this to production?
31. Two walls App trace with body Wall 1 – Performance cost • Bodies are large and attached to every request • Latency and trace volume grow with traffic Trace Backend (Jaeger、Datadog…) Wall 2 – Data leaving the boundary • Trace backend is often SaaS • Sensitive data leaves the system, by default
32. For Wall 1, Selection: a feature flag decides what to capture { Rules - /items/* - /items { “http.route”: “/items/1”, “http.request.body”: …, “http.response.body”: …, “body_captured”: true, “http.route”: “/helloworld”, … } } Test data App OTel Collector Trace Backend
33. For Wall 2, Control: the Collector routes it away from SaaS processors: filter/body_captured: error_mode: ignore traces: span: - 'attributes["body_captured"] != true' filter/others: error_mode: ignore traces: span: - 'attributes["body_captured"] == true' service: pipelines: traces/body_captured: receivers: [otlp] processors: [filter/body_captured] exporters: [file/body_captured] traces/others: receivers: [otlp] processors: [filter/others] exporters: [otlp/jaeger] exporters: file/body_captured: path: /var/log/otel/body_captured.jsonl append: true otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true { { “http.route”: “/items/1”, “http.request.body”: …, “http.response.body”: …, “body_captured”: true, “http.route”: “/helloworld”, … } } Test data App OTel Collector Trace Backend
34. Learning 2 You cannot capture everything. What, when, and how much – That choice is the design
35. OK! It’s fine…?
36. Premise: we can instrument Sometimes, hard to instrument • Not instrumented at all • Nobody knows how the system is put together • Nobody knows where the source code lives etc… Such systems with high hurdle tend to be targets for migration ):
37. Zero-code Instrumentation by OBI
38. OBI (OpenTelemetry eBPF Instrumentation) • Initial created as Grafana Beyla and donated to OpenTelemetry by Grafana Labs • Traces generated from eBPF-captured packets • Without code changes, language or framework dependency https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation
39. How it works App OBI OTel Collector https://opentelemetry.io/docs/zero-code/obi/#how-obi-works
40. Attach body by OBI App OBI ebpf: buffer_sizes: http: 8192 payload_extraction: http: enrichment: enabled: true policy: default_action: headers: include body: include OTel Collector Test data
41. Select traces by custom OTel Collector with feature flag Rules: /items/* App OBI processors: featureflag: exporters: file/body_captured: path: /var/log/otel/body_captured.jsonl append: true service: pipelines: traces/body_captured: receivers: [otlp] processors: [featureflag, batch] exporters: [file/body_captured] Test data OTel Collector { “http.route”: “/hellowold”, … { } “http.route”: “/items/1”, “http.request.body”: …, “http.response.body”: …, “body_captured”: true, }
42. Learning 3 The vantage point can move outside the application process. Bodies follows it. Application context does not.
43. Migration from Java to Go with OBI It works fine 🎉 Rules: - /items/* - /items Java App OBI OTel Collector Test data E2E test for GO
44. Summary OTel traces can be repurposed as characterization tests for system migration • Evidence: Behavior recorded as the input/output of a system • Containment: Limited impact on the application – with an off switch • Coverage: Deployable on the system you actually need to prove
45. Future Work: further considerations • Consider the side-effect API (Update, Delete…) • Manage test data • Security capability to use eBPF • Cost to maintain OTel-Collector, OBI …
46. Thank you for listening!