Tracing The Future - Turning Observability Noise into Measurable Values

Amadeus is revolutionizing airline retail by laying the foundation for a cloud-native platform capable of handling thousands of transactions per second—powering billions of journeys annually. In this high-throughput, distributed environment, traditional logging quickly becomes noise. This session explores how Amadeus is embracing modern observability practices to transform that noise into actionable, measurable value.

We’ll demystify the differences between logging, tracing, and metrics, and introduce OpenTelemetry as a cornerstone of this transformation. From span events and attributes to sampling strategies, we’ll walk through the developer prerequisites and tooling needed to implement observability at scale. You’ll learn how we tackled challenges like telemetry data retention, performance analyses and metrics-tracing connection —while unlocking advanced tracing capabilities that go beyond troubleshooting to deliver business insights per tenant and per feature.

Expect a live demo, real-world lessons, and a practical look at how observability, when done right, becomes a strategic asset—not just a debugging tool.

Transcript

1. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Tracing The Future Turning Observability Noise into Amadeus Engineers Measurable Values Riccardo Puppo TSI-TSR-DEP-IDQ, Darren Costello TRU-AIR-ENG-OFR
2. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries About us Riccardo Puppo Darren Costello • Cloud-native apps, DevOps, Quarkus/Skube microservices, CI/CD automation • Cloud-native apps, DevOps, Quarkus/Skube microservices, observability • Lead Principal Engineer • Principal Engineer • One highlight of my journey: taking part in a hackathon that became the starting point for Nevio Offer Orchestration. • Motivations: Collaborating on challenging projects with great team mates. Amadeus Engineers
3. agenda 1. Case Study 2. Lost in the noise 3. Breaking the Observability Silos 4. Thinking in Traces 5. Live Demo 6. Best Practices & The Path Forward 7. Q&A Amadeus Engineers
4. Case Study: Flight Booking Online Friday Afternoon fire CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Amadeus Engineers
5. Case Study: Flight Booking Online Friday Afternoon fire CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Latency spike in CNA Prime Booking flow Amadeus Engineers
6. Case Study Friday Afternoon CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries fire : meet Logan and Tracey devops Latency spike in Prime Booking flow How do we quickly diagnose and resolve the issue in Engineers modern CNA distributed system ? Amadeus
7. Case Study Friday Afternoon CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries fire Latency spike in Prime Booking flow Metrics Response Time Percentiles 5s 99 Percentile spike 50 Percentile spike 3s 2.5s Amadeus 17:14 17:18 Engineers
8. Case Study Friday Afternoon CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries fire Latency spike in Prime Booking flow Metrics Logs Response Time Percentiles Manual/Scripted Correlation correlation_id 5s 99 Percentile spike 50 Percentile spike μsA μsB Paas 1 μsC 3s Paas 2 2.5s Amadeus 17:14 17:18 17:14 17:18 Engineers
9. Case Study Friday Afternoon CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries fire Latency spike in Prime Booking flow Metrics Logs Response Time Percentiles Manual/Scripted Correlation correlation_id 5s 99 Percentile spike 50 Percentile spike μsA Traces Tentative Trace analysis μsB Paas 1 μsC 3s Paas 2 2.5s Amadeus 17:14 17:18 17:14 17:18 17:14 Engineers
10. Lost in noise Key Takeaways CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Latency spike in Prime Booking flow Why Logan's approach is breaking down • High - volume, low - value telemetry • Siloed signals across tools • No causality in logs or metrics High MTTR and devops Cognitive Overload Amadeus Engineers
11. Breaking the silos Amadeus Engineers
12. Breaking the silos Metrics CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Logs Traces Tell us WHAT Tell us WHY Tell us WHERE and HOW Great for trends and alerts Detailed snapshot of an event I nherent causality Struggle with High Cardinality Lack inherent causality Designed for high cardinality Lack link with aggregated metrics world Amadeus Engineers
13. Breaking the silos CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Metrics Logs Traces Breaking the silos and connect the pillars to enable a modern distributed observability model Amadeus Engineers
14. First crack of silos Friday Afternoon fire : Tracey's approach CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Latency spike in Prime Booking flow Amadeus Engineers
15. First crack of silos Friday Afternoon fire : Tracey's approach CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Latency spike in Prime Booking flow Metrics 5s Response 99 Percentile spike Time Percentiles 50 Percentile spike 3s 2.5s Amadeus 17:14 17:18 Engineers
16. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries First crack of silos Friday Afternoon fire : Tracey's approach Latency spike in Prime Booking flow Metrics 5s • Exemplars are specific data points that provide context to general aggregations • They allow correlation between metrics and the original API calls where measurement are recorded Amadeus Engineers 17:14 17:18
17. First crack of silos App-wise Exemplars under the Hood CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries • A metadata attached to a metric at the moment of its measurement. exemplar • Quarkus micrometer extension provide exemplars built-in feature for http metrics metric + exemplar Code + OTel SDK Generate Metrics Attach TraceId (Exemplar ) • Exemplars are generated only if tracing context is active http_request_duration_seconds_bucket{le="0.1"} 240 # {trace_id="abc123", span_id="def456"} metric exemplar Instrumented Microservice Active Tracing Context Amadeus Engineers Amadeus hybrid cloud ▼
18. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries First crack of silos Exemplars under the Hood Prometheus-wise • An optional feature to store exemplar the trace_id as a special label RAM Cyclic Exemplar Storage alongside the time-series data Scrape and ingest metric samples and associated exemplar Code + OTel SDK Generate Metrics Attach TraceId (Exemplar ) Prometheus server exemplar • Exemplar are stored in RAM metric into a Cyclic Buffer for each TSDB Metrics individual time series Instrumented Microservice Active Tracing Context Amadeus Engineers Amadeus hybrid cloud ▼
19. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries First crack of silos Friday Afternoon fire : Tracey's approach Metrics Latency spike in Prime Booking flow Traces Logs Exemplar link to trace • Traces enhanced with logs Amadeus Engineers
20. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries First crack of silos Key Takeaways Latency spike in Prime Booking flow Why Tracey's approach works better • Exploit exemplars and power of connected observability • Switch from noise to unified narrative Metrics Traces Logs • Making sense of Observability Data - Drastically reduce MTTR and devops Cognitive overload - Efficient Use of Resources Amadeus Engineers
21. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries First crack of silos Key Takeaways Why Tracey's approach works better • Beyond exemplars… Metrics Logs Traces Amadeus Engineers
22. Thinking in Traces Amadeus Engineers
23. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Traces Metrics Logs #2 #1 #3 • Elevate traces to the primary tool in investigations Amadeus Engineers
24. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Logs Traces Logs Traces Component A Span Span Component B Span Span Component C Span Span Amadeus Engineers
25. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Logs Traces Traces + Logs Component A Span Span Component B Span Span Component C Span Span Amadeus Engineers
26. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Vocab Description Trace A record of a single request's journey through a distributed system Span Timed operations representing a single unit of work Logs Traces Attribute Key value pair that can be added to spans and events as enriching metadata Span Event Discrete event on a span, like a structured log Amadeus Engineers
27. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Logs Traces Span Events - Span events bring the WHY of logs into traces o Define key - value attributes that aid investigation o Structured for easy searching o Low cost – simple metadata attaches to spans Amadeus Engineers
28. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Logs Traces Exceptions - Special type of span event o Visual error indicator o Searchable error status o Includes ▪ Exception message ▪ Stack traces ▪ Types o Can add attributes Amadeus Engineers
29. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Thinking in Traces Logs Traces Searching Amadeus Engineers
30. Live Demo Amadeus Engineers
31. Conclusions Amadeus Engineers
32. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Recommendations Reconnect Observability Pillars Leverage Exemplars & Span Events Phased Approach SDKs available Grafana & Open Telemetry Semantic Naming conventions Shared libraries Amadeus Engineers
33. CONFIDENTIAL & RESTRICTED © A madeus IT Group and its affiliates & subisdiaries Recommendations ENRICH • Use Examplars to get from metrics to traces • Enrich Spans with attributes & events instead of logging Phased Approach REPLACE • Drop high volume, low value logging cases • Replace other use cases by span events • Realise the cost savings of reduced logging UNDERSTAND • Useful logging – audit, startup/shutdown, dumps • Filter tracing – sampling to keep what you need • Export for Analysis Amadeus Engineers
34. Ask me anything Amadeus Engineers