Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Distributed Tracing at UBER Scale
Search
Yuri Shkuro
May 24, 2017
Programming
1
420
Distributed Tracing at UBER Scale
Presented at Monitorama-PDF 2017.
Video:
https://vimeo.com/221070602
Yuri Shkuro
May 24, 2017
Tweet
Share
More Decks by Yuri Shkuro
See All by Yuri Shkuro
TEMPLE: Six Pillars of Telemetry
yurishkuro
0
640
Schema-first application telemetry
yurishkuro
0
330
CNCF Webinar Series - Introducing Jaeger 1.0
yurishkuro
1
340
Would You Like Some Tracing With Your Monitoring?
yurishkuro
0
440
From zero to distributed traces: an OpenTracing tutorial
yurishkuro
1
840
Other Decks in Programming
See All in Programming
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
150
プロダクト志向なエンジニアがもう一歩先の価値を目指すために意識したこと
nealle
0
130
#QiitaBash MCPのセキュリティ
ryosukedtomita
1
1.4k
Discover Metal 4
rei315
2
140
iOS 26にアップデートすると実機でのHot Reloadができない?
umigishiaoi
0
130
技術同人誌をMCP Serverにしてみた
74th
1
650
RailsGirls IZUMO スポンサーLT
16bitidol
0
190
Hack Claude Code with Claude Code
choplin
4
2.1k
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
160
AI コーディングエージェントの時代へ:JetBrains が描く開発の未来
masaruhr
1
160
第9回 情シス転職ミートアップ 株式会社IVRy(アイブリー)の紹介
ivry_presentationmaterials
1
320
効率的な開発手段として VRTを活用する
ishkawa
0
140
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
A Tale of Four Properties
chriscoyier
160
23k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Embracing the Ebb and Flow
colly
86
4.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
Rails Girls Zürich Keynote
gr2m
95
14k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Transcript
Distributed Tracing at UBER Scale Crea7ng a treasure map for
your monitoring data Yuri Shkuro, UBER Technologies
ABOUT ME • SoAware Engineer on the Observability team in
NYC • Working on the open source distributed tracing system Jaeger • Co-founded the OpenTracing project • Banking industry survivor • Github: yurishkuro • TwiLer: @yurishkuro
Would You Like Some Tracing with Your Monitoring? What does
it take to roll it out?
Why Distributed Tracing • Distributed transac7on monitoring • Performance /
latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
JAEGER, Distributed Tracing • Open Source • OpenTracing inside •
In ac7ve development • PRs are welcome • Zipkin compa7ble • github.com/uber/jaeger
Who Thinks Tracing is Awesome?
None
None
Why Doesn’t Everyone Do Tracing?
Tracing Instrumenta7on is HARD EXPENSIVE BORING
Instrumenta7on • Metrics and logging are not new • Tracing
is both new and harder
Context Propaga7on A B C D E {context} {context} {context}
{context} Unique ID → {context} Edge service
Headers: . . . Trace ID . . . Instrumentation
APPLICATION / MICROSERVICE Handler Context [Span] Client Context [Span] Inbound HTTP Request Instrumentation Headers: . . . Trace ID . . . Outbound HTTP Request Context Propaga7on
In-Process Context Propaga7on Implicit, via Thread-Locals but: thread pools, futures
Explicit
It’s Also the Frameworks • Go: stdlib, gorilla, … •
Java: jaxrs2, okhLp, ApacheHLpClient, … • Python: Flask, Django, Tornado, urllib2, … • Node.js – who knows…
OpenTracing to the Rescue
No Help With In-Process Propaga7on • Must be done manually
• UBER has 2000-3000 microservices • Resources of the tracing team are limited • Developers must instrument their code!
BITE MAKE ME! How do we mobilize the org?
Traveling Salesman Problem 2017 edi7on
They Must Want Your Product or S7cks and Carrots
Recap: Why Distributed Tracing • Distributed transac7on monitoring • Performance
/ latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
Service Dependency Analysis • Explain to us what we just
built • Who are my dependencies • Workflow analysis • Where is all this traffic coming from? • Service 7ers
Baggage • Tenancy, test or produc7on – Set at the top
– Used at the storage layer, prod or test DB • Authen7ca7on tokens – Signed user or service iden7ty – Checked at mul7ple levels
S7cks and Carrots • Get other teams build features on
top – Performance team – Capacity & cost accoun7ng – Baggage • More carrots • Eventually they become s7cks (peer pressure)
Each Organiza7on is Different Find what works best
How to Measure Adop7on? Measure everything
Does Service X Report Traces? • Daily aggrega7on job •
Auto-book 7ckets • Build a dashboard • Pass/Fail: too easy to pass
Trace Quality Score • Inspect traces – See a caller, but
no spans • Join with other data – Rou7ng logs • Auto-book 7ckets (carefully, not for everyone) – With detailed report
Trace Quality Metrics by Service
Thank You • Jaeger – hLps://github.com/uber/jaeger – Blog: Evolving Distributed
Tracing at UBER – Blog: Take OpenTracing for a HotROD Ride • OpenTracing: hLp://opentracing.io/ • We are hiring • @yurishkuro