Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Distributed Tracing at UBER Scale
Search
Yuri Shkuro
May 24, 2017
Programming
1
420
Distributed Tracing at UBER Scale
Presented at Monitorama-PDF 2017.
Video:
https://vimeo.com/221070602
Yuri Shkuro
May 24, 2017
Tweet
Share
More Decks by Yuri Shkuro
See All by Yuri Shkuro
TEMPLE: Six Pillars of Telemetry
yurishkuro
0
650
Schema-first application telemetry
yurishkuro
0
330
CNCF Webinar Series - Introducing Jaeger 1.0
yurishkuro
1
340
Would You Like Some Tracing With Your Monitoring?
yurishkuro
0
440
From zero to distributed traces: an OpenTracing tutorial
yurishkuro
1
840
Other Decks in Programming
See All in Programming
MCPを使ってイベントソーシングのAIコーディングを効率化する / Streamlining Event Sourcing AI Coding with MCP
tomohisa
0
190
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ~継続的な取り組みを添えて~
takutakahashi
4
1.4k
#QiitaBash TDDで(自分の)開発がどう変わったか
ryosukedtomita
1
110
Advanced Micro Frontends: Multi Version/ Framework Scenarios
manfredsteyer
PRO
0
120
Claude Code で Astro blog を Pages から Workers へ移行してみた
codehex
0
170
React 使いじゃなくても知っておきたい教養としての React
oukayuka
15
4.3k
「次に何を学べばいいか分からない」あなたへ──若手エンジニアのための学習地図
panda_program
3
670
型で語るカタ
irof
1
860
マッチングアプリにおけるフリックUIで苦労したこと
yuheiito
0
250
中級グラフィックス入門~効率的なメッシュレット描画~
projectasura
3
1.9k
AIコーディングエージェント全社導入とセキュリティ対策
hikaruegashira
15
8.6k
商品比較サービス「マイベスト」における パーソナライズレコメンドの第一歩
ucchiii43
0
230
Featured
See All Featured
Designing Experiences People Love
moore
142
24k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
4 Signs Your Business is Dying
shpigford
184
22k
A designer walks into a library…
pauljervisheath
207
24k
Thoughts on Productivity
jonyablonski
69
4.8k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Typedesign – Prime Four
hannesfritz
42
2.7k
Into the Great Unknown - MozCon
thekraken
40
1.9k
What's in a price? How to price your products and services
michaelherold
246
12k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Transcript
Distributed Tracing at UBER Scale Crea7ng a treasure map for
your monitoring data Yuri Shkuro, UBER Technologies
ABOUT ME • SoAware Engineer on the Observability team in
NYC • Working on the open source distributed tracing system Jaeger • Co-founded the OpenTracing project • Banking industry survivor • Github: yurishkuro • TwiLer: @yurishkuro
Would You Like Some Tracing with Your Monitoring? What does
it take to roll it out?
Why Distributed Tracing • Distributed transac7on monitoring • Performance /
latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
JAEGER, Distributed Tracing • Open Source • OpenTracing inside •
In ac7ve development • PRs are welcome • Zipkin compa7ble • github.com/uber/jaeger
Who Thinks Tracing is Awesome?
None
None
Why Doesn’t Everyone Do Tracing?
Tracing Instrumenta7on is HARD EXPENSIVE BORING
Instrumenta7on • Metrics and logging are not new • Tracing
is both new and harder
Context Propaga7on A B C D E {context} {context} {context}
{context} Unique ID → {context} Edge service
Headers: . . . Trace ID . . . Instrumentation
APPLICATION / MICROSERVICE Handler Context [Span] Client Context [Span] Inbound HTTP Request Instrumentation Headers: . . . Trace ID . . . Outbound HTTP Request Context Propaga7on
In-Process Context Propaga7on Implicit, via Thread-Locals but: thread pools, futures
Explicit
It’s Also the Frameworks • Go: stdlib, gorilla, … •
Java: jaxrs2, okhLp, ApacheHLpClient, … • Python: Flask, Django, Tornado, urllib2, … • Node.js – who knows…
OpenTracing to the Rescue
No Help With In-Process Propaga7on • Must be done manually
• UBER has 2000-3000 microservices • Resources of the tracing team are limited • Developers must instrument their code!
BITE MAKE ME! How do we mobilize the org?
Traveling Salesman Problem 2017 edi7on
They Must Want Your Product or S7cks and Carrots
Recap: Why Distributed Tracing • Distributed transac7on monitoring • Performance
/ latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
Service Dependency Analysis • Explain to us what we just
built • Who are my dependencies • Workflow analysis • Where is all this traffic coming from? • Service 7ers
Baggage • Tenancy, test or produc7on – Set at the top
– Used at the storage layer, prod or test DB • Authen7ca7on tokens – Signed user or service iden7ty – Checked at mul7ple levels
S7cks and Carrots • Get other teams build features on
top – Performance team – Capacity & cost accoun7ng – Baggage • More carrots • Eventually they become s7cks (peer pressure)
Each Organiza7on is Different Find what works best
How to Measure Adop7on? Measure everything
Does Service X Report Traces? • Daily aggrega7on job •
Auto-book 7ckets • Build a dashboard • Pass/Fail: too easy to pass
Trace Quality Score • Inspect traces – See a caller, but
no spans • Join with other data – Rou7ng logs • Auto-book 7ckets (carefully, not for everyone) – With detailed report
Trace Quality Metrics by Service
Thank You • Jaeger – hLps://github.com/uber/jaeger – Blog: Evolving Distributed
Tracing at UBER – Blog: Take OpenTracing for a HotROD Ride • OpenTracing: hLp://opentracing.io/ • We are hiring • @yurishkuro