Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Distributed Tracing at UBER Scale
Yuri Shkuro
May 24, 2017
Programming
1
210
Distributed Tracing at UBER Scale
Presented at Monitorama-PDF 2017.
Video:
https://vimeo.com/221070602
Yuri Shkuro
May 24, 2017
Tweet
Share
More Decks by Yuri Shkuro
See All by Yuri Shkuro
CNCF Webinar Series - Introducing Jaeger 1.0
yurishkuro
1
270
Would You Like Some Tracing With Your Monitoring?
yurishkuro
0
240
From zero to distributed traces: an OpenTracing tutorial
yurishkuro
1
760
Other Decks in Programming
See All in Programming
Seleniumでイキってたらサーバを絞め落としかけてた話
kenfujita
0
360
heyにおけるCI/CDの現状と課題
fufuhu
1
550
シェーダー氷山発掘記
logilabo
0
140
Amazon ECSのネットワーク関連コストの話
msato
0
610
VisualProgramming_GoogleHome_LINE
nearmugi
1
140
A Philosophy of Software Design 後半
yosuke_furukawa
PRO
9
2.6k
Beyond Micro Frontends: Frontend Moduliths for the Enterprise @wad2022
manfredsteyer
PRO
0
120
GoogleI/O2022 LT報告会資料
shinsukefujita1126
0
230
はじめてのプルリク - BLEA 編
watany
0
140
Embracing commonMain for Android Development - Droidcon SF 2022
handstandsam
4
230
JSのウェブフレームワークで高速なルーターを実装する方法
usualoma
1
1.3k
Web API連携でCSRF対策がどう実装されてるか調べた / how to implements csrf-detection on Web API
yasuakiomokawa
2
250
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.2k
VelocityConf: Rendering Performance Case Studies
addyosmani
316
22k
Pencils Down: Stop Designing & Start Developing
hursman
112
9.8k
StorybookのUI Testing Handbookを読んだ
zakiyama
4
2.2k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
100
5.9k
Building Applications with DynamoDB
mza
83
4.7k
Bootstrapping a Software Product
garrettdimon
296
110k
Statistics for Hackers
jakevdp
781
210k
GraphQLの誤解/rethinking-graphql
sonatard
27
6.5k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
351
21k
Docker and Python
trallard
27
1.6k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
181
15k
Transcript
Distributed Tracing at UBER Scale Crea7ng a treasure map for
your monitoring data Yuri Shkuro, UBER Technologies
ABOUT ME • SoAware Engineer on the Observability team in
NYC • Working on the open source distributed tracing system Jaeger • Co-founded the OpenTracing project • Banking industry survivor • Github: yurishkuro • TwiLer: @yurishkuro
Would You Like Some Tracing with Your Monitoring? What does
it take to roll it out?
Why Distributed Tracing • Distributed transac7on monitoring • Performance /
latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
JAEGER, Distributed Tracing • Open Source • OpenTracing inside •
In ac7ve development • PRs are welcome • Zipkin compa7ble • github.com/uber/jaeger
Who Thinks Tracing is Awesome?
None
None
Why Doesn’t Everyone Do Tracing?
Tracing Instrumenta7on is HARD EXPENSIVE BORING
Instrumenta7on • Metrics and logging are not new • Tracing
is both new and harder
Context Propaga7on A B C D E {context} {context} {context}
{context} Unique ID → {context} Edge service
Headers: . . . Trace ID . . . Instrumentation
APPLICATION / MICROSERVICE Handler Context [Span] Client Context [Span] Inbound HTTP Request Instrumentation Headers: . . . Trace ID . . . Outbound HTTP Request Context Propaga7on
In-Process Context Propaga7on Implicit, via Thread-Locals but: thread pools, futures
Explicit
It’s Also the Frameworks • Go: stdlib, gorilla, … •
Java: jaxrs2, okhLp, ApacheHLpClient, … • Python: Flask, Django, Tornado, urllib2, … • Node.js – who knows…
OpenTracing to the Rescue
No Help With In-Process Propaga7on • Must be done manually
• UBER has 2000-3000 microservices • Resources of the tracing team are limited • Developers must instrument their code!
BITE MAKE ME! How do we mobilize the org?
Traveling Salesman Problem 2017 edi7on
They Must Want Your Product or S7cks and Carrots
Recap: Why Distributed Tracing • Distributed transac7on monitoring • Performance
/ latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
Service Dependency Analysis • Explain to us what we just
built • Who are my dependencies • Workflow analysis • Where is all this traffic coming from? • Service 7ers
Baggage • Tenancy, test or produc7on – Set at the top
– Used at the storage layer, prod or test DB • Authen7ca7on tokens – Signed user or service iden7ty – Checked at mul7ple levels
S7cks and Carrots • Get other teams build features on
top – Performance team – Capacity & cost accoun7ng – Baggage • More carrots • Eventually they become s7cks (peer pressure)
Each Organiza7on is Different Find what works best
How to Measure Adop7on? Measure everything
Does Service X Report Traces? • Daily aggrega7on job •
Auto-book 7ckets • Build a dashboard • Pass/Fail: too easy to pass
Trace Quality Score • Inspect traces – See a caller, but
no spans • Join with other data – Rou7ng logs • Auto-book 7ckets (carefully, not for everyone) – With detailed report
Trace Quality Metrics by Service
Thank You • Jaeger – hLps://github.com/uber/jaeger – Blog: Evolving Distributed
Tracing at UBER – Blog: Take OpenTracing for a HotROD Ride • OpenTracing: hLp://opentracing.io/ • We are hiring • @yurishkuro