Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Distributed Tracing at UBER Scale
Search
Yuri Shkuro
May 24, 2017
Programming
1
300
Distributed Tracing at UBER Scale
Presented at Monitorama-PDF 2017.
Video:
https://vimeo.com/221070602
Yuri Shkuro
May 24, 2017
Tweet
Share
More Decks by Yuri Shkuro
See All by Yuri Shkuro
TEMPLE: Six Pillars of Telemetry
yurishkuro
0
400
Schema-first application telemetry
yurishkuro
0
220
CNCF Webinar Series - Introducing Jaeger 1.0
yurishkuro
1
310
Would You Like Some Tracing With Your Monitoring?
yurishkuro
0
340
From zero to distributed traces: an OpenTracing tutorial
yurishkuro
1
800
Other Decks in Programming
See All in Programming
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
170
GitHub Copilotのススメ
marcy731
1
200
SIMD Parallel Programming with the Vector API
josepaumard
0
180
Git Rebase
bkuhlmann
11
1.6k
[技育CAMPアカデミア]アイディアを形に!【超入門】スマホアプリ開発〜リリースまでの流れをご紹介
teamlab
PRO
0
370
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
Kotlin Multiplatform at Stable and Beyond (Android Makers 2024)
zsmb
0
260
VSCodeでのDatabricks開発もお勧めしたい/I would also recommend Databricks development with VSCode.
kazumain
0
250
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
360
#phpcon_odawara オープン・クローズドなテストフィクスチャを求めて / open closed test fixtures
77web
3
230
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
2
190
Tailwind CSSを本気でカスタマイズする方法
fsubal
13
5.3k
Featured
See All Featured
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
What the flash - Photography Introduction
edds
64
11k
Typedesign – Prime Four
hannesfritz
36
2.1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
244
20k
Mobile First: as difficult as doing things right
swwweet
216
8.6k
Making Projects Easy
brettharned
108
5.5k
10 Git Anti Patterns You Should be Aware of
lemiorhan
648
58k
Debugging Ruby Performance
tmm1
70
11k
YesSQL, Process and Tooling at Scale
rocio
164
13k
Navigating Team Friction
lara
178
13k
Building Effective Engineering Teams - LeadDev
addyosmani
28
1.8k
Web development in the modern age
philhawksworth
202
10k
Transcript
Distributed Tracing at UBER Scale Crea7ng a treasure map for
your monitoring data Yuri Shkuro, UBER Technologies
ABOUT ME • SoAware Engineer on the Observability team in
NYC • Working on the open source distributed tracing system Jaeger • Co-founded the OpenTracing project • Banking industry survivor • Github: yurishkuro • TwiLer: @yurishkuro
Would You Like Some Tracing with Your Monitoring? What does
it take to roll it out?
Why Distributed Tracing • Distributed transac7on monitoring • Performance /
latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
JAEGER, Distributed Tracing • Open Source • OpenTracing inside •
In ac7ve development • PRs are welcome • Zipkin compa7ble • github.com/uber/jaeger
Who Thinks Tracing is Awesome?
None
None
Why Doesn’t Everyone Do Tracing?
Tracing Instrumenta7on is HARD EXPENSIVE BORING
Instrumenta7on • Metrics and logging are not new • Tracing
is both new and harder
Context Propaga7on A B C D E {context} {context} {context}
{context} Unique ID → {context} Edge service
Headers: . . . Trace ID . . . Instrumentation
APPLICATION / MICROSERVICE Handler Context [Span] Client Context [Span] Inbound HTTP Request Instrumentation Headers: . . . Trace ID . . . Outbound HTTP Request Context Propaga7on
In-Process Context Propaga7on Implicit, via Thread-Locals but: thread pools, futures
Explicit
It’s Also the Frameworks • Go: stdlib, gorilla, … •
Java: jaxrs2, okhLp, ApacheHLpClient, … • Python: Flask, Django, Tornado, urllib2, … • Node.js – who knows…
OpenTracing to the Rescue
No Help With In-Process Propaga7on • Must be done manually
• UBER has 2000-3000 microservices • Resources of the tracing team are limited • Developers must instrument their code!
BITE MAKE ME! How do we mobilize the org?
Traveling Salesman Problem 2017 edi7on
They Must Want Your Product or S7cks and Carrots
Recap: Why Distributed Tracing • Distributed transac7on monitoring • Performance
/ latency op7miza7on • Root cause analysis • Service dependency analysis • Distributed context propaga7on (“baggage”)
Service Dependency Analysis • Explain to us what we just
built • Who are my dependencies • Workflow analysis • Where is all this traffic coming from? • Service 7ers
Baggage • Tenancy, test or produc7on – Set at the top
– Used at the storage layer, prod or test DB • Authen7ca7on tokens – Signed user or service iden7ty – Checked at mul7ple levels
S7cks and Carrots • Get other teams build features on
top – Performance team – Capacity & cost accoun7ng – Baggage • More carrots • Eventually they become s7cks (peer pressure)
Each Organiza7on is Different Find what works best
How to Measure Adop7on? Measure everything
Does Service X Report Traces? • Daily aggrega7on job •
Auto-book 7ckets • Build a dashboard • Pass/Fail: too easy to pass
Trace Quality Score • Inspect traces – See a caller, but
no spans • Join with other data – Rou7ng logs • Auto-book 7ckets (carefully, not for everyone) – With detailed report
Trace Quality Metrics by Service
Thank You • Jaeger – hLps://github.com/uber/jaeger – Blog: Evolving Distributed
Tracing at UBER – Blog: Take OpenTracing for a HotROD Ride • OpenTracing: hLp://opentracing.io/ • We are hiring • @yurishkuro