Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed Tracing at Netflix

A3668e66eb7b8980ac91daaa4e9fe691?s=47 Nitesh Kant
July 10, 2015
Technology
5
3.3k

Distributed Tracing at Netflix

This was presented in a private workshop on distributed tracing and it provides details about the current state of distributed tracing at netflix (July 2015)

A3668e66eb7b8980ac91daaa4e9fe691?s=128

Nitesh Kant

July 10, 2015
Tweet

More Decks by Nitesh Kant

See All by Nitesh Kant
A3668e66eb7b8980ac91daaa4e9fe691?s=48 niteshkant
0
1k
A3668e66eb7b8980ac91daaa4e9fe691?s=48 niteshkant
1
160
A3668e66eb7b8980ac91daaa4e9fe691?s=48 niteshkant
10
1.9k
A3668e66eb7b8980ac91daaa4e9fe691?s=48 niteshkant
1
880
A3668e66eb7b8980ac91daaa4e9fe691?s=48 niteshkant
2
110

Other Decks in Technology

See All in Technology
B4dde7c05044041075d7e55d705e9cc3?s=48 youmitsu
1
2.5k
A645e7c3a6635ead8fa323c583769591?s=48 hololab
0
630
4e523d3c4d6047382d22f61ebe53c554?s=48 ore88ore
0
1.5k
49f49940f0831a426745c028684bcdad?s=48 hatena
1
3k
4f43f9f4dc4cc2897841877e8f7b032f?s=48 cielan
1
290
A45ab154d523736344fce693977abf32?s=48 raksulrecruiting
0
110
6d161fe47f36ac295189c3b6a0db2a9a?s=48 masahirokawahara
0
290
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
PRO
0
440
49f49940f0831a426745c028684bcdad?s=48 hatena
0
3.3k
E6d26a51159a7863cac28e9d12ccd389?s=48 rmakiyama
4
4.6k
44a210d1299a727e9d601a47dfedeabf?s=48 mot_techtalk
0
120
9b63847a4c413a2604e1d64e5d595dab?s=48 humank
0
140

Featured

See All Featured
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
69
3.2k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
91
8.9k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
45
2.5k
7559f6cff1f5efc2d210965febd4d71c?s=48 bermonpainter
348
27k
0fe2973fa8d27d96547e43322341183a?s=48 swwweet
209
7.2k
79acae287842acf29084005533a7fb3b?s=48 hursman
111
9.5k
C157b16234f1e75e8eac3698c1d4414a?s=48 ddemaree
273
32k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
122
8.2k
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
298
39k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
396
61k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
171
7.7k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
498
130k

Transcript

  1. Nitesh Kant - @NiteshKant - Software Engineer, Cloud Platform, Netflix

    Distributed Tracing @Netflix a.k.a Salp July 2015

  2. Nitesh Kant Who Am I? ❖ Engineer, Cloud Platform, Netflix.

    ❖ Leading Reactive IPC @Netflix ❖ Core contributor, RxNetty* ❖ Introduced distributed tracing inside Netflix (!Yet OSS) * https://github.com/ReactiveX/RxNetty @NiteshKant

  3. 62 Million Subscribers world-wide 10 Billion hours / month http://ir.netflix.com/

  4. AWS Availability Zone AWS Availability Zone AWS Availability Zone

  5. Availability Zone Zuul (Routing layer) Edge Services Various Microservices

  6. The problem

  7. The food chain A simplistic hypothetical call chain. Zuul API

    Recommendation service Subscriber service Cache C*

  8. The food chain A simplistic hypothetical call chain. Zuul API

    Recommendation service Subscriber service Cache C* What services does a request touch?

  9. The food chain A simplistic hypothetical call chain. Zuul API

    Recommendation service Subscriber service Cache C* Who Am I dependent on? What services does a request touch?

  10. The food chain A simplistic hypothetical call chain. Zuul API

    Recommendation service Subscriber service Cache C* Who Am I dependent on? Who Am I dependent on? Who calls me? What services does a request touch?

  11. The food chain A simplistic hypothetical call chain. Zuul API

    Recommendation service Subscriber service Cache C* Who Am I dependent on? Who Am I dependent on? Who calls me? Who overwhelmed me now? What services does a request touch?

  12. Back in the days

  13. Request Trace In-memory storage of data till request completion. Zuul

    API Recommendation service Subscriber service Cache C* A simplistic hypothetical call chain.

  14. Request Trace In-memory storage of data till request completion. Zuul

    API Recommendation service Subscriber service Cache C* A simplistic hypothetical call chain.

  15. Request Trace In-memory storage of data till request completion. Zuul

    API Recommendation service Subscriber service Cache C* A simplistic hypothetical call chain.

  16. Request Trace In-memory storage of data till request completion. Zuul

    API Recommendation service Subscriber service Cache C* A simplistic hypothetical call chain.

  17. Request Trace In-memory storage of data till request completion. Inflexible

    data model (averse to custom data) Zuul API Recommendation service Subscriber service Cache C* A simplistic hypothetical call chain.

  18. Today* * July 2015

  19. Salp (Transparent Microservices Interactions)

  20. Zuul API Recommendation service Subscriber service Cache C* A simplistic

    hypothetical call chain.

  21. User Request Tracing Filter Zuul

  22. User Request Tracing Filter Local tracing decision Zuul

  23. User Request Tracing Filter Dynamic Config Service Get sampling rate

    Zuul

  24. User Request Tracing Filter Initialize local tracing context, when traced

    Zuul

  25. User Request Tracing Filter IPC instrumentation filter (Server) Zuul

  26. User Request Tracing Filter IPC instrumentation filter (Server) Emit server

    receive annotation Zuul

  27. User Request Tracing Filter IPC instrumentation filter (Server) Async publish

    to Suro* Logging Filter * https://github.com/Netflix/suro Zuul

  28. User Request Tracing Filter IPC instrumentation filter (Server) Zuul IPC

    instrumentation filter (Client) Emit client send annotation Dependency request

  29. Tracing Filter IPC instrumentation filter (Server) Zuul IPC instrumentation filter

    (Client) Emit client receive annotation Dependency response

  30. IPC instrumentation filter (Server) Zuul IPC instrumentation filter (Client) Emit

    server send annotation Dependency response Response

  31. IPC instrumentation filter (Server) Zuul IPC instrumentation filter (Client) Emit

    server send annotation Dependency response Response Any other service

  32. Tracing Context Propagation.

  33. In Process Within thread boundaries Thread local

  34. In Process Within thread boundaries Thread local Across thread boundaries

    Thread “Variable”. Copied on thread boundaries.

  35. Across processes A simplistic hypothetical call chain. Zuul API Recommendation

    Subscriber service Cache C*

  36. Across processes A simplistic hypothetical call chain. Zuul API Recommendation

    Subscriber service Cache C* ❖ Proprietary generic over the network “context propagation” library. ❖ Only HTTP traffic, context propagated as headers. ❖ On receipt, realize back to thread local.

  37. Sampling

  38. User Request Tracing Filter Dynamic Config Service Get sampling rate

    Zuul

  39. User Request Tracing Filter Dynamic Config Service Get sampling rate

    Zuul ❖ Simple sample rate (% of requests) stored in the configuration service. ❖ Granularity: Per application. ❖ Dynamic: Checked per request.

  40. Targeted tracing

  41. User Request Tracing Filter Dynamic Config Service Get sampling rate

    Zuul

  42. User Request Tracing Filter Dynamic Config Service Get sampling rate

    Zuul OR Dynamic criterion on request headers, URI, etc.
  43. None
  44. None
  45. None
  46. None
  47. None

  48. Data Publishing (via Suro) Tracing Filter IPC instrumentation filter (Server)

    Async publish to Suro* Logging Filter * https://github.com/Netflix/suro Zuul
  49. None
  50. None

  51. Data Format Custom K-V pair over Thrift

  52. Data Format Custom K-V pair over Thrift JSON JSON

  53. Publishing Rate Annotations per second. Daily (for a week)

  54. Publishing Rate Bytes per second. Daily (for a week)

  55. Visualizations

  56. Request traces Source: ElasticSearch. Granularity: Per request.

  57. Flow graphs Source: Druid. Granularity: Aggregated over per cluster.

  58. Flow graphs Source: Druid. Granularity: Aggregated over per cluster. Call

    volume & HTTP response code distribution per edge.

  59. Another visualization Source: API. Granularity: Aggregated. http://techblog.netflix.com/2015/02/a-microscope-on-microservices.html

  60. Tech landscape (Today*) Homogeneous architecture. Apps in critical path (customer

    request) are all JVM based.* HTTP/1.1 based services. * July 2015 * Node.js front-end app but IPC calls via JVM sidecar.

  61. Tech landscape (Future) Heterogenous architecture. Polyglot environment. Full duplex communication.

  62. Thank You! @NiteshKant July 2015