1 Expedia Group Proprietary and Confidential Instrumentation: Fly safe in production Distributed Tracing NYC Meetup

2 Expedia Group Proprietary and Confidential José Carlos Chávez ● Software Engineer at Expedia ● Zipkin core team and open source contributor @jcchavezs

3 Expedia Group Proprietary and Confidential Instrumentation @jcchavezs

4 Expedia Group Proprietary and Confidential It is the act of measurement properties of relevant events in the software operations. e.g. - request.sent (method, URI, body.size) - response.received (status_code, error) What is instrumentation? @jcchavezs

5 Expedia Group Proprietary and Confidential What is instrumentation? https://twitter.com/autoletics/status/1163345131128401920 @jcchavezs

6 Expedia Group Proprietary and Confidential Instrumentation is a step towards increasing observability, but it’s not observability in itself. One instruments and monitors a system as part of a broader strategy to make the system more observable. The goal of instrumentation @jcchavezs

7 Expedia Group Proprietary and Confidential Both are the two faces of the same coin, what is instrumented for observation dictates the degree of controllability the system has. Observability vs Controllability @jcchavezs

8 Expedia Group Proprietary and Confidential Stopover 1: Consuming instrumentation @jcchavezs

9 Expedia Group Proprietary and Confidential Adding instrumentation should be part of the software building process (rather proactive than reactive) As a user, one should focus on: ● Understand the use case to choose the right instrument ● Avoid writing manual or custom instrumentation ● Consider costs and privacy Consuming instrumentation @jcchavezs

10 Expedia Group Proprietary and Confidential Stopover 2: Writing instrumentable code @jcchavezs

11 Expedia Group Proprietary and Confidential Code that was designed with instrumentation as a first-class use case and that organically allows users to plug instrumentation on it. What is instrumentable code? @jcchavezs

12 Expedia Group Proprietary and Confidential Classically, instrumentation should focus on latency source or error prone operations but there are more factors to take into consideration: ● relevant events and properties ● collecting more data doesn't mean it can be analyzed ● operational costs What to allow for instrumentation @jcchavezs

13 Expedia Group Proprietary and Confidential ● Aim for well known patterns: ○ decoration (example) ○ event listeners (example) ○ middlewares (example) ○ interceptors (example) ● Focus on events to describe operations (example) ● When possible, use standard components (e.g. standard libraries, http client, sql driver) Rules of thumb for instrumentable code @jcchavezs

14 Expedia Group Proprietary and Confidential ● Don’t dictate the instrument ● Don’t restrict measures ● Don’t dispose of behavior Rules of thumb for accepting instrumentation @jcchavezs

15 Expedia Group Proprietary and Confidential ● For API designers, instrumentation should be a first class use case. ● The fewer assumptions about type of instrumentation, the better. ● The more relevant data instrumentation can collect, the better. In summary @jcchavezs

16 Expedia Group Proprietary and Confidential Stopover 3: Writing instrumentation

17 Expedia Group Proprietary and Confidential ● Easy to use and to adopt. ● Negligible impact in performance. ● Flexible collection based on user needs. ● Keeps the focus on relevant data. What is good instrumentation? @jcchavezs

18 Expedia Group Proprietary and Confidential ● Don’t ignore costs. ● Don’t ignore privacy and security. ● All components -not just critical services- should be instrumented. ● Be in control to what users can do on top of it What instrumentation should consider? @jcchavezs

19 Expedia Group Proprietary and Confidential As a user: choose the right (existing) instrumentation, help your future you by adding it. As a framework/library author: bear in mind instrumentation, use standard components when possible. As an instrumentation author: focus on UX, don’t allow introducing bugs on instrumentation. TL;DR @jcchavezs

20 Expedia Group Proprietary and Confidential Gracias Q & A

21 Expedia Group Proprietary and Confidential Read more about the topic on: ● Is Operational Sympathy A (Good) Thing? - @autoletics ● Go for industrial programming - @peterbourgon ● Brave instrumentation - Zipkin authors Look around