Instrumentation: fly safe in production

Transcript

1. 1 Expedia Group Proprietary and Confidential Instrumentation: Fly safe in production Distributed Tracing NYC Meetup
2. 2 Expedia Group Proprietary and Confidential José Carlos Chávez • Software Engineer at Expedia • Zipkin core team and open source contributor @jcchavezs
3. 3 Expedia Group Proprietary and Confidential Instrumentation @jcchavezs
4. 4 Expedia Group Proprietary and Confidential It is the act of measurement properties of relevant events in the software operations. e.g. - request.sent (method, URI, body.size) - response.received (status_code, error) What is instrumentation? @jcchavezs
5. 5 Expedia Group Proprietary and Confidential What is instrumentation? https://twitter.com/autoletics/status/1163345131128401920 @jcchavezs
6. 6 Expedia Group Proprietary and Confidential Instrumentation is a step towards increasing observability, but it’s not observability in itself. One instruments and monitors a system as part of a broader strategy to make the system more observable. The goal of instrumentation @jcchavezs
7. 7 Expedia Group Proprietary and Confidential Both are the two faces of the same coin, what is instrumented for observation dictates the degree of controllability the system has. Observability vs Controllability @jcchavezs
8. 8 Expedia Group Proprietary and Confidential Stopover 1: Consuming instrumentation @jcchavezs
9. 9 Expedia Group Proprietary and Confidential Adding instrumentation should be part of the software building process (rather proactive than reactive) As a user, one should focus on: • Understand the use case to choose the right instrument • Avoid writing manual or custom instrumentation • Consider costs and privacy Consuming instrumentation @jcchavezs
10. 10 Expedia Group Proprietary and Confidential Stopover 2: Writing instrumentable code @jcchavezs
11. 11 Expedia Group Proprietary and Confidential Code that was designed with instrumentation as a first-class use case and that organically allows users to plug instrumentation on it. What is instrumentable code? @jcchavezs
12. 12 Expedia Group Proprietary and Confidential Classically, instrumentation should focus on latency source or error prone operations but there are more factors to take into consideration: • relevant events and properties • collecting more data doesn't mean it can be analyzed • operational costs What to allow for instrumentation @jcchavezs
13. 13 Expedia Group Proprietary and Confidential • Aim for well known patterns: ◦ decoration (example) ◦ event listeners (example) ◦ middlewares (example) ◦ interceptors (example) • Focus on events to describe operations (example) • When possible, use standard components (e.g. standard libraries, http client, sql driver) Rules of thumb for instrumentable code @jcchavezs
14. 14 Expedia Group Proprietary and Confidential • Don’t dictate the instrument • Don’t restrict measures • Don’t dispose of behavior Rules of thumb for accepting instrumentation @jcchavezs
15. 15 Expedia Group Proprietary and Confidential • For API designers, instrumentation should be a first class use case. • The fewer assumptions about type of instrumentation, the better. • The more relevant data instrumentation can collect, the better. In summary @jcchavezs
16. 16 Expedia Group Proprietary and Confidential Stopover 3: Writing instrumentation
17. 17 Expedia Group Proprietary and Confidential • Easy to use and to adopt. • Negligible impact in performance. • Flexible collection based on user needs. • Keeps the focus on relevant data. What is good instrumentation? @jcchavezs
18. 18 Expedia Group Proprietary and Confidential • Don’t ignore costs. • Don’t ignore privacy and security. • All components -not just critical services- should be instrumented. • Be in control to what users can do on top of it What instrumentation should consider? @jcchavezs
19. 19 Expedia Group Proprietary and Confidential As a user: choose the right (existing) instrumentation, help your future you by adding it. As a framework/library author: bear in mind instrumentation, use standard components when possible. As an instrumentation author: focus on UX, don’t allow introducing bugs on instrumentation. TL;DR @jcchavezs
20. 20 Expedia Group Proprietary and Confidential Gracias Q & A
21. 21 Expedia Group Proprietary and Confidential Read more about the topic on: • Is Operational Sympathy A (Good) Thing? - @autoletics • Go for industrial programming - @peterbourgon • Brave instrumentation - Zipkin authors Look around