Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instrumentation: fly safe in production

Instrumentation: fly safe in production

E8ab578dea21d57898977701c583152a?s=128

José Carlos Chávez

September 05, 2019
Tweet

Transcript

  1. 1 Expedia Group Proprietary and Confidential Instrumentation: Fly safe in

    production Distributed Tracing NYC Meetup
  2. 2 Expedia Group Proprietary and Confidential José Carlos Chávez •

    Software Engineer at Expedia • Zipkin core team and open source contributor @jcchavezs
  3. 3 Expedia Group Proprietary and Confidential Instrumentation @jcchavezs

  4. 4 Expedia Group Proprietary and Confidential It is the act

    of measurement properties of relevant events in the software operations. e.g. - request.sent (method, URI, body.size) - response.received (status_code, error) What is instrumentation? @jcchavezs
  5. 5 Expedia Group Proprietary and Confidential What is instrumentation? https://twitter.com/autoletics/status/1163345131128401920

    @jcchavezs
  6. 6 Expedia Group Proprietary and Confidential Instrumentation is a step

    towards increasing observability, but it’s not observability in itself. One instruments and monitors a system as part of a broader strategy to make the system more observable. The goal of instrumentation @jcchavezs
  7. 7 Expedia Group Proprietary and Confidential Both are the two

    faces of the same coin, what is instrumented for observation dictates the degree of controllability the system has. Observability vs Controllability @jcchavezs
  8. 8 Expedia Group Proprietary and Confidential Stopover 1: Consuming instrumentation

    @jcchavezs
  9. 9 Expedia Group Proprietary and Confidential Adding instrumentation should be

    part of the software building process (rather proactive than reactive) As a user, one should focus on: • Understand the use case to choose the right instrument • Avoid writing manual or custom instrumentation • Consider costs and privacy Consuming instrumentation @jcchavezs
  10. 10 Expedia Group Proprietary and Confidential Stopover 2: Writing instrumentable

    code @jcchavezs
  11. 11 Expedia Group Proprietary and Confidential Code that was designed

    with instrumentation as a first-class use case and that organically allows users to plug instrumentation on it. What is instrumentable code? @jcchavezs
  12. 12 Expedia Group Proprietary and Confidential Classically, instrumentation should focus

    on latency source or error prone operations but there are more factors to take into consideration: • relevant events and properties • collecting more data doesn't mean it can be analyzed • operational costs What to allow for instrumentation @jcchavezs
  13. 13 Expedia Group Proprietary and Confidential • Aim for well

    known patterns: ◦ decoration (example) ◦ event listeners (example) ◦ middlewares (example) ◦ interceptors (example) • Focus on events to describe operations (example) • When possible, use standard components (e.g. standard libraries, http client, sql driver) Rules of thumb for instrumentable code @jcchavezs
  14. 14 Expedia Group Proprietary and Confidential • Don’t dictate the

    instrument • Don’t restrict measures • Don’t dispose of behavior Rules of thumb for accepting instrumentation @jcchavezs
  15. 15 Expedia Group Proprietary and Confidential • For API designers,

    instrumentation should be a first class use case. • The fewer assumptions about type of instrumentation, the better. • The more relevant data instrumentation can collect, the better. In summary @jcchavezs
  16. 16 Expedia Group Proprietary and Confidential Stopover 3: Writing instrumentation

  17. 17 Expedia Group Proprietary and Confidential • Easy to use

    and to adopt. • Negligible impact in performance. • Flexible collection based on user needs. • Keeps the focus on relevant data. What is good instrumentation? @jcchavezs
  18. 18 Expedia Group Proprietary and Confidential • Don’t ignore costs.

    • Don’t ignore privacy and security. • All components -not just critical services- should be instrumented. • Be in control to what users can do on top of it What instrumentation should consider? @jcchavezs
  19. 19 Expedia Group Proprietary and Confidential As a user: choose

    the right (existing) instrumentation, help your future you by adding it. As a framework/library author: bear in mind instrumentation, use standard components when possible. As an instrumentation author: focus on UX, don’t allow introducing bugs on instrumentation. TL;DR @jcchavezs
  20. 20 Expedia Group Proprietary and Confidential Gracias Q & A

  21. 21 Expedia Group Proprietary and Confidential Read more about the

    topic on: • Is Operational Sympathy A (Good) Thing? - @autoletics • Go for industrial programming - @peterbourgon • Brave instrumentation - Zipkin authors Look around