Instrumentation: fly safe in production

Transcript

1. 1
   Expedia Group Proprietary and Confidential
   Instrumentation: Fly safe
   in production
   Distributed Tracing NYC Meetup
   

   View Slide

2. 2
   Expedia Group Proprietary and Confidential
   José Carlos
   Chávez
   ● Software Engineer
   at Expedia
   ● Zipkin core team
   and open source
   contributor
   @jcchavezs
   

   View Slide

3. 3
   Expedia Group Proprietary and Confidential
   Instrumentation
   @jcchavezs
   

   View Slide

4. 4
   Expedia Group Proprietary and Confidential
   It is the act of measurement properties of relevant events in
   the software operations.
   e.g.
   - request.sent (method, URI, body.size)
   - response.received (status_code, error)
   What is instrumentation?
   @jcchavezs
   

   View Slide

5. 5
   Expedia Group Proprietary and Confidential
   What is instrumentation?
   https://twitter.com/autoletics/status/1163345131128401920
   @jcchavezs
   

   View Slide

6. 6
   Expedia Group Proprietary and Confidential
   Instrumentation is a step towards increasing observability,
   but it’s not observability in itself. One instruments and
   monitors a system as part of a broader strategy to make the
   system more observable.
   The goal of instrumentation
   @jcchavezs
   

   View Slide

7. 7
   Expedia Group Proprietary and Confidential
   Both are the two faces of the same coin, what is
   instrumented for observation dictates the degree of
   controllability the system has.
   Observability vs Controllability
   @jcchavezs
   

   View Slide

8. 8
   Expedia Group Proprietary and Confidential
   Stopover 1:
   Consuming instrumentation
   @jcchavezs
   

   View Slide

9. 9
   Expedia Group Proprietary and Confidential
   Adding instrumentation should be part of the software
   building process (rather proactive than reactive)
   As a user, one should focus on:
   ● Understand the use case to choose the right instrument
   ● Avoid writing manual or custom instrumentation
   ● Consider costs and privacy
   Consuming instrumentation
   @jcchavezs
   

   View Slide

10. 10
    Expedia Group Proprietary and Confidential
    Stopover 2:
    Writing instrumentable code
    @jcchavezs
    

    View Slide

11. 11
    Expedia Group Proprietary and Confidential
    Code that was designed with instrumentation as a first-class
    use case and that organically allows users to plug
    instrumentation on it.
    What is instrumentable code?
    @jcchavezs
    

    View Slide

12. 12
    Expedia Group Proprietary and Confidential
    Classically, instrumentation should focus on latency source
    or error prone operations but there are more factors to take
    into consideration:
    ● relevant events and properties
    ● collecting more data doesn't mean it can be analyzed
    ● operational costs
    What to allow for instrumentation
    @jcchavezs
    

    View Slide

13. 13
    Expedia Group Proprietary and Confidential
    ● Aim for well known patterns:
    ○ decoration (example)
    ○ event listeners (example)
    ○ middlewares (example)
    ○ interceptors (example)
    ● Focus on events to describe operations (example)
    ● When possible, use standard components (e.g. standard
    libraries, http client, sql driver)
    Rules of thumb for instrumentable code
    @jcchavezs
    

    View Slide

14. 14
    Expedia Group Proprietary and Confidential
    ● Don’t dictate the instrument
    ● Don’t restrict measures
    ● Don’t dispose of behavior
    Rules of thumb for accepting instrumentation
    @jcchavezs
    

    View Slide

15. 15
    Expedia Group Proprietary and Confidential
    ● For API designers, instrumentation should be a first class
    use case.
    ● The fewer assumptions about type of instrumentation, the
    better.
    ● The more relevant data instrumentation can collect, the
    better.
    In summary
    @jcchavezs
    

    View Slide

16. 16
    Expedia Group Proprietary and Confidential
    Stopover 3:
    Writing instrumentation
    

    View Slide

17. 17
    Expedia Group Proprietary and Confidential
    ● Easy to use and to adopt.
    ● Negligible impact in performance.
    ● Flexible collection based on user needs.
    ● Keeps the focus on relevant data.
    What is good instrumentation?
    @jcchavezs
    

    View Slide

18. 18
    Expedia Group Proprietary and Confidential
    ● Don’t ignore costs.
    ● Don’t ignore privacy and security.
    ● All components -not just critical services- should be
    instrumented.
    ● Be in control to what users can do on top of it
    What instrumentation should consider?
    @jcchavezs
    

    View Slide

19. 19
    Expedia Group Proprietary and Confidential
    As a user: choose the right (existing) instrumentation, help
    your future you by adding it.
    As a framework/library author: bear in mind
    instrumentation, use standard components when possible.
    As an instrumentation author: focus on UX, don’t allow
    introducing bugs on instrumentation.
    TL;DR
    @jcchavezs
    

    View Slide

20. 20
    Expedia Group Proprietary and Confidential
    Gracias
    Q & A
    

    View Slide

21. 21
    Expedia Group Proprietary and Confidential
    Read more about the topic on:
    ● Is Operational Sympathy A (Good) Thing? - @autoletics
    ● Go for industrial programming - @peterbourgon
    ● Brave instrumentation - Zipkin authors
    Look around
    

    View Slide