Betting on Observability at Simplebet

Register and watch this talk now! https://o11yfest.org/attend

Adopting Observability can be an organizational challenge. At Simplebet, we introduced OpenTelemetry and o11y successfully, then published a case study on our experience with Lightstep. We will share our experience gaining traction so that you can learn how to introduce o11y at your company.

Simplebet introduced its first products into the market in 2020, which required a shift in organizational thinking around software deployments, operations, scale, etc. As we geared up for our first public launch, we needed to overcome the challenges of scaling a backend to support thousands of concurrent users at the onset. With the experience of team members who work on the Erlang Observability Working Group, we made a case for introducing distributed tracing with Lightstep as a tool to help us scale. After the first team adopted Otel and made use of distributed tracing, we mechanically implemented the same strategy to other teams in our organization. Through this instrumentation, we were able to isolate performance-critical operations, build evidence for long-standing issues, and change the way the team thought about our production system.

This talk will spend time talking about this experience but spend the majority of the time offering practical guidance for how a developer can gain traction with o11y and Otel at their organization.

Transcript

1. Hello!
   I am Dave Luci
   a
   
   I am a technology leader at Simplebet, active in the
   Elixir community, and new to the o11y movemen
   t
   
   You can find me at @davydog187
   1
   

   View Slide

2. Betting on Observability at
   

   View Slide

3. What is Simplebet
   ?
   
   We turn every moment of a sports
   event into a betting opportunit
   y
   
   ◇ Odds as a servic
   e
   
   ◇ Powered by Machine-Learning
   
   
   ◇ Craft branded end-user
   experiences
   3
   

   View Slide

4. Releasing products
   Our first public product release necessitated
   a change in organizational thinking
   4
   

   View Slide

5. How do you build a project that will need to scale up immediately?
   5
   

   View Slide

6. Why o11y?
   A successful adoption of o11y starts
   with understanding its value
   6
   

   View Slide

7. Adopting o11y
   7
   1 3 5
   6
   4
   2
   Understand your need
   for observability Executive buy-in Instrument your product
   Communicate and
   demonstrate its value to
   your organization
   Choose a vendor Tend your o11y garden
   

   View Slide

8. Understanding your needs for
   observability
   O11y has many dimensions. The state of your o11y adoption is not
   binary.
   1
   

   View Slide

9. A path towards o11y
   Logs APM Metrics
   9
   Tracing
   

   View Slide

10. “ Observability starts with gathering the data at the right level of abstraction, organized around the request path,
    such that you can slice and dice and group and look for patterns and cross-correlations in the requests
    
    
    — Charity Majors
    10
    

    View Slide

11. Communicate and demonstrate
    o11y’s value to your organization
    2
    

    View Slide

12. Benefits of tracing visualization
    MTT
    R
    
    Reduce the amount of
    time it takes to resolve a
    production issue
    N+1 Querie
    s
    
    Visually discover
    performance problems
    Measure, then decid
    e
    
    Squash vague
    arguments like “for
    performance reasons”
    12
    

    View Slide

13. Executive buy-in
    Observability isn’t free. Translate organizational value into impact
    3
    

    View Slide

14. o11y
    o11y has impact.
    Culture Quality
    14
    

    View Slide

15. Choose a vendor
    Some try to do everything, others specializ
    e
    
    Pick what works for your organization.
    4
    

    View Slide

16. View Slide

17. Instrument your product
    You need to start somewhere
    5
    

    View Slide

18. Where to start
    ◇ Familiarize yourself with Ote
    l
    
    ◇ Leverage framework instrumentatio
    n
    
    ◇ Start at the boundarie
    s
    
    Focus on instrumenting the areas of value and risk to your
    product.
    18
    

    View Slide

19. Tend your o11y garden
    Observability needs maintenance to suit your needs, it is evergreen
    6
    

    View Slide

20. Grow your
    o11
    y
    
    Pull weeds,
    fertilize,
    
    
    plant.
    20
    

    View Slide

21. Engage with the community
    ◇ Follow community leaders on social
    medi
    a
    
    ◇ Get involved with Ote
    l
    
    ◇ Be a lurker!
    21
    

    View Slide

22. Thanks!
    Any questions
    ?
    
    You can find me at
    :
    
    ◇ @davydog18
    7
    
    ◇ dave@simplebet.io
    22
    

    View Slide