Betting on Observability at Simplebet

Register and watch this talk now! https://o11yfest.org/attend

Adopting Observability can be an organizational challenge. At Simplebet, we introduced OpenTelemetry and o11y successfully, then published a case study on our experience with Lightstep. We will share our experience gaining traction so that you can learn how to introduce o11y at your company.

Simplebet introduced its first products into the market in 2020, which required a shift in organizational thinking around software deployments, operations, scale, etc. As we geared up for our first public launch, we needed to overcome the challenges of scaling a backend to support thousands of concurrent users at the onset. With the experience of team members who work on the Erlang Observability Working Group, we made a case for introducing distributed tracing with Lightstep as a tool to help us scale. After the first team adopted Otel and made use of distributed tracing, we mechanically implemented the same strategy to other teams in our organization. Through this instrumentation, we were able to isolate performance-critical operations, build evidence for long-standing issues, and change the way the team thought about our production system.

This talk will spend time talking about this experience but spend the majority of the time offering practical guidance for how a developer can gain traction with o11y and Otel at their organization.

Transcript

1. Hello! I am Dave Luci a I am a technology leader at Simplebet, active in the Elixir community, and new to the o11y movemen t You can find me at @davydog187 1
2. Betting on Observability at
3. What is Simplebet ? We turn every moment of a sports event into a betting opportunit y ◇ Odds as a servic e ◇ Powered by Machine-Learning ◇ Craft branded end-user experiences 3
4. Releasing products Our first public product release necessitated a change in organizational thinking 4
5. How do you build a project that will need to scale up immediately? 5
6. Why o11y? A successful adoption of o11y starts with understanding its value 6
7. Adopting o11y 7 1 3 5 6 4 2 Understand your need for observability Executive buy-in Instrument your product Communicate and demonstrate its value to your organization Choose a vendor Tend your o11y garden
8. Understanding your needs for observability O11y has many dimensions. The state of your o11y adoption is not binary. 1
9. A path towards o11y Logs APM Metrics 9 Tracing
10. “ Observability starts with gathering the data at the right level of abstraction, organized around the request path, such that you can slice and dice and group and look for patterns and cross-correlations in the requests — Charity Majors 10
11. Communicate and demonstrate o11y’s value to your organization 2
12. Benefits of tracing visualization MTT R Reduce the amount of time it takes to resolve a production issue N+1 Querie s Visually discover performance problems Measure, then decid e Squash vague arguments like “for performance reasons” 12
13. Executive buy-in Observability isn’t free. Translate organizational value into impact 3
14. o11y o11y has impact. Culture Quality 14
15. Choose a vendor Some try to do everything, others specializ e Pick what works for your organization. 4
17. Instrument your product You need to start somewhere 5
18. Where to start ◇ Familiarize yourself with Ote l ◇ Leverage framework instrumentatio n ◇ Start at the boundarie s Focus on instrumenting the areas of value and risk to your product. 18
19. Tend your o11y garden Observability needs maintenance to suit your needs, it is evergreen 6
20. Grow your o11 y Pull weeds, fertilize, plant. 20
21. Engage with the community ◇ Follow community leaders on social medi a ◇ Get involved with Ote l ◇ Be a lurker! 21
22. Thanks! Any questions ? You can find me at : ◇ @davydog18 7 ◇ dave@simplebet.io 22