Challenges in adopting a culture of observability

Let's talk humans, organisations and their relationship with tech. Not everyone is ready for observability!

Transcript

1. Challenges in adopting a culture of observability Let's talk humans, organisations and their relationship with tech. Not everyone is ready for observability! @adrien-f
2. @adrien-f • Somehow became a SRE in e-commerce • Bordeaux, France •
3. https://www.welcometothejungle.co/en/companies/manomano
4. A story of tech and people
5. Adopt a product based approach Simply because that's what we're trying to achieve here. • Who are the real users of this? • Who's going to pay ? • Why are we even doing this ? • Find the value stream
6. Why are we even doing this ? • Is there existing tooling? Is it not good enough ? • Why do we want to change ? We want to be able to explain to others why we want this. What it will bring to the company.
7. Find your users • For whom am I doing this ? • What are their pain points with existing tooling ? • What value can I bring to them ?
8. Challenges 1. Why are we doing this ? 2. For whom ? 3. How do build something usable ?
9. For Operations Apps Services Hardware / OS Network What they care about:
10. The Network Operations Center effect • Big wall of screens, dashboards all green, waiting for something to happen. • Is it really observability ? Can't we do better ? • Anticipate, don't just react • Observability will change their daily life • One of the first lever for creating a new culture • And getting feedback
11. For Engineering Apps Services Hardware / OS Network What they care about:
12. The Rockstar Team opportunity • New project from scratch • Out of processes • Favored by the business Opportunity: • They'll need tools to guarantee the success of the project • Sit down with the developers, show them what you can do in an afternoon • Business owners and Devs will crave for this • when they'll go back to their original team
13. Own the libraries • You’ll need to provide engineering with a set of libraries, APIs and tools to get into your system • Be a part of the development of those libraries • Show off the result, demo days, pair program. • Create evangelists • This will also help propagate what your product is about, what you’re trying to achieve
14. For RelEng / QA Apps Services Hardware / OS Network Users What they care about:
15. Testing vs O11Y • How long does it take to run your full test suites ? • vs How long does it take to deploy to 0.1% of your users ? • Ship often, ship fast, rollback even more This can help creating a culture of experimenting in production
16. For Business Apps Services Hardware / OS Network Users What they care about:
17. Real User Monitoring • Not an easy thing • What are our users doing with our products ? • Helps feed the feedback loop, encourage product oriented teams
18. It’s all about the execution
19. It seems easy, it’s just tech • We install Prometheus, a few exporters • Elasticsearch and FluentBit • Grafana, a few dashboards, and done But it’s not ! • Or we spend $$$$ on SaaS tools and let someone else deal with it
20. It’s not just tech, it’s also about changing your company • For the first time in the history of your company, you’ll finally know what’s truly wrong with your IT system ◦ Is everyone ready for this ? ◦ Are people aware what it means for them ? ◦ People could get hostile • People not responsible for what they ship will soon become the center of attention ◦ Is the organisation centered around this shared responsibility model ?
21. Creating a culture • Need to tell people about shortcomings of the solution: cardinality, retention • Think about integrations, with old tools, new tools, how can this help anyone • People need to trust your product, be transparent about outages, SLOs, … ◦ Beware if you lose that trust… • Facilitate onboarding and configuration ◦ Prometheus rules auto-gen and auto-deploy ◦ Guidebooks with how to create the perfect metric
22. Create a brand, a product • Internal meetups with demos are a great way to excite people ! ◦ Find a recurring pain point, quickly instrument it with metrics, logs, traces and a few dashboards ◦ Live code an instrumentation • Get some Swag ◦ Stickers, hoodies, print out some posters https://stores.kotisdesign.com/grafana/posters
23. If I had to redo it all again 1. Get top management on this 2. Provide a rock solid set of libraries and maintain them 3. Onboard everyone from the beginning 4. The tech is not important, humans are 5. Define rules, not too strict but not too lax 6. Iterate, find opportunities and reach for the goal
24. Cheers ! Any questions ?