Cloudy with a Chance of Insights: Demystifying Observability in the Cloud

You finally did it, you moved to the cloud! All infrastructure has been set up, your services are running, customer orders are flooding in... everything seems fine. But how can you know for sure that that is the case and that this does not change? And in case of problems, how to narrow them down to the source of error in the quickest way possible? The answer is, increased focus on the oh so often overlooked topic of observability.
In this talk you will learn best practices of observability in the cloud which will help you guarantee correct functioning of your cloud enterprises technically as well as from a business standpoint. Learn how to collect telemetry data, analyze it and react accordingly. We might even see some hands-on action!
Uncover the fun side of cloud observability with us and don't forget: great infrastructure comes with great responsibility (of observing its correct functioning)!

Transcript

2. 2 ?
3. “CLOUDY, WITH A CHANCE OF INSIGHTS” Demystifying Observability in the Cloud PHILIPP SEIFERT-KEHRER 4 July 2024
4. 4 WHAT ABOUT ME? • Developer for 7+ years • @ Posedio since 2022 • Blood type AB- • Love cats, crocheting & crocheting cats • Monoliths: hehe • Shared libs: not hehe Philipp Seifert-Kehrer
5. ROADMAP 1. Debugging like it’s 1999 2. Observability Xs and Os 3. Tracing 101 4.Introducing OpenTelemetry 5. Where benefits?
6. 6 SO, WHAT TO EXPECT FROM THIS TALK? (Re)Gain confidence to deploy…
7. DEBUGGING LIKE IT’S 1999 1
8. 8 CLASSICAL DEBUGGING PHD in domain knowledge Sherlock gene
9. 9 LONGEST EMPLOYEE = BEST DEBUGGER? I haven’t moved away from here in 15 years
10. 10 READING THE TEA LEAVES
11. 11 MONITORING = BYOC Bring Your Own Context
12. 12 BUT, BUT, BUT RUNBOOKS • No documentation > wrong documentation
13. 13 HELLO CLOUD
14. 14 BUT WHY MONITORING NOT CUTTIN’ IT? Built upon assumptions Only useful for known unknowns Purely reactive Linear increase of cost Uptime is so 2001
15. 2 OBSERVABILITY XS AND OS
16. 16 WHAT ARE WE TALKIN’ ABOUT? “How well internal states of a system can be inferred from knowledge of its external outputs”
17. 17 PLATO WAS RIGHT
18. 18 BUT FOR SOFTWARE? Understand inner workings of application Understand any system state Understand only by observing/external tools Understand without need to ship new code
19. 19 GANG OF 3 Centralized Logging Metrics & Dashboards Distributed Tracing
20. 20 LET THEM WITHOUT SIN CAST THE FIRST STONE Every transaction No issues with cardinality Proportional growth of cost Centralized Logging ✅ ✅ 🙅 Metrics & Dashboards ✅ 🙅 ✅ Distribruted Tracing 🙅 ✅ ✅
21. 21 GANG OF 3 GANG OF 3 Centralized Logging Metrics & Dashboards Distributed Tracing
22. 22 GIMME SOME EXPLORABILITY
23. TRACING 3
24. 24 SERVICE GRAPH
25. 25 TRACING EXAMPLE
26. 26 ONE TRACE TO RULE ‘EM ALL
27. 27 ANATOMY OF A SPAN
28. 28 EVENTS
29. 29 IT’S ALL ABOUT THE CONTEXT Context propagation Baggage Marshall Context matters (… even after the 14th blunt)
30. OPENTELEMETRY 4
31. 31 OTEL OFFERING … that you can’t refuse Open specification Instrumentation APIs & SDKs Semantic conventions Agent to collect telemetry Optimized protocol OTLP
32. 32 COLLECTOR
33. 33 SEMANTIC CONVENTIONS
34. 34 INSTRUMENTATION Black vs White box Application vs System Distributed vs Monolith
35. 35 DO’S AND DON’TS Do Don’t Feature flags Be aware of PII Verbocity level Start off with automatic instrumentation Ignore potential resource overhead Non-standard format Too high default resolution
36. WHERE BENEFITS 5
37. 37 WHY ALL OF THIS? Less interpretation = less discussion Foster proactive debugging Reliability Good for application, organization & employee (mental) health No more manual, mental joins Learning opportunity instead of learning prerequisite Link anomalies to distinct version
38. 38 A NEW NON-FUNCTIONAL REQUIREMENT Maintainability Availability Scalability Observability in the cloud
39. 39 BUT IT’S SO MUCH OVERHEAD, RIGHT?
40. 40 SAMPLING
41. 41 WHAT IF I JUST KEEP LOGGING EVERYTHING?
42. 42 WRAPPING IT UP? Maintainability Availability Scalability Sustainability Better understanding of architecture More accessible debugging Not all can be fixed… and also should not! Benefits without too much effort possible No panacea
43. 43 Failure is inevitable … so when the shit goes down, you better be ready
44. THANK YOU! POSEDIO GMBH +43 660 9154419 contact@posedio.com Weyringergasse 1-3, 1040 Wien, Millenium Park 4, 6980 Lustenau www.posedio.com Dilly dilly!