Observability: Superpowers for Developers

Who wants to spend time dreaming about the ability to leap tall buildings with a single bound, when we can recast stories we live day to day as powers of our own... and improve our own lives in a practical way? When observability is folded into the development process itself, it represents the potential for a beautifully virtuous cycle: production stops being just where our development code runs into issues, and it becomes where part of our development process *lives*.

Transcript

1. observability: Superpowers for Developers @cyen @honeycombio
2. EXPOSITION SETTING THE SCENE
3. DEV Christine
4. DEV WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT
5. DEV OPS WRITE → TEST → COMMIT → RELEASE → DEBUG → FIX
6. "Works on my machine" DEV "The only good diff is a red diff" OPS
7. —Subbu Allamaraju, Expedia, Feb 2019
 https://m.subbu.org/incidents-trends-from-the-trenches-e2f8497d52ed "Observation 1: Change is the most common trigger"
8. APP API GATEWAY USER MGMT BILLING WEB UI PARTNER MGMT PAYMENTS INTERNAL WEB UI TXN MGMT NOTIFICATION SYSTEM REST API REST API REST API REST API REST API REST API THEN NOW
9. "Works on my machine" DEV "The only good diff is a red diff" OPS
11. THE FIRST WAVE: THE SECOND WAVE: OPS DEV teaching devs to own code in production getting ops folks to code
12. WHY DEVS? CHARACTER BUILDING
13. EXPECTED ACTUAL vs
14. --- FAIL: TestUnitTest (0.00s) talk_test.go:10: -- actual: 5 (type int) expected: 4 (type int) ACTUAL EXPECTED
15. WRITE → TEST → COMMIT → RELEASE → OBSERVE TEST OBSERVE DEV
16. DEV OPS
17. WRITE → TEST → COMMIT → RELEASE → OBSERVE TEST OBSERVE DEV OPS
18. WHAT DOES "O11Y AS A SUPERPOWER" MEAN? DISCOVERING-NEW-POWERS MONTAGE
19. ⬜ SPEAKS THE LANGUAGE OF PROD ⬜ SEES THROUGH PROD TO CODE ⬜ EXPERIMENTS+CHECKS HYPOTHESES
20. PROD ME SPEAKS THE LANGUAGE OF PROD
21. ▸ Tools support the vocabulary natural to devs SPEAKS THE LANGUAGE OF PROD CPU utilization AWS availability zone kafka partition Cassandra hostname
22. SPEAKS THE LANGUAGE OF PROD ▸ Tools support the vocabulary natural to devs CPU utilization AWS availability zone kafka partition Cassandra hostname $YOUR_BIZ-relevant ID build ID API endpoint HTTP verb
23. ▸ Tools support the vocabulary natural to devs ▸ Tools support (custom) high-cardinality data SPEAKS THE LANGUAGE OF PROD payload size client OS build ID API endpoint HTTP verb time to render $YOUR_BIZ-relevant ID
24. ▸ Tools support the vocabulary natural to devs ▸ Tools support (custom) high-cardinality data SPEAKS THE LANGUAGE OF PROD
25. ▸ Tools support the vocabulary natural to devs ▸ Tools support (custom) high-cardinality data ▸ Instrumentation under constant evolution and tinkering SPEAKS THE LANGUAGE OF PROD dev prod
26. SEES THROUGH PROD TO CODE PROD ME CODE
27. SEES THROUGH PROD TO CODE ▸ Tracing: not just for concurrent or distributed systems
28. SEES THROUGH PROD TO CODE ▸ Tracing: not just for concurrent or distributed systems ▸ Describes describe shapes/workflows that feel familiar
29. SEES THROUGH PROD TO CODE
30. SEES THROUGH PROD TO CODE ▸ Tracing: not just for concurrent or distributed systems ▸ Describes describe shapes/workflows that feel familiar ▸ Support natural investigative flow
31. EXPERIMENTS + CHECKS HYPOTHESES PROD ME CODE
32. EXPERIMENTS + CHECKS HYPOTHESES ▸ Understand what "normal" is for the system
33. EXPERIMENTS + CHECKS HYPOTHESES ▸ Understand what "normal" is for the system ▸ Production as a testbed for in-development code
34. EXPERIMENTS + CHECKS HYPOTHESES ▸ Understand what "normal" is for the system ▸ Production as a testbed for in-development code ▸ Feature flags + observability = WRITE → TEST → RELEASE → OBSERVE OBSERVE OBSERVE OBSERVE OBSERVE OBSERVE
35. ☑ SPEAKS THE LANGUAGE OF PROD ☑ SEES THROUGH PROD TO CODE ☑ EXPERIMENTS+CHECKS HYPOTHESES
36. BUILDING A SHARED BRAIN BUILDING A SUPERHERO CREW
37. PAST PRESENT FUTURE TEAM BUILDING A SHARED BRAIN
38. "Leaps of intuition akin to magic"? EMPOWERING NON-EXPERTS active vs passive knowledge transfer Learnable + shareable
39. … ANTICLIMAX AKA, WHERE’S THE BATTLE SCENE?
40. REDUCE THE BATTLES WE HAVE TO FIGHT
41. (OR ELSE RISK BURNOUT)
42. FINAL VOICEOVER SUPERHERO RETROSPECTIVE
43. OPS: DEVS: embrace observability, bring production closer to development. share the great responsibility
 (and great power!) YOU CAN HAVE SUPERPOWERS NOW.
44. WRITE → TEST → COMMIT → RELEASE → OBSERVE TEST OBSERVE DEV OPS
45. ASK NEW QUESTIONS SHIP BETTER SOFTWARE thanks! q’s? @cyen @honeycombio ON TO THE NEXT ADVENTURE…