Anti-DevOps: SRE at Honeycomb

Siloes and boundaries are natural and useful; they constrain the problem space, make it possible to reason in the face of overwhelming complexity, and give space for ideas and people that don’t fit within the majority. However, those boundaries can also become so deeply entrenched and solid that they become counterproductive (thus, DevOps) - and yet the cycle repeats itself, where “DevOps Engineer” becomes a new way to say “infrastructure engineer” in many companies, someone who is holed away creating the platonic ideal of a well-managed Kubernetes cluster.

At Honeycomb, our Site Reliability Engineering team does financial modeling and product management, embeds with our Customer Support team (who is critical to tuning our SLOs), and still finds time to be Terraform wranglers. In this talk, we’ll walk through the benefits and occupational hazards of breaking down barriers, why all your managers should strive to be accomplished sociologists, and how to touch grass with your SLOs.

Embrace Complexity; Tighten Your Feedback Loops, Fred Hebert
Exit, Voice, and Loyalty, Albert O. Hirschman
Effective DevOps, Jennifer Davis & Ryn Daniels
10+ Deploys Per Day, Paul Hammond & John Allspaw
If You Build It, They Will Come (podcast including Honeycomb
name story)

Transcript

2. about me • reid (they/them) • manage the sre team at honeycomb • local! • new to speaking, not new to devops
3. about honeycomb • started in 2016 • ingests your telemetry really fast and let you do cool things with it • complete quiz.honeydemo.io for a lego set!
4. anti-devops
5. • transparency ◦ bad • silos ◦ good • shipping iteratively ◦ bad this talk …without clarity …at the right size, with cross-pollination …
6. transparency bad 01
7. early days of honeycomb • scrappy startup by people burned by big tech • transparent • engineering-driven • asynchronous • limited organizational innovation tokens
8. organizational innovation tokens? • ricardo semler, CEO of Semco • “if you look at Semco’s numbers, we’ve grown 27.5 percent a year for 14 years…” • radically transparent • self-organized, nearly no management • …he’s the former CEO’s son • …company existed for ~30 years beforehand • …hospitalized after passing out 4 years in at age 25 • …nearly a decade of flailing
9. “i’d take a bad organizational structure with good people over a good organizational structure with bad people.”
10. [nearly] everything public! [nearly] everything asynchronous! transparency
11. • too many slack channels and too much to read • hard to understand strategy • unclear how to succeed! honeycomb 2023
12. transparency is bad? transparency without clarity is bad.
13. “i really need to pee” is providing transparency. “do you see a bathroom around?” is providing clarity. the fundamentals of “company” (relationship) strategy
15. • managers: translate ◦ no poop shielding!! • ICs: demand ◦ more important as your seniority increases be clear
16. silos good 02
17. breaking down barriers is an expansion • removal of process • creating feedback loops • sharing knowledge • results in an expansion of: ◦ communication channels ◦ complexity ◦ relationships ◦ responsibility ◦ charter + work!
18. expansions decrease slack • Exit, Voice, and Loyalty ◦ organizational slack • slack = (resources * ability) / obligations
19. telegraph your obligations • problem: obligations are harder to see • your responsibility to telegraph the slack of your team • solution: product management
20. send your team to product school let your roadmap argue for you.
21. how do you increase slack? • replacing relationships with process • reducing scope ◦ charter shrinkage • removing a feedback loop
22. you don’t need to know everything because a lot of it is boring! at the very least, it can be overwhelming boundaries exist for a reason …except for infra/SRE/DevOps teams.
23. listening to the spider web • we embed with (nearly) all engineering teams ◦ except those with an SRE champion • be a spider worth trusting
24. being the feedback loop security -> engineering: vulnerability scans support <-> engineering: on-call tickets leadership <- engineering: incident comms
25. water break.
26. stop shipping iteratively 03
27. why do we say “ship iteratively”? • we used to not • tempting to go off in a hole and build platonically ideal applications ◦ this does not work for startups. • a very good way to get feedback quickly
28. please don’t ship my house iteratively • there’s also things that don’t work as well with iterative development: ◦ embedded software ◦ telecom equipment ◦ medical devices ◦ infrastructure changes • but that also comes with slow feedback loops
29. • because we can. • so we can see if we just broke the app • so management can tell if we just broke the app • but how are you supposed to do that? ◦ monitoring can support slow feedback loops ◦ observability can support slow and fast feedback loops why do we want quick feedback?
30. the old days app server 1 app server 2 database load balancer cpu memory disk port 443 cpu memory disk port 80 cpu memory disk port 80 cpu memory disk port 5432
31. the “big green button”
32. “observability” app server 1 app server 2 database load balancer user: deez error: nope! duration: 1ms
33. are we meeting our SLOs?
34. touching grass
35. • DevOps is good enough name, but much more than Dev and Ops • stop selling it • not dead, just smells funny anti-DevOps?
36. break down silos thesis antithesis synthesis be transparent anti-devops? ship iteratively create silos hide information ship monolithically analyze interactions add clarity touch grass
37. find me on: everyone’s favorite social media & dating site, LinkedIn bluesky, @reid9.bsky.social thanks!
38. resources, further reading Embrace Complexity; Tighten Your Feedback Loops, Fred Hebert Exit, Voice, and Loyalty, Albert O. Hirschman Effective DevOps, Jennifer Davis & Ryn Daniels 10+ Deploys Per Day, Paul Hammond & John Allspaw If You Build It, They Will Come (podcast including Honeycomb name story)