Keep Calm and Carry On: Scaling Your Org With Microservices

Ask people about their experience rolling out microservices, and one theme dominates: engineering is the easy part, people are super hard! Everybody knows about Conway's Law, everybody knows they need to make changes to their organization to support a different product model, but what are those changes? How do you know if you're succeeding or failing, if people are struggling and miserable or just experiencing the discomfort of learning new skills? We'll talk through real stories of pain and grief as people modernize their team and their stack.

Transcript

1. Keep Calm and Carry On: Scaling Your Org With Microservices Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
2. Keep Calm and Carry On: Scaling Your Org With Microservices Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
3. @mipsytipsy engineer, cofounder, CEO @bridgetkromhout #opslife, storyteller
4. @mipsytipsy engineer, cofounder, CEO @bridgetkromhout #opslife, storyteller
5. What even is a microservice (No one knows)
6. What are microservices? • Independently deployable, small modular services • Monorepo vs multiple repos • Decentralized governance • Small teams, up to maybe a dozen people (“two pizzas”) • Operating independently, interacting with other teams via APIs
7. Microservices are about changes.
9. Conway’s “Law”
10. Conway’s Law, post-Jobs
11. “Conway’s Law” is not a law (and the most important word in Conway’s Law is “communication”)
12. Growing your microservices org • Interfaces and abstractions • Data is just another service. • Ops is just another software engineering skill. • Implicit communication channels matter too. • Observability must be democratized.
13. • Team structure (Conway’s Law?) • Communication pathways • “Smarter Edges”: For individual contributors • “Dumb Pipes”: for managers • Transitions are hard i can haz microservices?
14. YAS! Has microservices: just the good parts • Don’t get religious. It’s not all or nothing. • What are your team’s strengths? What are their weaknesses? • Account for the operational cost
16. How many engineers do you have? How good are they at operations? ** you need to be REALLY GOOD at operations to do microservices.
17. How many products/services do you really have? Use a big fat service if it helps, plus some smaller ones Don’t microservice your shared libs, storage, or registry
18. Don’t reinvent too many wheels. new wheels have too many unknown-unknowns (“choose boring technology”: still applies)
19. “Dear Twitter …”
20. “Software deploys … that take days to run, when they run.” “I’m responsible for it, but I can’t log in to it.” Hard things are hard.
22. Interfaces and abstractions
23. Scaling considerations for services (also teams!!) • Scalability • Redundancy and resiliency • Consensus knowledge of processes and arch • Load balancing, early warning alerts, graceful degradation • Communication problems to debug, black-box debugging with remote hands Interfaces
24. Your team is a service, your humans are nodes Interfaces
25. Interfaces Ownership is super key. Every service must be owned by a human just like it must be served by a dedicated set of resources. we’ve all been on teams that spend more time circularly routing jobs around or blackholing them than actually fixing them
26. Management role #1: define the mission. Repeat the mission. Bore everyone to death with the mission. Interfaces Management role #2: routing, load balancing, health checking
27. You can chaos monkey your people! You should!! \o/ Interfaces
28. Interfaces
29. (what the actual fuck? do it anyway.) Interfaces
30. Interfaces
31. Communication channels
32. Implicit communication channels matter just as much (more?) than explicit channels. Comms
33. You can’t debug something you can’t name, describe, or understand. Comms Understand your communication channels.
34. Pull requests, code reviews. Reporting structures. Work schedules. WFH vs in-office. 1x1s. On-call rotations, escalation paths. Promotions, interviews, recruiting, hiring pipelines, mentorship. Gossip. Happy hours. Comms Examples of communication flows:
35. smart nodes, dumb pipes provision automatedly Managers’ job is primarily facilitating nodes Comms
36. The more you map and understand these, the more power you have to effect change. Comms
37. On call questions • Who is on call? Is it a necessary part of being an engineer? • How many rotations are there? • How often do people get woken up? *who* gets woken up? • How do you know? Who keeps track? • Are there different rotations for stateful and stateless services, front-end and backend? • Is there an escalation path? Comms
38. Operations is just another software engineering skill
39. You can’t be an effective SWE in a modern organization without ops skills. Ops
40. Empowerment and responsibility go hand in hand … you can’t ask someone to care about something and fix it without also giving them the power Ops
42. Do SWEs have to be on call? shrug. it helps. but it’s all about creating virtuous cause/effect loops Ops
44. Snowflakes are enormously costly. The larger your org gets, the fewer snowflakes you are allowed to have. Ops
46. networking: common theme
47. Probe every software engineering candidate for their ops experience & attitude. … yep, even FE/mobile devs!
48. “Operations is valued here.” you are signaling …
49. Senior software engineers should be reasonably good at these things. So if they are not, don’t promote them. Operations engineering is about making systems maintainable, reliable, and comprehensible.
50. Data ... is just another service.
51. 1) it’s impossible to treat stateful services exactly like stateless services. Data
52. YOU SHOULD TRY. The more you treat your stateful services like stateless ones, the more you win the future Data
53. Common pattern: state is the last to microserviceify. Monolith db layer serves many small services. (because it’s hard, and usually is not the most evident source of pain) Data
54. In the future, YOU are the DBA.** Data ** (Everything is going to be okay. Trust me.)
55. Data
56. and from a DBA at a different company … … Data
57. Observability … is the rock on which your castle is built
58. Technical observability: debugging, monitoring, metrics, instrumentation Observability
59. People observability: 1x1s, email, asking questions. Looking at their face and seeing if they are ok. Observability
60. If you’re doing microservices, you’re signing up for hard people problems. Observability
61. Observability
62. You have a responsibility to your team’s well-being whether you’re a manager or not. Observability
63. #truestory Observability
64. Observability
66. Talk to people BEFORE you launch any grand initiatives. Get their buy-in.
67. if you didn’t … #truestory
68. seek feedback move forward <3 change is the only constant
69. Get buy-in from *all* stakeholders.
70. Tech leads, senior ICs
71. Most failures happen around transitions. • unpacking a monolith -> microservices • rewriting from node.js into golang • acquiring or being acquired • migrating from hdfs in to new datastore • becoming a manager, or moving back to IC • getting married or divorced, having a kid
72. Choose the problems you are not going to solve, or they will choose you.
73. Making decisions: Get ready to talk to people a lot more about microservices. Sorry!
74. TL;DR: • Innovate only where you need to/where you'll gain (and yes, this includes microservices, function-as-a service, and whatever's next) • Empower yourself; don't wait. Actively decentralize power and you'll decentralize points of failure. • Ask for permission strategically; move your org towards assume- yes. • Communication (implicit and explicit) is key to decentralizing & microservices • Look for the uncomfortable places. Be happy when you find them; that's where you and the org can grow.
75. There is no fairy-tale answer Microservices give you flexibility; the rest is up to you, because hard things are still hard even when they're distributed and small.
78. Operability / Teams. • The mission • Build a cult (j/k) (no really) • Let your team innovate.
79. most outages are triggered by “events”, from humans. draw a line.
80. Pair responsibility with empowerment.
81. Have you considered … valuing non generalist SWEs and their work?
82. Deploys On-Call Pull requests, arch reviews Observability Communication channels
83. Deploys
84. Deploys must be: • Fast. Rolling. Roll-back-able. • Reliable. Breaks rarely. • Draws a tagged vertical line in graphs. • *Anyone* should be able to invoke deploy • For bonus points: canarying or automated
85. Revisit these tools regularly. part of every post mortem.
86. On Call
87. Haha, no.
88. What should leaders know? Managers, tech leads, and engineers
89. Things about leadership • Leadership is not a zero sum game. The best leaders try to empower literally everyone to perform a leadership role in at least some areas. • Create guard-rails, not walls. • Be conventional in the big things (salary, org), unconventional in the small. • If you give a shit about diversity, don’t wait 'til you’re “ready” to hire them … look for ways to support underrepresented groups now. Make friends. Help people. Diversify your friend groups and personal networks. Be creative.
90. Management • Put the humans first, and the mission a close second • Be an enabler. Don’t starve your tech leads of growth opportunities by sucking all oxygen. • Reward intentionally. • Leadership is not zero-sum; encourage leadership everywhere • Managers, be friends with each other! Tolerance is not enough
91. The most powerful weapon in your arsenal is always cause and effect.
92. Engineers should be on call for their own services.
93. Yes but …. Yes, microservices helps you drift a little bit and innovate independently … BUT, not as much as you might think. You all still share a fabric, after all. Stateful still gonna ruin your party. (and IPC, sec discovery, caching, cd pipelines, databases etc.)
94. • I don’t think anyone should approach management as a thing they move in to permanently. It’s psychologically disfiguring. • Nor is the maturation process one way. New teams within the company should be springing up. Hackathons can be a great way, esp if it involves dogfooding. Empathy needs constant renewal. • Practice making mistakes together. Practice cheerful apologies, asking questions, giving awkward feedback. It gets easier.
95. Charity Majors @mipsytipsy Bridget Kromhout @bridgetkromhout