Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Keep Calm and Carry On: Scaling Your Org With Microservices

Keep Calm and Carry On: Scaling Your Org With Microservices

Ask people about their experience rolling out microservices, and one theme dominates: engineering is the easy part, people are super hard! Everybody knows about Conway's Law, everybody knows they need to make changes to their organization to support a different product model, but what are those changes? How do you know if you're succeeding or failing, if people are struggling and miserable or just experiencing the discomfort of learning new skills? We'll talk through real stories of pain and grief as people modernize their team and their stack.


Charity Majors

March 26, 2017

More Decks by Charity Majors

Other Decks in Technology


  1. Keep Calm and Carry On: Scaling Your Org With Microservices

    Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
  2. Keep Calm and Carry On: Scaling Your Org With Microservices

    Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
  3. @mipsytipsy engineer, cofounder, CEO @bridgetkromhout #opslife, storyteller

  4. @mipsytipsy engineer, cofounder, CEO @bridgetkromhout #opslife, storyteller

  5. What even is a microservice (No one knows)

  6. What are microservices? • Independently deployable, small modular services •

    Monorepo vs multiple repos • Decentralized governance • Small teams, up to maybe a dozen people (“two pizzas”) • Operating independently, interacting with other teams via APIs
  7. Microservices are about changes.

  8. None
  9. Conway’s “Law”

  10. Conway’s Law, post-Jobs

  11. “Conway’s Law” is not a law (and the most important

    word in Conway’s Law is “communication”)
  12. Growing your microservices org • Interfaces and abstractions • Data

    is just another service. • Ops is just another software engineering skill. • Implicit communication channels matter too. • Observability must be democratized.
  13. • Team structure (Conway’s Law?) • Communication pathways • “Smarter

    Edges”: For individual contributors • “Dumb Pipes”: for managers • Transitions are hard i can haz microservices?
  14. YAS! Has microservices: just the good parts • Don’t get

    religious. It’s not all or nothing. • What are your team’s strengths? What are their weaknesses? • Account for the operational cost
  15. None
  16. How many engineers do you have? How good are they

    at operations? ** you need to be REALLY GOOD at operations to do microservices.
  17. How many products/services do you really have? Use a big

    fat service if it helps, plus some smaller ones Don’t microservice your shared libs, storage, or registry
  18. Don’t reinvent too many wheels. new wheels have too many

    unknown-unknowns (“choose boring technology”: still applies)
  19. “Dear Twitter …”

  20. “Software deploys … that take days to run, when they

    run.” “I’m responsible for it, but I can’t log in to it.” Hard things are hard.
  21. None
  22. Interfaces and abstractions

  23. Scaling considerations for services (also teams!!) • Scalability • Redundancy

    and resiliency • Consensus knowledge of processes and arch • Load balancing, early warning alerts, graceful degradation • Communication problems to debug, black-box debugging with remote hands Interfaces
  24. Your team is a service, your humans are nodes Interfaces

  25. Interfaces Ownership is super key. Every service must be owned

    by a human just like it must be served by a dedicated set of resources. we’ve all been on teams that spend more time circularly routing jobs around or blackholing them than actually fixing them
  26. Management role #1: define the mission. Repeat the mission. Bore

    everyone to death with the mission. Interfaces Management role #2: routing, load balancing, health checking
  27. You can chaos monkey your people! You should!! \o/ Interfaces

  28. Interfaces

  29. (what the actual fuck? do it anyway.) Interfaces

  30. Interfaces

  31. Communication channels

  32. Implicit communication channels matter just as much (more?) than explicit

    channels. Comms
  33. You can’t debug something you can’t name, describe, or understand.

    Comms Understand your communication channels.
  34. Pull requests, code reviews. Reporting structures. Work schedules. WFH vs

    in-office. 1x1s. On-call rotations, escalation paths. Promotions, interviews, recruiting, hiring pipelines, mentorship. Gossip. Happy hours. Comms Examples of communication flows:
  35. smart nodes, dumb pipes provision automatedly Managers’ job is primarily

    facilitating nodes Comms
  36. The more you map and understand these, the more power

    you have to effect change. Comms
  37. On call questions • Who is on call? Is it

    a necessary part of being an engineer? • How many rotations are there? • How often do people get woken up? *who* gets woken up? • How do you know? Who keeps track? • Are there different rotations for stateful and stateless services, front-end and backend? • Is there an escalation path? Comms
  38. Operations is just another software engineering skill

  39. You can’t be an effective SWE in a modern organization

    without ops skills. Ops
  40. Empowerment and responsibility go hand in hand … you can’t

    ask someone to care about something and fix it without also giving them the power Ops
  41. None
  42. Do SWEs have to be on call? shrug. it helps.

    but it’s all about creating virtuous cause/effect loops Ops
  43. None
  44. Snowflakes are enormously costly. The larger your org gets, the

    fewer snowflakes you are allowed to have. Ops
  45. None
  46. networking: common theme

  47. Probe every software engineering candidate for their ops experience &

    attitude. … yep, even FE/mobile devs!
  48. “Operations is valued here.” you are signaling …

  49. Senior software engineers should be reasonably good at these things.

    So if they are not, don’t promote them. Operations engineering is about making systems maintainable, reliable, and comprehensible.
  50. Data ... is just another service.

  51. 1) it’s impossible to treat stateful services exactly like stateless

    services. Data
  52. YOU SHOULD TRY. The more you treat your stateful services

    like stateless ones, the more you win the future Data
  53. Common pattern: state is the last to microserviceify. Monolith db

    layer serves many small services. (because it’s hard, and usually is not the most evident source of pain) Data
  54. In the future, YOU are the DBA.** Data ** (Everything

    is going to be okay. Trust me.)
  55. Data

  56. and from a DBA at a different company … …

  57. Observability … is the rock on which your castle is

  58. Technical observability: debugging, monitoring, metrics, instrumentation Observability

  59. People observability: 1x1s, email, asking questions. Looking at their face

    and seeing if they are ok. Observability
  60. If you’re doing microservices, you’re signing up for hard people

    problems. Observability
  61. Observability

  62. You have a responsibility to your team’s well-being whether you’re

    a manager or not. Observability
  63. #truestory Observability

  64. Observability

  65. None
  66. Talk to people BEFORE you launch any grand initiatives. Get

    their buy-in.
  67. if you didn’t … #truestory

  68. seek feedback move forward <3 change is the only constant

  69. Get buy-in from *all* stakeholders.

  70. Tech leads, senior ICs

  71. Most failures happen around transitions. • unpacking a monolith ->

    microservices • rewriting from node.js into golang • acquiring or being acquired • migrating from hdfs in to new datastore • becoming a manager, or moving back to IC • getting married or divorced, having a kid
  72. Choose the problems you are not going to solve, or

    they will choose you.
  73. Making decisions: Get ready to talk to people a lot

    more about microservices. Sorry!
  74. TL;DR: • Innovate only where you need to/where you'll gain

    (and yes, this includes microservices, function-as-a service, and whatever's next) • Empower yourself; don't wait. Actively decentralize power and you'll decentralize points of failure. • Ask for permission strategically; move your org towards assume- yes. • Communication (implicit and explicit) is key to decentralizing & microservices • Look for the uncomfortable places. Be happy when you find them; that's where you and the org can grow.
  75. There is no fairy-tale answer Microservices give you flexibility; the

    rest is up to you, because hard things are still hard even when they're distributed and small.
  76. None
  77. None
  78. Operability / Teams. • The mission • Build a cult

    (j/k) (no really) • Let your team innovate.
  79. most outages are triggered by “events”, from humans. draw a

  80. Pair responsibility with empowerment.

  81. Have you considered … valuing non generalist SWEs and their

  82. Deploys On-Call Pull requests, arch reviews Observability Communication channels

  83. Deploys

  84. Deploys must be: • Fast. Rolling. Roll-back-able. • Reliable. Breaks

    rarely. • Draws a tagged vertical line in graphs. • *Anyone* should be able to invoke deploy • For bonus points: canarying or automated
  85. Revisit these tools regularly. part of every post mortem.

  86. On Call

  87. Haha, no.

  88. What should leaders know? Managers, tech leads, and engineers

  89. Things about leadership • Leadership is not a zero sum

    game. The best leaders try to empower literally everyone to perform a leadership role in at least some areas. • Create guard-rails, not walls. • Be conventional in the big things (salary, org), unconventional in the small. • If you give a shit about diversity, don’t wait 'til you’re “ready” to hire them … look for ways to support underrepresented groups now. Make friends. Help people. Diversify your friend groups and personal networks. Be creative.
  90. Management • Put the humans first, and the mission a

    close second • Be an enabler. Don’t starve your tech leads of growth opportunities by sucking all oxygen. • Reward intentionally. • Leadership is not zero-sum; encourage leadership everywhere • Managers, be friends with each other! Tolerance is not enough
  91. The most powerful weapon in your arsenal is always cause

    and effect.
  92. Engineers should be on call for their own services.

  93. Yes but …. Yes, microservices helps you drift a little

    bit and innovate independently … BUT, not as much as you might think. You all still share a fabric, after all. Stateful still gonna ruin your party. (and IPC, sec discovery, caching, cd pipelines, databases etc.)
  94. • I don’t think anyone should approach management as a

    thing they move in to permanently. It’s psychologically disfiguring. • Nor is the maturation process one way. New teams within the company should be springing up. Hackathons can be a great way, esp if it involves dogfooding. Empathy needs constant renewal. • Practice making mistakes together. Practice cheerful apologies, asking questions, giving awkward feedback. It gets easier.
  95. Charity Majors @mipsytipsy Bridget Kromhout @bridgetkromhout