Keep Calm and Carry On: Scaling Your Org With Microservices

Keep Calm and Carry On: Scaling Your Org With Microservices

Ask people about their experience rolling out microservices, and one theme dominates: engineering is the easy part, people are super hard! Everybody knows about Conway's Law, everybody knows they need to make changes to their organization to support a different product model, but what are those changes? How do you know if you're succeeding or failing, if people are struggling and miserable or just experiencing the discomfort of learning new skills? We'll talk through real stories of pain and grief as people modernize their team and their stack.

Ac734fc32781678475b577944bb5a9ae?s=128

Charity Majors

March 26, 2017
Tweet

Transcript

  1. 1.

    Keep Calm and Carry On: Scaling Your Org With Microservices

    Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
  2. 2.

    Keep Calm and Carry On: Scaling Your Org With Microservices

    Charity Majors, @mipsytipsy Bridget Kromhout, @bridgetkromhout
  3. 6.

    What are microservices? • Independently deployable, small modular services •

    Monorepo vs multiple repos • Decentralized governance • Small teams, up to maybe a dozen people (“two pizzas”) • Operating independently, interacting with other teams via APIs
  4. 8.
  5. 11.

    “Conway’s Law” is not a law (and the most important

    word in Conway’s Law is “communication”)
  6. 12.

    Growing your microservices org • Interfaces and abstractions • Data

    is just another service. • Ops is just another software engineering skill. • Implicit communication channels matter too. • Observability must be democratized.
  7. 13.

    • Team structure (Conway’s Law?) • Communication pathways • “Smarter

    Edges”: For individual contributors • “Dumb Pipes”: for managers • Transitions are hard i can haz microservices?
  8. 14.

    YAS! Has microservices: just the good parts • Don’t get

    religious. It’s not all or nothing. • What are your team’s strengths? What are their weaknesses? • Account for the operational cost
  9. 15.
  10. 16.

    How many engineers do you have? How good are they

    at operations? ** you need to be REALLY GOOD at operations to do microservices.
  11. 17.

    How many products/services do you really have? Use a big

    fat service if it helps, plus some smaller ones Don’t microservice your shared libs, storage, or registry
  12. 18.

    Don’t reinvent too many wheels. new wheels have too many

    unknown-unknowns (“choose boring technology”: still applies)
  13. 20.

    “Software deploys … that take days to run, when they

    run.” “I’m responsible for it, but I can’t log in to it.” Hard things are hard.
  14. 21.
  15. 23.

    Scaling considerations for services (also teams!!) • Scalability • Redundancy

    and resiliency • Consensus knowledge of processes and arch • Load balancing, early warning alerts, graceful degradation • Communication problems to debug, black-box debugging with remote hands Interfaces
  16. 25.

    Interfaces Ownership is super key. Every service must be owned

    by a human just like it must be served by a dedicated set of resources. we’ve all been on teams that spend more time circularly routing jobs around or blackholing them than actually fixing them
  17. 26.

    Management role #1: define the mission. Repeat the mission. Bore

    everyone to death with the mission. Interfaces Management role #2: routing, load balancing, health checking
  18. 33.

    You can’t debug something you can’t name, describe, or understand.

    Comms Understand your communication channels.
  19. 34.

    Pull requests, code reviews. Reporting structures. Work schedules. WFH vs

    in-office. 1x1s. On-call rotations, escalation paths. Promotions, interviews, recruiting, hiring pipelines, mentorship. Gossip. Happy hours. Comms Examples of communication flows:
  20. 36.

    The more you map and understand these, the more power

    you have to effect change. Comms
  21. 37.

    On call questions • Who is on call? Is it

    a necessary part of being an engineer? • How many rotations are there? • How often do people get woken up? *who* gets woken up? • How do you know? Who keeps track? • Are there different rotations for stateful and stateless services, front-end and backend? • Is there an escalation path? Comms
  22. 40.

    Empowerment and responsibility go hand in hand … you can’t

    ask someone to care about something and fix it without also giving them the power Ops
  23. 41.
  24. 42.

    Do SWEs have to be on call? shrug. it helps.

    but it’s all about creating virtuous cause/effect loops Ops
  25. 43.
  26. 44.

    Snowflakes are enormously costly. The larger your org gets, the

    fewer snowflakes you are allowed to have. Ops
  27. 45.
  28. 47.
  29. 49.

    Senior software engineers should be reasonably good at these things.

    So if they are not, don’t promote them. Operations engineering is about making systems maintainable, reliable, and comprehensible.
  30. 52.

    YOU SHOULD TRY. The more you treat your stateful services

    like stateless ones, the more you win the future Data
  31. 53.

    Common pattern: state is the last to microserviceify. Monolith db

    layer serves many small services. (because it’s hard, and usually is not the most evident source of pain) Data
  32. 54.
  33. 55.
  34. 65.
  35. 71.

    Most failures happen around transitions. • unpacking a monolith ->

    microservices • rewriting from node.js into golang • acquiring or being acquired • migrating from hdfs in to new datastore • becoming a manager, or moving back to IC • getting married or divorced, having a kid
  36. 73.

    Making decisions: Get ready to talk to people a lot

    more about microservices. Sorry!
  37. 74.

    TL;DR: • Innovate only where you need to/where you'll gain

    (and yes, this includes microservices, function-as-a service, and whatever's next) • Empower yourself; don't wait. Actively decentralize power and you'll decentralize points of failure. • Ask for permission strategically; move your org towards assume- yes. • Communication (implicit and explicit) is key to decentralizing & microservices • Look for the uncomfortable places. Be happy when you find them; that's where you and the org can grow.
  38. 75.

    There is no fairy-tale answer Microservices give you flexibility; the

    rest is up to you, because hard things are still hard even when they're distributed and small.
  39. 76.
  40. 77.
  41. 78.

    Operability / Teams. • The mission • Build a cult

    (j/k) (no really) • Let your team innovate.
  42. 83.
  43. 84.

    Deploys must be: • Fast. Rolling. Roll-back-able. • Reliable. Breaks

    rarely. • Draws a tagged vertical line in graphs. • *Anyone* should be able to invoke deploy • For bonus points: canarying or automated
  44. 86.
  45. 87.
  46. 89.

    Things about leadership • Leadership is not a zero sum

    game. The best leaders try to empower literally everyone to perform a leadership role in at least some areas. • Create guard-rails, not walls. • Be conventional in the big things (salary, org), unconventional in the small. • If you give a shit about diversity, don’t wait 'til you’re “ready” to hire them … look for ways to support underrepresented groups now. Make friends. Help people. Diversify your friend groups and personal networks. Be creative.
  47. 90.

    Management • Put the humans first, and the mission a

    close second • Be an enabler. Don’t starve your tech leads of growth opportunities by sucking all oxygen. • Reward intentionally. • Leadership is not zero-sum; encourage leadership everywhere • Managers, be friends with each other! Tolerance is not enough
  48. 93.

    Yes but …. Yes, microservices helps you drift a little

    bit and innovate independently … BUT, not as much as you might think. You all still share a fabric, after all. Stateful still gonna ruin your party. (and IPC, sec discovery, caching, cd pipelines, databases etc.)
  49. 94.

    • I don’t think anyone should approach management as a

    thing they move in to permanently. It’s psychologically disfiguring. • Nor is the maturation process one way. New teams within the company should be springing up. Hackathons can be a great way, esp if it involves dogfooding. Empathy needs constant renewal. • Practice making mistakes together. Practice cheerful apologies, asking questions, giving awkward feedback. It gets easier.