Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsDays Cuba 2017: Evolving Architecture And Organization Together - Lessons From Google And EBay

DevOpsDays Cuba
October 26, 2017

DevOpsDays Cuba 2017: Evolving Architecture And Organization Together - Lessons From Google And EBay

Author: Randy Shoup
Summary: The “right” architecture and organization depends on the size and scale of your company. The only constant is change, and what works for 5 engineers does not work for 5000. Based upon lessons from Google and eBay, learn how to evolve both technology and organization together successfully.

This presentation is based on many hard-won lessons by the speaker, who led large-scale engineering teams at Google and eBay, but also co-founded a tiny startup and tried (unsuccessfully) to apply the same techniques. This session hopes to help others from making the same mistakes by introducing the concept of “Minimal Viable Architecture”. It outlines the common architectural evolution of a company or project through the search, execution, and scaling phases, and discusses the appropriate technologies, disciplines, and organizational structures at each phase.

DevOpsDays Cuba

October 26, 2017


  1. Evolving Architecture and Organization Together Lessons from Google and eBay

    Randy Shoup @randyshoup linkedin.com/in/randyshoup
  2. Conway’s Law • Organization determines architecture o Design of a

    system will be a reflection of the communication paths within the organization • We can engineer the system we want by engineering the organization
  3. Small “Service” Teams • Amazon “2 Pizza” Teams o No

    team should be larger than can be fed by 2 large pizzas o Typically 4-6 people • Aligned to Business Domains o Clear, well-defined area of responsibility o Single service or set of related services o Minimal, well-defined “interface” @randyshoup linkedin.com/in/randyshoup
  4. Full-Stack Teams • All disciplines required for the team’s function

    o Design o Development o Quality and Performance o Maintenance o Operations • Depend on other teams for supporting services, libraries, and tools @randyshoup linkedin.com/in/randyshoup
  5. “Tell us how you did things at Google and eBay.”

    “Sure, I’ll tell you, but you have to promise not to do them! [… yet]”
  6. Architecture Evolution • eBay • 5th generation today • Monolithic

    Perl à Monolithic C++ à Java à microservices • Twitter • 3rd generation today • Monolithic Rails à JS / Rails / Scala à microservices • Amazon • Nth generation today • Monolithic Perl / C++ à Java / Scala à microservices
  7. Microservices Each unit is simple Independent scaling and performance Independent

    testing and deployment Can optimally tune performance (caching, replication, etc.) Pros Many cooperating units Many small repos Requires more sophisticated tooling and dependency management Network latencies Cons
  8. Why Rearchitect? • Velocity o Teams step on each others’

    toes, and can no longer develop independently o Difficult for new engineers to be productive • Scaling o Vertical scaling of the monolith no longer works o Parts of the system need to scale independently of others
  9. Why Rearchitect? • Deployment o Parts of the system need

    to deploy independently of others o Monolithic release is too slow, too complicated, too risky
  10. Carving up the Monolith • Look for (or create) a

    “seam” in the monolith o This is often the hardest part (!) • Wall it off behind an interface • Write automated tests around the interface • Replace implementation with an independent component • è Rinse and Repeat
  11. Ownership • End-to-end Ownership o Team owns service from design

    to deployment to retirement • Responsible for o Features o Quality o Performance o Operations o Maintenance
  12. Cross-Functional Collaboration • Open communication o Individuals encouraged to work

    directly with each other o Prefer informal cooperation over formal channels • Best decisions made through partnership o Agreement on goals and priorities makes it easier to agree on tactics o Given common context, well-meaning people will generally agree o “Disagree and Commit”
  13. None of us is as smart as all of us.

    -- Japanese proverb, as quoted by Bob Taylor
  14. Continuous Integration • Small incremental changes o Faster feedback loop

    o Easy to understand, code-review, test, roll back • Larger changes are much riskier o Risk of code change is nonlinear in the size of the change o Ex. Initial memcache service submission • Main code branch always shippable o Increased rate of change while reducing risk of change
  15. Continuous Delivery • More solid systems o Release smaller units

    of work o Faster to repair, easier to diagnose o Small change to roll back or roll forward • Enables experimentation o Small experiments and rapid iteration are cheap • Enabled by o PaaS, containers, cloud