Service Backplanes for the Modern Data Center

Modern data centers and clouds allow cluster resources to be mapped to applications dynamically, which can dramatically improve utilization and deployment agility. However, deploying off-the-shelf software in this environment is not easy: devops staff typically need to write "backplanes" to deploy, manage, and upgrade each software package that is deployed at scale. Despite their ad-hoc nature, these backplanes quickly grow in complexity and importance: a bug in the application backplane often means a production outage.

What is needed is a consistent framework for designing "backplanes": controllers that mediate between cluster resource APIs and server software. Backplanes capture much of the operational complexity of running a software package in production, allowing off-the-shelf software to be turned into an elastic service. In this talk, we motivate the need for service backplanes, discuss the functionality that belongs in a backplane, and consider how backplanes should be built, using Apache Mesos as an example.

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1 NABDConf 2016 - Neil Conway Service Backplanes for the Modern Data Center
2. © 2016 Mesosphere, Inc. All Rights Reserved. 2 About Me 2002-2008 Postgres Developer 2006-2008 Stream Processing Startup 2008-2014 PhD, Distributed Computing 2015- Mesosphere
3. © 2016 Mesosphere, Inc. All Rights Reserved. 3 Sound and Fury?
4. © 2016 Mesosphere, Inc. All Rights Reserved. 4 One Possible Reaction
5. © 2016 Mesosphere, Inc. All Rights Reserved. 5 1. What has changed about modern distributed systems? 2. What is a service backplane? 3. How should we build service backplanes? Outline
6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 The Good Old Days
7. © 2016 Mesosphere, Inc. All Rights Reserved. 7 Old-School Clusters Size Rate of Change Data Sets Small (TBs) Applications Few (1-10s) Machines Few (~100s) User Expertise High Humans involved
8. © 2016 Mesosphere, Inc. All Rights Reserved. 8 Resource Allocation Via Configuration n1 n4 n2 n5 n3 n6 n7 n8 n9 Config: • N1, N4, N7 ➝ Hadoop • N2, N5, N8 ➝ Postgres • N3, N6, N9 ➝ NGINX n1 n4 n2 n5 n3 n6 n7 n8 n9 Manual, static configuration Applications take resource allocation as an input
9. © 2016 Mesosphere, Inc. All Rights Reserved. 9 Analogy: Manual Memory Management 0x0 0x8 Config: • [0x0,0x1) ➝ calc.exe • [0x1,0x4) ➝ winmine.exe • [0x4,0x8) ➝ notepad.exe Physical Memory 0x0 0x8 Applications take physical memory address range as an input
10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 Consequences Utilization Low ❌ Deployment Agility Low ❌ Elasticity None ❌ Test / Dev / Staging Envs Difficult ❌ Simplicity High ✅ … but it basically worked, and it was simple.
11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 Modern Clusters Size Rate of Change Data Sets Massive Applications Many Machines Many User Expertise “Less” Expert Increasingly automated
12. © 2016 Mesosphere, Inc. All Rights Reserved. 12 Scaling Static Configuration? f(resources, apps) ➝ resource allocation n1 n3 n2 n4 n1 n4 n2 n5 n3 n6 n7 n8 n9 Static Config Tool
13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 Dynamic Resource Management “Service Backplane” n1 n3 n2 n4 Replace static configuration with program logic Unmodified application software
14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 Architecture • Allow unmodified application software to run at scale • Interface between application instances and provisioning APIs Service Backplanes Cassandra Backplane n1 n3 n2 n4 Postgres Backplane
15. © 2016 Mesosphere, Inc. All Rights Reserved. 15 Resource Management • Allocate resources to apps ◦ Fairness, utilization, etc. • Elasticity and auto-scaling • Oversubscription, perf isolation • Abstractions for complex resources (e.g., GPUs) Key Backplane Functionality Lifecycle Management • Replace failed instances ◦ Migrate state/data as needed • Allow machines, racks to be replaced (safely!) • Allow apps to be upgraded (safely!) Resource Management Lifecycle Management Backplane: interface between application and “cluster context”
16. © 2016 Mesosphere, Inc. All Rights Reserved. 16 Upgrading 3-10 Cassandra nodes: annoying but manageable. Upgrading 25k Cassandra nodes: really hard problem. Example: Upgrades at Scale Challenges: • Roll-backs, non- destructive upgrades • Deploy upgrade to subset of cluster • Move traffic away to avoid downtime • Data migration Hard to solve “inside” the app
17. © 2016 Mesosphere, Inc. All Rights Reserved. 17 • Scheduling is important • But: much more to backplanes than bin-packing or max-min fairness • Requires deep knowledge of ◦ Application semantics ◦ Ops procedures • Goal: transform prepackaged “server software” into “service” Not (Just) “Scheduling” or “Container Orchestration” “... there are not very many things that have aged as well as the [Linux] scheduler. Which is just another proof that scheduling is easy.” —Linus Torvalds, 2001
18. © 2016 Mesosphere, Inc. All Rights Reserved. 18 The State of the Art
19. © 2016 Mesosphere, Inc. All Rights Reserved. 19 Many organizations already build service backplanes. What Do People Do Today? … they just don’t know it.
20. © 2016 Mesosphere, Inc. All Rights Reserved. 20 Goal Provide a software service to the rest of the organization E.g., object storage, streaming data analysis, batch analytics, ML, etc. Common Pattern Solution • Start with off-the-shelf (OSS) software package • Write “scripts” to deploy, manage, and upgrade instances
21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 Building fault-tolerant control planes for cluster services is not easy! Problem #1: Backplanes Are Hard • Often >10,000s LOCs • Hard to test and debug • Maintenance burden Backplane downtime is service downtime
22. © 2016 Mesosphere, Inc. All Rights Reserved. 22 • In many cases, the service is the “product” • Backplane is just a “bunch of scripts” ◦ Not a distinct component of the system architecture • Sometimes built in an ad-hoc way • Often no rigorous specification or API Problem #2: Not Seen As A Product
23. © 2016 Mesosphere, Inc. All Rights Reserved. 23 • Many backplanes are similar • Typically built by different teams that don’t collaborate ◦ No opportunity for code reuse ◦ No shared infrastructure • Each backplane cannot examine global cluster state • Hard to define global policies that apply to all backplanes Problem #3: Redundancy Between Services
24. © 2016 Mesosphere, Inc. All Rights Reserved. 24 • Many organizations have custom- written backplanes for Cassandra, Kafka, HDFS, etc. • Often tightly coupled to their production environment ◦ Result: fragile, not portable to other environments Problem #4: Redundancy Between Organizations
25. © 2016 Mesosphere, Inc. All Rights Reserved. 25 Developer “ships” a release of their software package • Then >10k LOC is needed to deploy it at scale! This sucks • The upstream developer is the domain expert • Developer ships code their customer can’t (directly) use The Gap From “Done” to “Deployable” Can we standardize the functionality needed for large- scale deployments? • Allow backplane functionality to move “up” the stack • Tested and developed as part of the upstream software
26. © 2016 Mesosphere, Inc. All Rights Reserved. 26 1. Deploy to prod and pray 2. Document best practices (“runbook”) 3. Write scripts to handle common scenarios 4. Encode best practices as a service backplane Opportunity: Shrink Runbooks
27. © 2016 Mesosphere, Inc. All Rights Reserved. 27 A Modest Proposal
28. © 2016 Mesosphere, Inc. All Rights Reserved. 28 1. Embrace backplanes as a standard component in large- scale distributed systems • Not just “a few scripts” 2. Build infrastructure to make writing backplanes easier 3. Define standard APIs for communicating between backplanes and cluster infrastructure 4. Enable upstream software developers to ship backplanes as part of their software packages Rethinking Service Backplanes
29. © 2016 Mesosphere, Inc. All Rights Reserved. 29 Example Architecture Backplane Manager Cassandra Backplane Postgres Backplane Abstract away details of cloud or on-prem env. Clear API / interface for service backplanes Cluster Operator Single operator interface, define global policy
30. © 2016 Mesosphere, Inc. All Rights Reserved. 30 • “Manage your data center as a single pool of resources.” • UC Berkeley: 2008 • Battle-tested at Twitter: 2009-2016 • Other users: Apple, eBay, Netflix, Microsoft, PayPal, AirBnb, Criteo, Yelp, Uber, ... Background: Apache Mesos Mesos Master Scheduler X Mesos Agent Task Executor Scheduler Y Machine M “I have 8 CPUs, 8 disks, 64GB RAM” “Offer: would you like 8 CPUs, 8 disks, and 64GB of RAM?” “Accept: Launch container X.” “Launch container X.”
31. © 2016 Mesosphere, Inc. All Rights Reserved. 31 Mesos + Service Backplanes? Mesos Master Cassandra Framework Postgres Framework Cluster Operator
32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 Open Question: Software Architecture Backplane as part of application, or as separate component?
33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 • Backplane ↔ backplane manager • Application ↔ backplane • Dimensions: ◦ Push or pull (offer vs. request) ◦ Optimistic or pessimistic ◦ Declarative or imperative ◦ Narrow or wide • How to represent cluster resources? Open Question: APIs
34. © 2016 Mesosphere, Inc. All Rights Reserved. 34 • Where does the functionality live? ◦ Application, backplane, or backplane manager • Does this change how we should build common service features? ◦ Security? Logging? Metrics? Fault tolerance? Service discovery? Data migration? Open Question: Co-Design of Applications and Backplanes
35. © 2016 Mesosphere, Inc. All Rights Reserved. 35 1. Many people are building service backplanes, even if they don’t call them that 2. Driven by industry forces that are likely to persist 3. We should embrace the need for backplanes and figure out how to build them properly Conclusion
36. © 2016 Mesosphere, Inc. All Rights Reserved. 36 Thanks! neil@mesosphere.io @neil_conway