Operating Systems in Cloud City

Workshop for undergrad operating systems students, showing how operating system concepts apply when building cloud applications.

Transcript

1. Copyright Shopify 2018 Operating Systems in Cloud City John Arthorne @jarthorne Shopify Production Engineering May the Fourth, 2018
2. Copyright Shopify 2018
3. Copyright Shopify 2018 What is a Computer?
4. Copyright Shopify 2018 ROM Network CPU RAM I/O BUS Disk
5. Copyright Shopify 2018 What is an Operating System?
6. Copyright Shopify 2018 ROM Network CPU RAM I/O BUS Disk Virtual Memory Process Model Scheduler File System Sockets Device Controller Memory Manager
7. Copyright Shopify 2018 Applications in the Cloud • Applications in the cloud tend to span across multiple processes and computers • Applications are broken into specialized parts • Each part has duplicate copies, either for resiliency or scale • A traditional operating system can’t reason about or manage them
8. Copyright Shopify 2018 What are some parts that make up a cloud app?
9. Copyright Shopify 2018 Load Balancer HTTPS Web Server Redis/ Memcached Kafka MySQL Job Server webhooks
10. Copyright Shopify 2018 Processes in the Cloud • The unit of computation on the cloud computer are containers (eg Docker) • Containers are grouped into pods, similar to a group of threads forming a process • Inter-process communication is largely over TCP/IP network connections • Contention and coordination between containers is managed externally ◦ Eg, we use ZooKeeper for service discovery and routing
11. Copyright Shopify 2018 Producer/Consumer and Semaphores • Semaphores were first invented to prevent a producer writing to a buffer faster than a consumer could read • Producer/consumer problems arise in many places in cloud apps ◦ Processing webhooks ◦ Writing to a 3rd party API faster than it can receive ◦ Job queues “consuming” the available worker pool • In many cases we don’t have shared state between producer and consumer that can be used to store a shared semaphore • Sometimes have other feedback mechanisms such as HTTP 429 (“too many requests”) • When there is no feedback loop we can use a producer-side semaphore (circuit breaker)
12. Copyright Shopify 2018 Semaphore example: job queues • Limited number of available job workers • If all workers are busy, we can have starvation of waiting jobs • Notably, payment jobs are used to complete purchases • A large flash sale could exhaust workers for all shops • We use a semaphore to limit number of payment jobs running per pod
13. Copyright Shopify 2018 Local Memory in the Cloud • Containers need memory allocation just like normal processes • Kubernetes scheduler is a cross between a memory manager and a process scheduler • Scheduler allocates containers to physical computers (nodes) based on memory and CPU requirements (also potentially disk and other resources) • Containers can be swapped and evicted just like memory blocks and processes on a local computer
14. Copyright Shopify 2018 Scheduling CPU vs Memory What are some key differences between allocating CPU and memory? How can we handle exceeded limits in each case? How do we make maximum use of the underlying resource?
15. Copyright Shopify 2018 Shared Memory in the Cloud • Containers are ephemeral and disposable ◦ Eviction means killing the container • Any valuable state needs to be stored externally • Apps like Redis and memcached act like “RAM” of the cloud computer • Two main uses for shared memory: caches and queues • Cache memory is not allocated to a container, it is typically MRU • Queue memory is segmented by priority
16. Copyright Shopify 2018 Batch Scheduling: Jobs • Long running work is performed by background jobs • Dedicated group of containers constantly running jobs • We use resque which is a job scheduler built on redis • FIFO with multiple queues based on priority • Big jobs must be preemptible ◦ We built a custom framework for Shopify core: BackgroundQueue::Iteration
17. Copyright Shopify 2018 Interactive Scheduling: Load Balancers • Our core apps receive millions of HTTP requests/minute at all times • We need to spread these requests out to handle the load ◦ Much like an OS process scheduler allocating CPU time for processes • We use a mixture of commonly used algorithms ◦ Exponentially Weighted Moving Average (EWMA) - most recently least loaded ◦ Round Robin (RR) ◦ Power of 2 Choices (P2C) - Pick two at random, select least loaded
18. Copyright Shopify 2018 Story Time: Load Balancer Scheduling
19. Copyright Shopify 2018 Printer Spools • What problems come up when processes need to print?
20. Copyright Shopify 2018 Spooling and Kafka • Printer spools mediate between fast processors and slow I/O • Spools allow many things to write to one consumer in an orderly way • Kafka is a messaging system used heavily at Shopify, for similar reasons • Producers can deliver messages to a local Kafka client quickly • Kafka then sends them to a server, where they are spooled by topic • Consumers can consume from the spool/topic at their own pace
21. Copyright Shopify 2018 RAID and ElasticSearch • Redundant Array of Independent Disks (RAID) is a way of configuring disk drives • Distributed databases use a similar strategy to store multiple copies • ElasticSearch is one example that we use. It stores documents in multiple nodes, somewhat like block-level striping in RAID-5 • When one node (disk) fails, nothing is lost, but the entire cluster (drive) has slower performance while it rebalances
22. Copyright Shopify 2018 Summary • Operating systems provide a toolkit of abstractions to make life easier for application programs (and their developers) • Modern web apps run across hundreds or thousands of hosts (the “cloud computer”), where OS-level abstractions can’t help us • Many of these OS abstractions are applicable in surprising ways for apps running on the “cloud computer” • This is NOT a straight-forward mapping, but frameworks such as kubernetes help simplify some of the problems that emerge
23. Copyright Shopify 2018 A little bit of history... David Clark Post-PC Internet April 1999 http://bit.ly/post-pc-internet