Operating Systems in Cloud City

Slide 1

Slide 1 text

Copyright Shopify 2018 Operating Systems in Cloud City John Arthorne @jarthorne Shopify Production Engineering May the Fourth, 2018

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Copyright Shopify 2018 ROM Network CPU RAM I/O BUS Disk Virtual Memory Process Model Scheduler File System Sockets Device Controller Memory Manager

Slide 7

Slide 7 text

Copyright Shopify 2018 Applications in the Cloud ● Applications in the cloud tend to span across multiple processes and computers ● Applications are broken into specialized parts ● Each part has duplicate copies, either for resiliency or scale ● A traditional operating system can’t reason about or manage them

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Copyright Shopify 2018 Processes in the Cloud ● The unit of computation on the cloud computer are containers (eg Docker) ● Containers are grouped into pods, similar to a group of threads forming a process ● Inter-process communication is largely over TCP/IP network connections ● Contention and coordination between containers is managed externally ○ Eg, we use ZooKeeper for service discovery and routing

Slide 11

Slide 11 text

Copyright Shopify 2018 Producer/Consumer and Semaphores ● Semaphores were first invented to prevent a producer writing to a buffer faster than a consumer could read ● Producer/consumer problems arise in many places in cloud apps ○ Processing webhooks ○ Writing to a 3rd party API faster than it can receive ○ Job queues “consuming” the available worker pool ● In many cases we don’t have shared state between producer and consumer that can be used to store a shared semaphore ● Sometimes have other feedback mechanisms such as HTTP 429 (“too many requests”) ● When there is no feedback loop we can use a producer-side semaphore (circuit breaker)

Slide 12

Slide 12 text

Copyright Shopify 2018 Semaphore example: job queues ● Limited number of available job workers ● If all workers are busy, we can have starvation of waiting jobs ● Notably, payment jobs are used to complete purchases ● A large flash sale could exhaust workers for all shops ● We use a semaphore to limit number of payment jobs running per pod

Slide 13

Slide 13 text

Copyright Shopify 2018 Local Memory in the Cloud ● Containers need memory allocation just like normal processes ● Kubernetes scheduler is a cross between a memory manager and a process scheduler ● Scheduler allocates containers to physical computers (nodes) based on memory and CPU requirements (also potentially disk and other resources) ● Containers can be swapped and evicted just like memory blocks and processes on a local computer

Slide 14

Slide 14 text

Copyright Shopify 2018 Scheduling CPU vs Memory What are some key differences between allocating CPU and memory? How can we handle exceeded limits in each case? How do we make maximum use of the underlying resource?

Slide 15

Slide 15 text

Copyright Shopify 2018 Shared Memory in the Cloud ● Containers are ephemeral and disposable ○ Eviction means killing the container ● Any valuable state needs to be stored externally ● Apps like Redis and memcached act like “RAM” of the cloud computer ● Two main uses for shared memory: caches and queues ● Cache memory is not allocated to a container, it is typically MRU ● Queue memory is segmented by priority

Slide 16

Slide 16 text

Copyright Shopify 2018 Batch Scheduling: Jobs ● Long running work is performed by background jobs ● Dedicated group of containers constantly running jobs ● We use resque which is a job scheduler built on redis ● FIFO with multiple queues based on priority ● Big jobs must be preemptible ○ We built a custom framework for Shopify core: BackgroundQueue::Iteration

Slide 17

Slide 17 text

Copyright Shopify 2018 Interactive Scheduling: Load Balancers ● Our core apps receive millions of HTTP requests/minute at all times ● We need to spread these requests out to handle the load ○ Much like an OS process scheduler allocating CPU time for processes ● We use a mixture of commonly used algorithms ○ Exponentially Weighted Moving Average (EWMA) - most recently least loaded ○ Round Robin (RR) ○ Power of 2 Choices (P2C) - Pick two at random, select least loaded

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Copyright Shopify 2018 Spooling and Kafka ● Printer spools mediate between fast processors and slow I/O ● Spools allow many things to write to one consumer in an orderly way ● Kafka is a messaging system used heavily at Shopify, for similar reasons ● Producers can deliver messages to a local Kafka client quickly ● Kafka then sends them to a server, where they are spooled by topic ● Consumers can consume from the spool/topic at their own pace

Slide 21

Slide 21 text

Copyright Shopify 2018 RAID and ElasticSearch ● Redundant Array of Independent Disks (RAID) is a way of configuring disk drives ● Distributed databases use a similar strategy to store multiple copies ● ElasticSearch is one example that we use. It stores documents in multiple nodes, somewhat like block-level striping in RAID-5 ● When one node (disk) fails, nothing is lost, but the entire cluster (drive) has slower performance while it rebalances

Slide 22

Slide 22 text

Copyright Shopify 2018 Summary ● Operating systems provide a toolkit of abstractions to make life easier for application programs (and their developers) ● Modern web apps run across hundreds or thousands of hosts (the “cloud computer”), where OS-level abstractions can’t help us ● Many of these OS abstractions are applicable in surprising ways for apps running on the “cloud computer” ● This is NOT a straight-forward mapping, but frameworks such as kubernetes help simplify some of the problems that emerge