Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Operating Systems in Cloud City

Operating Systems in Cloud City

Workshop for undergrad operating systems students, showing how operating system concepts apply when building cloud applications.


John Arthorne

May 04, 2018


  1. Copyright Shopify 2018 Operating Systems in Cloud City John Arthorne

    @jarthorne Shopify Production Engineering May the Fourth, 2018
  2. Copyright Shopify 2018

  3. Copyright Shopify 2018 What is a Computer?

  4. Copyright Shopify 2018 ROM Network CPU RAM I/O BUS Disk

  5. Copyright Shopify 2018 What is an Operating System?

  6. Copyright Shopify 2018 ROM Network CPU RAM I/O BUS Disk

    Virtual Memory Process Model Scheduler File System Sockets Device Controller Memory Manager
  7. Copyright Shopify 2018 Applications in the Cloud • Applications in

    the cloud tend to span across multiple processes and computers • Applications are broken into specialized parts • Each part has duplicate copies, either for resiliency or scale • A traditional operating system can’t reason about or manage them
  8. Copyright Shopify 2018 What are some parts that make up

    a cloud app?
  9. Copyright Shopify 2018 Load Balancer HTTPS Web Server Redis/ Memcached

    Kafka MySQL Job Server webhooks
  10. Copyright Shopify 2018 Processes in the Cloud • The unit

    of computation on the cloud computer are containers (eg Docker) • Containers are grouped into pods, similar to a group of threads forming a process • Inter-process communication is largely over TCP/IP network connections • Contention and coordination between containers is managed externally ◦ Eg, we use ZooKeeper for service discovery and routing
  11. Copyright Shopify 2018 Producer/Consumer and Semaphores • Semaphores were first

    invented to prevent a producer writing to a buffer faster than a consumer could read • Producer/consumer problems arise in many places in cloud apps ◦ Processing webhooks ◦ Writing to a 3rd party API faster than it can receive ◦ Job queues “consuming” the available worker pool • In many cases we don’t have shared state between producer and consumer that can be used to store a shared semaphore • Sometimes have other feedback mechanisms such as HTTP 429 (“too many requests”) • When there is no feedback loop we can use a producer-side semaphore (circuit breaker)
  12. Copyright Shopify 2018 Semaphore example: job queues • Limited number

    of available job workers • If all workers are busy, we can have starvation of waiting jobs • Notably, payment jobs are used to complete purchases • A large flash sale could exhaust workers for all shops • We use a semaphore to limit number of payment jobs running per pod
  13. Copyright Shopify 2018 Local Memory in the Cloud • Containers

    need memory allocation just like normal processes • Kubernetes scheduler is a cross between a memory manager and a process scheduler • Scheduler allocates containers to physical computers (nodes) based on memory and CPU requirements (also potentially disk and other resources) • Containers can be swapped and evicted just like memory blocks and processes on a local computer
  14. Copyright Shopify 2018 Scheduling CPU vs Memory What are some

    key differences between allocating CPU and memory? How can we handle exceeded limits in each case? How do we make maximum use of the underlying resource?
  15. Copyright Shopify 2018 Shared Memory in the Cloud • Containers

    are ephemeral and disposable ◦ Eviction means killing the container • Any valuable state needs to be stored externally • Apps like Redis and memcached act like “RAM” of the cloud computer • Two main uses for shared memory: caches and queues • Cache memory is not allocated to a container, it is typically MRU • Queue memory is segmented by priority
  16. Copyright Shopify 2018 Batch Scheduling: Jobs • Long running work

    is performed by background jobs • Dedicated group of containers constantly running jobs • We use resque which is a job scheduler built on redis • FIFO with multiple queues based on priority • Big jobs must be preemptible ◦ We built a custom framework for Shopify core: BackgroundQueue::Iteration
  17. Copyright Shopify 2018 Interactive Scheduling: Load Balancers • Our core

    apps receive millions of HTTP requests/minute at all times • We need to spread these requests out to handle the load ◦ Much like an OS process scheduler allocating CPU time for processes • We use a mixture of commonly used algorithms ◦ Exponentially Weighted Moving Average (EWMA) - most recently least loaded ◦ Round Robin (RR) ◦ Power of 2 Choices (P2C) - Pick two at random, select least loaded
  18. Copyright Shopify 2018 Story Time: Load Balancer Scheduling

  19. Copyright Shopify 2018 Printer Spools • What problems come up

    when processes need to print?
  20. Copyright Shopify 2018 Spooling and Kafka • Printer spools mediate

    between fast processors and slow I/O • Spools allow many things to write to one consumer in an orderly way • Kafka is a messaging system used heavily at Shopify, for similar reasons • Producers can deliver messages to a local Kafka client quickly • Kafka then sends them to a server, where they are spooled by topic • Consumers can consume from the spool/topic at their own pace
  21. Copyright Shopify 2018 RAID and ElasticSearch • Redundant Array of

    Independent Disks (RAID) is a way of configuring disk drives • Distributed databases use a similar strategy to store multiple copies • ElasticSearch is one example that we use. It stores documents in multiple nodes, somewhat like block-level striping in RAID-5 • When one node (disk) fails, nothing is lost, but the entire cluster (drive) has slower performance while it rebalances
  22. Copyright Shopify 2018 Summary • Operating systems provide a toolkit

    of abstractions to make life easier for application programs (and their developers) • Modern web apps run across hundreds or thousands of hosts (the “cloud computer”), where OS-level abstractions can’t help us • Many of these OS abstractions are applicable in surprising ways for apps running on the “cloud computer” • This is NOT a straight-forward mapping, but frameworks such as kubernetes help simplify some of the problems that emerge
  23. Copyright Shopify 2018 A little bit of history... David Clark

    Post-PC Internet April 1999 http://bit.ly/post-pc-internet