Pinterest helps you discover and do what you love. A visual discovery engine at heart, Pinterest guides you through a billion possibilities to quickly discover & get inspired to do something. With over 150MM MAUs across the globe contributing & combing through a billion pins, Pinterest's Infrastructure is built to cater to this scale with very unique requirements -- Today, I'll be talking about how a company operating on the public cloud on VMs since its inception decided to move to containers.
This talk will primarily focus on four things:
1. Pinterest Infrastructure Overview (Offline Compute / Online Serving)
Pinterest was born on AWS. As of today, we operate tens and thousands of instances and process tens and hundreds of PBs of data. Data is the cornerstone of our business where freshness & relevance is key. We will deep dive into our processing & serving stack.
2. VMs vs. Containers - The Pros and Cons
In this section, we will cover the challenges along four key pillars:
a. Developer Velocity - We will discuss the overall job lifecycle workflow i.e build, setup, deploy, operations when using VMs or Containers.
b. Service Reliability - Constraints around resource isolation and standardization across health checks.
c. Infrastructure Governance - Attribution of resources both on utilization & Spend, Quotas
d. Efficiency - Specifically around auto scaling -- our learnings from using ASGs at scale & how this impacts VM vs. Container from an efficiency & operations perspective.
3. Move to Containers
Here we will discuss the use of Docker at Pinterest and more importantly the steps we took around evaluating various orchestration systems. I'll share the various dimensions we evaluated and our learnings when running on a public cloud environment. For ex, docker integration, scheduling, networking, community, stateful support, big data support, security support
4. Vision of the Compute Platform at Pinterest
Finally we will close out with the larger vision (next 18 months) for the Compute Platform at Pinterest.