Pinterest's Journey from VMs to Containers

Pinterest helps you discover and do what you love. A visual discovery engine at heart, Pinterest guides you through a billion possibilities to quickly discover & get inspired to do something. With over 150MM MAUs across the globe contributing & combing through a billion pins, Pinterest's Infrastructure is built to cater to this scale with very unique requirements -- Today, I'll be talking about how a company operating on the public cloud on VMs since its inception decided to move to containers.

This talk will primarily focus on four things:
1. Pinterest Infrastructure Overview (Offline Compute / Online Serving)
Pinterest was born on AWS. As of today, we operate tens and thousands of instances and process tens and hundreds of PBs of data. Data is the cornerstone of our business where freshness & relevance is key. We will deep dive into our processing & serving stack.

2. VMs vs. Containers - The Pros and Cons
In this section, we will cover the challenges along four key pillars:
a. Developer Velocity - We will discuss the overall job lifecycle workflow i.e build, setup, deploy, operations when using VMs or Containers.
b. Service Reliability - Constraints around resource isolation and standardization across health checks.
c. Infrastructure Governance - Attribution of resources both on utilization & Spend, Quotas
d. Efficiency - Specifically around auto scaling -- our learnings from using ASGs at scale & how this impacts VM vs. Container from an efficiency & operations perspective.

3. Move to Containers
Here we will discuss the use of Docker at Pinterest and more importantly the steps we took around evaluating various orchestration systems. I'll share the various dimensions we evaluated and our learnings when running on a public cloud environment. For ex, docker integration, scheduling, networking, community, stateful support, big data support, security support

4. Vision of the Compute Platform at Pinterest
Finally we will close out with the larger vision (next 18 months) for the Compute Platform at Pinterest.

Transcript

1. Journey from VMs to Containers Micheal Benedict (@micheal)
 Cloud & Data Infrastructure Lida Li (@lidali)
 Cloud Management Platform
2. About Pinterest Stats & Current State Compute Platform - Vision / Scope - Orchestration Evaluation - Moving to Containers Future 1 2 3 4
10. Diverse workloads Services
 (ONLINE) Number of Services: 103 Number of Hosts: 104
11. Diverse workloads Services
 (ONLINE) Batch Jobs
 (OFFLINE) Number of Services: 103 Number of Hosts: 104 Number of Data Jobs: 105 Model Training Analytics Pipeline 
 Block Storage in GBs: 105 Also used as a analytics backend (transactions)
12. Current state of the world PINLATER
 (JOB QUEUE / ASYNC EXECUTION) MONARCH & OVERWATCH (HADOOP + SPARK) LONG RUNNING (STATELSS / STATEFUL) CLOUD TELETRAAN
 (GENERAL COMPUTE ON VM) DIRECT API ACCESS RHODMUS TERRAFORM JOBS (ASYNC / DATA) • Lack of consistent E2E Developer Experience (Develop, Deploy, Operate) • Tech debt / moving to new platforms challenging • Support / Operations challenging • Difficult to implement Infrastructure Governance PROVISIONING COMPUTE PLATFORMS USE-CASES
13. Compute Platform
14. fastest path from an idea to production, without worrying about infrastructure Vision
15. focus #1 Simplify E2E Dev XP What are the steps a developer is required (but not expected) to do when building, launching & managing services, batch jobs, etc.?
16. focus #2 An integrated Infra Platform What is required to build a reliable, scalable, efficient & well integrated infrastructure platform?
17. focus #3 Infra Governance Without hampering developer experience and adding opswork, What controls are required to effectively utilize & manage Infrastructure
18. SETUP TEST & BUILD UNIT TEST APP IMAGE MANGEMENT OPERATIONS METRICS LOGS TRACING DEPLOY & RELEASE WORKFLOW MANAGEMENT JOB SUBMISSION INTEGRATION TEST OWNERSHIP SCAFFOLDING ROLES, KEYS & SECRETS RESOURCE MANAGEMENT QUOTA AMI MANAGEMENT CLUSTER PROVISIONING METERING HEALTH CHECK JOB STATUS JOB CONFIG Scope
19. H1 2016 H2 2016 H1 2017 H2 2017 Phase 2: Productionize Docker & Adoption • Metric, logging, security and high availability support. • Fully production ready and over one hundred services migrated (including major API fleet) Phase 1: Docker MVP • Developer Workflow • Image Management • Integration w/ existing security 
 & networking systems • First Production Service migrated Containers @Pinterest Kickoff Container Orchestration @Pinterest Kickoff • Orchestration Evaluation • MVP build & Operate production cluster for a use-case H1 2018 Timeline
20. H1 2016 H2 2016 H1 2017 H2 2017 Phase 2: Productionize Docker & Adoption • Metric, logging, security and high availability support. • Fully production ready and over one hundred services migrated (including major API fleet) Phase 1: Docker MVP • Developer Workflow • Image Management • Integration w/ existing security 
 & networking systems • First Production Service migrated Containers @Pinterest Kickoff Container Orchestration @Pinterest Kickoff • Orchestration Evaluation • MVP build & Operate production cluster for a use-case H1 2018 Timeline
21. CHOICES POC CRITERIA OUTCOME Container Orchestration Evaluation Framework
22. CHOICES POC CRITERIA OUTCOME • Resource and Task Scheduling (Flexibility, Multi-Tenancy, Extensibility etc.) • Scalability and Performance • Integration Cost • Docker Support, Sidecar support and Runtime extensibility • Network Support on AWS* • Security Support on AWS* • Stateful Service Support • Ecosystem and Community • Cluster Operations & Support Container Orchestration
23. Container Orchestration CHOICES POC CRITERIA OUTCOME Custom Scheduler
24. CHOICES POC CRITERIA OUTCOME Container Orchestration
25. Container Orchestration CHOICES POC CRITERIA OUTCOME
26. H2 2016 H1 2017 Phase 2: Productionize Docker & Adoption • Metric, logging, security and high availability support. • Fully production ready and over one hundred services migrated (including major API fleet) Phase 1: Docker MVP • Developer Workflow • Image Management • Integration w/ existing security 
 & networking systems • First Production Service migrated Timeline H1 2016 H2 2017 Containers @Pinterest Kickoff Container Orchestration @Pinterest Kickoff • Orchestration Evaluation • MVP build & Operate production cluster for a use-case H1 2018
27. Teletraan Monit Supervisor Upstart Service AMI Puppet Service Service Service Cron Jobs Crond • Multiple AMIs, complex management • End users did puppet authoring & testing • Unpredictability around Puppet runs • Disparate process management
 (monit, upstart, supervisor) Base AMI Launching a Service Before Containerization Project
28. Unified (single) AMI Container Container Container Container Teletraan + Telefig Container Engine (Docker) • Single AMI for Containers • No puppet authoring! • Unified process management • Immutable infrastructure 
 & deterministic behavior Launching a Service Post Containerization Project
29. Code Build Test Developer 
 Workflow Under the hood Developer Workflow - Docker Containers • Introduce Pinterest Service Description Language (PSDL) • Optimize Image building on large shared image • ECR and a Self Hosted Registry (HA) version: 1 myservice: docker: image: myservice-server user: prod environment: - CONFIG_FILE=config/myservice.props sidecars: zum: deps: #sidecar container - myservice.dep singer: #sidecar container property_sets: - myservice.singer
30. Logging Service Discovery Metrics AWS Service Service Proxy Config Files Application Secrets Management Process Control Application Runtime Under the hood Application Runtime - Docker Containers • Manage container run order for a service & its sidecars • Default —net=host • Docker engine running with --live- restore and overlay2 file system • Local Image cleanup (garbage collection) • Parallel prefetching images for deploy performance
31. 75% Hosts State of Migration Stateless Services Migrated
32. Team Status:
33. Learnings • Run Containers and Non-Containers together with adjustable ratio • Ability to measure & compare metrics • Automate deploy migration & Run time configuration Validation (IAM, Security Groups, Service Discovery) • Understand Company / Team Dynamics • Migrate a complex service early
34. H1 2016 H2 2016 H1 2017 H2 2017 Phase 2: Productionize Docker & Adoption • Metric, logging, security and high availability support. • Fully production ready and over one hundred services migrated (including major API fleet) Phase 1: Docker MVP • Developer Workflow • Image Management • Integration w/ existing security 
 & networking systems • First Production Service migrated Containers @Pinterest Kickoff Container Orchestration @Pinterest Kickoff • Orchestration Evaluation • MVP build & Operate production cluster for a use-case H1 2018 Timeline
35. Container Orchestration Proof of Concept (MVP)
36. Networking High Level • Use ENI natively supported by EC2 • Also support Different CNIs plugins (Configured by Pod annotations) • Support AWS IAM role and Security Group on ENI with our own meta-proxy • Collaborating w/ AWS on amazon-vpc-cni-k8s
37. • Proxy CNI invokes the daemon to get the actual CNI and parameters • Also run customized commands before and after CNI execution • Pod spec has annotation specify the network mode - pinterest.com/ networkmode: xxx Networking Pod Network Setup (Proxy CNI Daemon/Plugin)
38. IAM Pod IAM Setup • Role set as annotation of Pod • IPTables rule redirect to local metaproxy (Drome) • Drome consult Kubelet and get token from an external role assume service
39. Future
40. SETUP TEST & BUILD UNIT TEST APP IMAGE MANGEMENT OPERATIONS METRICS LOGS TRACING DEPLOY & RELEASE WORKFLOW MANAGEMENT JOB SUBMISSION INTEGRATION TEST OWNERSHIP SCAFFOLDING ROLES, KEYS & SECRETS RESOURCE MANAGEMENT QUOTA AMI MANAGEMENT CLUSTER PROVISIONING METERING HEALTH CHECK JOB STATUS JOB CONFIG Scope
41. H1 2018 • Productionize the Cluster (Setup & Operations) • Adoption - Initial use-case Jenkins, non-critical long running services • Experiment - Spark & TensorFlow on Cabernets • Job Definition Abstraction & Job Submission service (Data Jobs & Long running tasks) • Service Identity Management & Resource Metering
42. Thanks!