Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pinterest's Journey from VMs to Containers

Pinterest's Journey from VMs to Containers

Pinterest helps you discover and do what you love. A visual discovery engine at heart, Pinterest guides you through a billion possibilities to quickly discover & get inspired to do something. With over 150MM MAUs across the globe contributing & combing through a billion pins, Pinterest's Infrastructure is built to cater to this scale with very unique requirements -- Today, I'll be talking about how a company operating on the public cloud on VMs since its inception decided to move to containers.

This talk will primarily focus on four things:
1. Pinterest Infrastructure Overview (Offline Compute / Online Serving)
Pinterest was born on AWS. As of today, we operate tens and thousands of instances and process tens and hundreds of PBs of data. Data is the cornerstone of our business where freshness & relevance is key. We will deep dive into our processing & serving stack.

2. VMs vs. Containers - The Pros and Cons
In this section, we will cover the challenges along four key pillars:
a. Developer Velocity - We will discuss the overall job lifecycle workflow i.e build, setup, deploy, operations when using VMs or Containers.
b. Service Reliability - Constraints around resource isolation and standardization across health checks.
c. Infrastructure Governance - Attribution of resources both on utilization & Spend, Quotas
d. Efficiency - Specifically around auto scaling -- our learnings from using ASGs at scale & how this impacts VM vs. Container from an efficiency & operations perspective.

3. Move to Containers
Here we will discuss the use of Docker at Pinterest and more importantly the steps we took around evaluating various orchestration systems. I'll share the various dimensions we evaluated and our learnings when running on a public cloud environment. For ex, docker integration, scheduling, networking, community, stateful support, big data support, security support

4. Vision of the Compute Platform at Pinterest
Finally we will close out with the larger vision (next 18 months) for the Compute Platform at Pinterest.

Micheal Benedict (@micheal)

December 07, 2017
Tweet

More Decks by Micheal Benedict (@micheal)

Other Decks in Technology

Transcript

  1. Journey from VMs to
    Containers
    Micheal Benedict (@micheal)

    Cloud & Data Infrastructure
    Lida Li (@lidali)

    Cloud Management Platform

    View full-size slide

  2. About Pinterest
    Stats & Current State
    Compute Platform
    - Vision / Scope
    - Orchestration Evaluation
    - Moving to Containers
    Future
    1
    2
    3
    4

    View full-size slide

  3. Diverse workloads
    Services

    (ONLINE)
    Number of Services: 103
    Number of Hosts: 104

    View full-size slide

  4. Diverse workloads
    Services

    (ONLINE)
    Batch Jobs

    (OFFLINE)
    Number of Services: 103
    Number of Hosts: 104
    Number of Data Jobs: 105
    Model Training
    Analytics Pipeline

    Block Storage in GBs: 105
    Also used as a analytics backend
    (transactions)

    View full-size slide

  5. Current state of the world
    PINLATER

    (JOB QUEUE / ASYNC
    EXECUTION)
    MONARCH &
    OVERWATCH
    (HADOOP + SPARK)
    LONG RUNNING
    (STATELSS / STATEFUL)
    CLOUD
    TELETRAAN

    (GENERAL COMPUTE
    ON VM)
    DIRECT API
    ACCESS
    RHODMUS TERRAFORM
    JOBS
    (ASYNC / DATA)
    ● Lack of consistent E2E
    Developer Experience
    (Develop, Deploy, Operate)
    ● Tech debt / moving to new
    platforms challenging
    ● Support / Operations
    challenging
    ● Difficult to implement
    Infrastructure Governance
    PROVISIONING
    COMPUTE
    PLATFORMS
    USE-CASES

    View full-size slide

  6. Compute
    Platform

    View full-size slide

  7. fastest path from an
    idea to production,
    without worrying
    about infrastructure
    Vision

    View full-size slide

  8. focus #1
    Simplify
    E2E Dev XP
    What are the steps a developer is required (but not expected) to do when
    building, launching & managing services, batch jobs, etc.?

    View full-size slide

  9. focus #2
    An integrated
    Infra Platform
    What is required to build a reliable, scalable, efficient & well integrated
    infrastructure platform?

    View full-size slide

  10. focus #3
    Infra
    Governance
    Without hampering developer experience and adding opswork,
    What controls are required to effectively utilize & manage Infrastructure

    View full-size slide

  11. SETUP TEST & BUILD
    UNIT TEST
    APP IMAGE
    MANGEMENT
    OPERATIONS
    METRICS
    LOGS
    TRACING
    DEPLOY & RELEASE
    WORKFLOW
    MANAGEMENT
    JOB
    SUBMISSION
    INTEGRATION
    TEST
    OWNERSHIP
    SCAFFOLDING
    ROLES, KEYS &
    SECRETS
    RESOURCE
    MANAGEMENT
    QUOTA
    AMI
    MANAGEMENT
    CLUSTER
    PROVISIONING
    METERING
    HEALTH
    CHECK
    JOB STATUS
    JOB CONFIG
    Scope

    View full-size slide

  12. H1 2016
    H2 2016
    H1 2017
    H2 2017
    Phase 2: Productionize Docker & Adoption
    • Metric, logging, security and high
    availability support.
    • Fully production ready and over one
    hundred services migrated (including
    major API fleet)
    Phase 1: Docker MVP
    • Developer Workflow
    • Image Management
    • Integration w/ existing security 

    & networking systems
    • First Production Service migrated
    Containers @Pinterest Kickoff
    Container Orchestration @Pinterest
    Kickoff
    • Orchestration Evaluation
    • MVP build & Operate production cluster
    for a use-case
    H1 2018
    Timeline

    View full-size slide

  13. H1 2016
    H2 2016
    H1 2017
    H2 2017
    Phase 2: Productionize Docker & Adoption
    • Metric, logging, security and high
    availability support.
    • Fully production ready and over one
    hundred services migrated (including
    major API fleet)
    Phase 1: Docker MVP
    • Developer Workflow
    • Image Management
    • Integration w/ existing security 

    & networking systems
    • First Production Service migrated
    Containers @Pinterest Kickoff
    Container Orchestration @Pinterest
    Kickoff
    • Orchestration Evaluation
    • MVP build & Operate production cluster
    for a use-case
    H1 2018
    Timeline

    View full-size slide

  14. CHOICES POC
    CRITERIA OUTCOME
    Container Orchestration
    Evaluation Framework

    View full-size slide

  15. CHOICES POC
    CRITERIA OUTCOME
    ● Resource and Task Scheduling (Flexibility, Multi-Tenancy, Extensibility etc.)
    ● Scalability and Performance
    ● Integration Cost
    ● Docker Support, Sidecar support and Runtime extensibility
    ● Network Support on AWS*
    ● Security Support on AWS*
    ● Stateful Service Support
    ● Ecosystem and Community
    ● Cluster Operations & Support
    Container Orchestration

    View full-size slide

  16. Container Orchestration
    CHOICES POC
    CRITERIA OUTCOME
    Custom
    Scheduler

    View full-size slide

  17. CHOICES POC
    CRITERIA OUTCOME
    Container Orchestration

    View full-size slide

  18. Container Orchestration
    CHOICES POC
    CRITERIA OUTCOME

    View full-size slide

  19. H2 2016
    H1 2017
    Phase 2: Productionize Docker & Adoption
    • Metric, logging, security and high
    availability support.
    • Fully production ready and over one
    hundred services migrated (including
    major API fleet)
    Phase 1: Docker MVP
    • Developer Workflow
    • Image Management
    • Integration w/ existing security 

    & networking systems
    • First Production Service migrated
    Timeline
    H1 2016
    H2 2017
    Containers @Pinterest Kickoff
    Container Orchestration @Pinterest
    Kickoff
    • Orchestration Evaluation
    • MVP build & Operate production cluster
    for a use-case
    H1 2018

    View full-size slide

  20. Teletraan
    Monit Supervisor Upstart
    Service AMI
    Puppet
    Service Service Service
    Cron
    Jobs
    Crond
    ● Multiple AMIs, complex management
    ● End users did puppet authoring & testing
    ● Unpredictability around Puppet runs
    ● Disparate process management

    (monit, upstart, supervisor)
    Base AMI
    Launching a Service
    Before Containerization Project

    View full-size slide

  21. Unified (single) AMI
    Container Container Container Container
    Teletraan + Telefig
    Container Engine (Docker)
    ● Single AMI for Containers
    ● No puppet authoring!
    ● Unified process management
    ● Immutable infrastructure 

    & deterministic behavior
    Launching a Service
    Post Containerization Project

    View full-size slide

  22. Code Build
    Test
    Developer 

    Workflow
    Under the hood
    Developer Workflow - Docker Containers
    ● Introduce Pinterest
    Service Description
    Language (PSDL)
    ● Optimize Image building
    on large shared image
    ● ECR and a Self Hosted
    Registry (HA)
    version: 1
    myservice:
    docker:
    image: myservice-server
    user: prod
    environment:
    - CONFIG_FILE=config/myservice.props
    sidecars:
    zum:
    deps: #sidecar container
    - myservice.dep
    singer: #sidecar container
    property_sets:
    - myservice.singer

    View full-size slide

  23. Logging
    Service
    Discovery
    Metrics
    AWS Service
    Service
    Proxy
    Config Files
    Application
    Secrets
    Management
    Process
    Control
    Application Runtime
    Under the hood
    Application Runtime - Docker Containers
    ● Manage container run order for a service & its
    sidecars
    ● Default —net=host
    ● Docker engine running with --live-
    restore and overlay2 file system
    ● Local Image cleanup (garbage collection)
    ● Parallel prefetching images for deploy
    performance

    View full-size slide

  24. 75%
    Hosts
    State of Migration
    Stateless Services Migrated

    View full-size slide

  25. Team Status:

    View full-size slide

  26. Learnings
    ● Run Containers and Non-Containers together with adjustable ratio
    ● Ability to measure & compare metrics
    ● Automate deploy migration & Run time configuration Validation (IAM,
    Security Groups, Service Discovery)
    ● Understand Company / Team Dynamics
    ● Migrate a complex service early

    View full-size slide

  27. H1 2016
    H2 2016
    H1 2017
    H2 2017
    Phase 2: Productionize Docker & Adoption
    • Metric, logging, security and high
    availability support.
    • Fully production ready and over one
    hundred services migrated (including
    major API fleet)
    Phase 1: Docker MVP
    • Developer Workflow
    • Image Management
    • Integration w/ existing security 

    & networking systems
    • First Production Service migrated
    Containers @Pinterest Kickoff
    Container Orchestration @Pinterest
    Kickoff
    • Orchestration Evaluation
    • MVP build & Operate production cluster
    for a use-case
    H1 2018
    Timeline

    View full-size slide

  28. Container Orchestration
    Proof of Concept (MVP)

    View full-size slide

  29. Networking
    High Level
    ● Use ENI natively supported
    by EC2
    ● Also support Different CNIs
    plugins (Configured by Pod
    annotations)
    ● Support AWS IAM role and
    Security Group on ENI with
    our own meta-proxy
    ● Collaborating w/ AWS on
    amazon-vpc-cni-k8s

    View full-size slide

  30. ● Proxy CNI invokes the
    daemon to get the actual
    CNI and parameters
    ● Also run customized
    commands before and
    after CNI execution
    ● Pod spec has annotation
    specify the network
    mode - pinterest.com/
    networkmode: xxx
    Networking
    Pod Network Setup (Proxy CNI Daemon/Plugin)

    View full-size slide

  31. IAM
    Pod IAM Setup
    ● Role set as annotation of Pod
    ● IPTables rule redirect to local
    metaproxy (Drome)
    ● Drome consult Kubelet and get
    token from an external role
    assume service

    View full-size slide

  32. SETUP TEST & BUILD
    UNIT TEST
    APP IMAGE
    MANGEMENT
    OPERATIONS
    METRICS
    LOGS
    TRACING
    DEPLOY & RELEASE
    WORKFLOW
    MANAGEMENT
    JOB
    SUBMISSION
    INTEGRATION
    TEST
    OWNERSHIP
    SCAFFOLDING
    ROLES, KEYS &
    SECRETS
    RESOURCE
    MANAGEMENT
    QUOTA
    AMI
    MANAGEMENT
    CLUSTER
    PROVISIONING
    METERING
    HEALTH
    CHECK
    JOB STATUS
    JOB CONFIG
    Scope

    View full-size slide

  33. H1 2018
    ● Productionize the Cluster (Setup & Operations)
    ● Adoption - Initial use-case Jenkins, non-critical long running services
    ● Experiment - Spark & TensorFlow on Cabernets
    ● Job Definition Abstraction & Job Submission service (Data Jobs & Long
    running tasks)
    ● Service Identity Management & Resource Metering

    View full-size slide