Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Bethany Erskine

September 29, 2016
Tweet

More Decks by Bethany Erskine

Other Decks in Technology

Transcript

  1. Ruby on Rails Postgres Redis Memcache Nagios and Munin A

    whole bunch of BASH scripts Gorilla Glue Hopes and Dreams 2011
  2. PAPERLESS-CHEF DEPLOY WORKFLOW local git local devtools ON WORKSTATION pp

    chef deploy earth shining_star * tests cookbooks * bumps cookbook versions * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS pp chef deploy earth shining_star —local * checks out tag * uploads any changed cookbooks * uploads updated environment file chefserver
  3. ~$ bundle exec knife vsphere vm clone "production-september001.pp" --folder "templates"

    --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production-september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json-attributes='{"dc":"carpathia","provider":"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  4. ~$ pp chef create_node ~$ bundle exec knife vsphere vm

    clone "production-september001.pp" --folder "templates" --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production- september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json- attributes='{"dc":"carpathia","provider" :"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  5. ~$ pp chef create_node earth pickles —cpu 2 —ram 2

    ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4
  6. ~$ pp rails deploy earth 1974_cant_hide_love ~$ pp renderer deploy

    wind 1977_fantasy ~$ pp chef deploy fire 1977_serpentine_fire
  7. PAPERLESS APPLICATION DEPLOY WORKFLOW local git local devtools ON LOCAL:

    pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs database migrations * deploys code to each node returned by Chef API query * restarts application on each node application servers SSH deployparty application servers application servers
  8. PAPERLESS DOCKER DEPLOY WORKFLOW local git ON LOCAL: git push

    origin shining_star pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins local devtools jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs docker build * pushes tagged docker build to quay.io * SSH into each node returned by Chef search and: * run docker pull * restart docker container via SystemD script application servers SSH deployparty application servers application servers quay.io
  9. /etc/systemd/system/experiments.service PAPERLESS DOCKER 1.0 /etc/systemd/system/logspout.service /etc/systemd/system/docker-gc.service # experiments - systemd

    unit [Unit] Description=experiments Requires=docker.service After=docker.service [Service] Restart=on-failure # Allow start and restart tasks to take a while as the image download # can cause timeouts. RestartSec=300s TimeoutSec=300s # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run —env-file=/root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > / etc/experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} # Execute deploy-tasks (migrations, etc) if defined on the application # Stop and remove existing container ExecStartPre=-/usr/bin/docker kill experiments ExecStartPre=-/usr/bin/docker rm experiments # s'go ExecStart=/usr/bin/docker run --name experiments --rm -p 5000:41677 --env-file /home/paperless/.experimentsrc quay.io/pp/experiments:${EXPERIMENTS_TAG} # Don't remove the container automatically when stopping, leave it available for inspection ExecStop=-/usr/bin/docker stop experiments [Install] WantedBy=multi-user.target
  10. USES DEPLOYPARTY AS LOCAL TAG REGISTRY CURRENT_TAG SERVICE DeployParty CurrentTag

    app SystemD/Upstart script for application container … # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run --env-file=/ root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > /etc/ experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} …
  11. • WORKS WITH pp chef create_node • WORKS WITH pp

    <application> deploy • GIVES DEVS POWER • BUYS OPS TIME PAPERLESS DOCKER 1.0
  12. SENSU CHECKS: 1. Is the application container running? (check_process) 2.

    Is /health API OK? (check_http) docker_application CHEF DEFINITION
  13. PAPERLESS DOCKER 2.0 • docker_app module in Terraform: • EC2

    instances; Chef Bootstrap • Internal and external Elastic Load Balancers • DNS
  14. OLD WAY ~$ knife data bag create config keepalived_pickles_production_carpathia ~$

    pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2
  15. ~$ pp chef create_node earth pickles —cpu 2 — ram

    2 ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp — cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ bin/tf -a pickles -e earth apply ~$ bin/tf -a pickles -e production apply
  16. Ruby on Rails Javascript Postgres Redis Memcache Haproxy Elasticsearch Postgres

    Sensu Graphite & Grafana Filebeat/Elasticsearch/Logstash/Kibana Go Python NodeJS Vphere EC2 S3 Amazon RDS ELB Docker Terraform Hopes Dreams 2016
  17. 2016 Terraformed full-stack services utilizing AWS, Docker and Chef (

    vSphere nodes created with `pp chef create_node`
  18. WHERE WE ARE NOT • Mature CHEF cookbook and role

    testing • Continuous Provisioning / Autoscaling • Continuous Terraform module testing