Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Fd0f8aaadb6e8addbdb306f142989eba?s=128

Bethany Erskine

September 29, 2016
Tweet

Transcript

  1. DON’T STOP BELIEVING Our Infrastructure as Code Journey at Paperless

    Post Bethany Erskine
  2. A STARTUP IS BORN

  3. None
  4. TODAY 500 VIRTUAL SERVERS 135 EMPLOYEES 60 ENGINEERS 2 OFFICES

    IN 2 COUNTRIES 500K DAILY WWW VISITORS
  5. PAPERLESS POST OPS 2016

  6. Immutable Infrastructure

  7. ZERO to INFRASTRUCTURE AS CODE

  8. AN OPS TEAM IS BORN # Employee #35 Employee #36

    (me!)
  9. None
  10. Ruby on Rails Postgres Redis Memcache Nagios and Munin A

    whole bunch of BASH scripts Gorilla Glue Hopes and Dreams 2011
  11. None
  12. PAPERLESS-CHEF DEPLOY WORKFLOW local git local devtools ON WORKSTATION pp

    chef deploy earth shining_star * tests cookbooks * bumps cookbook versions * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS pp chef deploy earth shining_star —local * checks out tag * uploads any changed cookbooks * uploads updated environment file chefserver
  13. PAPERLESS-CHEF DEPLOY WORKFLOW Deploy to staging Pull Request Code Review!

    Deploy to Production
  14. ~$ bundle exec knife vsphere vm clone "production-september001.pp" --folder "templates"

    --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production-september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json-attributes='{"dc":"carpathia","provider":"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  15. None
  16. None
  17. None
  18. None
  19. NEW MANAGED HOSTING PROVIDER

  20. None
  21. ~$ pp chef create_node ~$ bundle exec knife vsphere vm

    clone "production-september001.pp" --folder "templates" --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production- september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json- attributes='{"dc":"carpathia","provider" :"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  22. ~$ pp chef create_node ~$ pp chef cleanup_node

  23. 200+ VIRTUAL SERVERS 2 HARDWARE DATABASES 1 WEEK BEFORE HURRICANE

    SANDY
  24. None
  25. None
  26. DEVTOOLS & HOTPOT

  27. None
  28. THE HIRING SPREE

  29. None
  30. ~$ pp chef create_node earth pickles —cpu 2 —ram 2

    ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4
  31. UBUNTU GO CENTOS RUBY

  32. None
  33. ~$ pp rails deploy earth 1974_cant_hide_love ~$ pp renderer deploy

    wind 1977_fantasy ~$ pp chef deploy fire 1977_serpentine_fire
  34. PAPERLESS APPLICATION DEPLOY WORKFLOW local git local devtools ON LOCAL:

    pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs database migrations * deploys code to each node returned by Chef API query * restarts application on each node application servers SSH deployparty application servers application servers
  35. None
  36. PAPERLESS DOCKER 1.0

  37. None
  38. ~$ pp noderenderer deploy production fix_api_url

  39. PAPERLESS DOCKER DEPLOY WORKFLOW local git ON LOCAL: git push

    origin shining_star pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins local devtools jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs docker build * pushes tagged docker build to quay.io * SSH into each node returned by Chef search and: * run docker pull * restart docker container via SystemD script application servers SSH deployparty application servers application servers quay.io
  40. PAPERLESS DOCKER 1.0

  41. PAPERLESS DOCKER 1.0

  42. /etc/systemd/system/experiments.service PAPERLESS DOCKER 1.0 /etc/systemd/system/logspout.service /etc/systemd/system/docker-gc.service # experiments - systemd

    unit [Unit] Description=experiments Requires=docker.service After=docker.service [Service] Restart=on-failure # Allow start and restart tasks to take a while as the image download # can cause timeouts. RestartSec=300s TimeoutSec=300s # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run —env-file=/root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > / etc/experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} # Execute deploy-tasks (migrations, etc) if defined on the application # Stop and remove existing container ExecStartPre=-/usr/bin/docker kill experiments ExecStartPre=-/usr/bin/docker rm experiments # s'go ExecStart=/usr/bin/docker run --name experiments --rm -p 5000:41677 --env-file /home/paperless/.experimentsrc quay.io/pp/experiments:${EXPERIMENTS_TAG} # Don't remove the container automatically when stopping, leave it available for inspection ExecStop=-/usr/bin/docker stop experiments [Install] WantedBy=multi-user.target
  43. USES DEPLOYPARTY AS LOCAL TAG REGISTRY CURRENT_TAG SERVICE DeployParty CurrentTag

    app SystemD/Upstart script for application container … # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run --env-file=/ root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > /etc/ experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} …
  44. • WORKS WITH pp chef create_node • WORKS WITH pp

    <application> deploy • GIVES DEVS POWER • BUYS OPS TIME PAPERLESS DOCKER 1.0
  45. docker_application CHEF DEFINITION INSTALLS DOCKER CONFIGURES LOGSPOUT CONFIGURES FILEBEAT MANAGES

    STARTUP SCRIPTS FOR EACH CONTAINER NGINX: FORWARDING AND HTTPS
  46. SENSU CHECKS: 1. Is the application container running? (check_process) 2.

    Is /health API OK? (check_http) docker_application CHEF DEFINITION
  47. docker_application IN A COOKBOOK

  48. pp chef create_node heart simple-docker-app --attributes '{"paperless-docker": {"simple_application_name": "rainbozos"}}' docker_application

    IN A CHEF ROLE
  49. IN PRODUCTION! %%%

  50. None
  51. THE SPAMMERS

  52. None
  53. RAINBOZOS PYTHON + DOCKER + CHEF

  54. None
  55. THE HOSTING PROVIDER WOES

  56. None
  57. None
  58. None
  59. FULL ORCHESTRATION!

  60. None
  61. None
  62. PAPERLESS DOCKER 2.0 • docker_app module in Terraform: • EC2

    instances; Chef Bootstrap • Internal and external Elastic Load Balancers • DNS
  63. AMAZON RDS

  64. ~$ terraform destroy

  65. None
  66. OLD WAY ~$ knife data bag create config keepalived_pickles_production_carpathia ~$

    pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2
  67. ~$ bin/tf --application price-gun --environment staging plan ~$ bin/tf --application

    price-gun --environment staging apply
  68. ~$ pp chef create_node earth pickles —cpu 2 — ram

    2 ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp — cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ bin/tf -a pickles -e earth apply ~$ bin/tf -a pickles -e production apply
  69. ~$ bin/tf -a chefserver -e testing apply

  70. PAPERLESS-TERRAFORM DEPLOY WORKFLOW Deploy to staging Pull Request Code Review!

    Deploy to Production
  71. None
  72. None
  73. None
  74. GROWING PAINS

  75. TERRAFORM + DEVTOOLS + JENKINS?

  76. THE ALERTS

  77. None
  78. ALERTSGALORE@PAPERLESSPOST.COM** ** not an actual email address

  79. None
  80. Ruby on Rails Javascript Postgres Redis Memcache Haproxy Elasticsearch Postgres

    Sensu Graphite & Grafana Filebeat/Elasticsearch/Logstash/Kibana Go Python NodeJS Vphere EC2 S3 Amazon RDS ELB Docker Terraform Hopes Dreams 2016
  81. 2016 Terraformed full-stack services utilizing AWS, Docker and Chef (

    vSphere nodes created with `pp chef create_node`
  82. WHERE WE ARE NOT • Mature CHEF cookbook and role

    testing • Continuous Provisioning / Autoscaling • Continuous Terraform module testing
  83. All the Way, Soon

  84. ? GantryD

  85. (NYC-BASED / REMOTE FRIENDLY!) WE’RE HIRING

  86. (NOT THERE YET!)

  87. IN SUMMARY

  88. None
  89. INFRASTRUCTURE AS CODE

  90. None
  91. None
  92. None
  93. None
  94. TWITTER: @skymob bethany@paperlesspost.com www.paperlesspost.com/jobs THANKS!