Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

Fd0f8aaadb6e8addbdb306f142989eba?s=128

Bethany Erskine

September 29, 2016
Tweet

More Decks by Bethany Erskine

Other Decks in Technology

Transcript

  1. DON’T STOP BELIEVING Our Infrastructure as Code Journey at Paperless

    Post Bethany Erskine
  2. A STARTUP IS BORN

  3. None
  4. TODAY 500 VIRTUAL SERVERS 135 EMPLOYEES 60 ENGINEERS 2 OFFICES

    IN 2 COUNTRIES 500K DAILY WWW VISITORS
  5. PAPERLESS POST OPS 2016

  6. Immutable Infrastructure

  7. ZERO to INFRASTRUCTURE AS CODE

  8. AN OPS TEAM IS BORN # Employee #35 Employee #36

    (me!)
  9. None
  10. Ruby on Rails Postgres Redis Memcache Nagios and Munin A

    whole bunch of BASH scripts Gorilla Glue Hopes and Dreams 2011
  11. None
  12. PAPERLESS-CHEF DEPLOY WORKFLOW local git local devtools ON WORKSTATION pp

    chef deploy earth shining_star * tests cookbooks * bumps cookbook versions * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS pp chef deploy earth shining_star —local * checks out tag * uploads any changed cookbooks * uploads updated environment file chefserver
  13. PAPERLESS-CHEF DEPLOY WORKFLOW Deploy to staging Pull Request Code Review!

    Deploy to Production
  14. ~$ bundle exec knife vsphere vm clone "production-september001.pp" --folder "templates"

    --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production-september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json-attributes='{"dc":"carpathia","provider":"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  15. None
  16. None
  17. None
  18. None
  19. NEW MANAGED HOSTING PROVIDER

  20. None
  21. ~$ pp chef create_node ~$ bundle exec knife vsphere vm

    clone "production-september001.pp" --folder "templates" --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production- september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json- attributes='{"dc":"carpathia","provider" :"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
  22. ~$ pp chef create_node ~$ pp chef cleanup_node

  23. 200+ VIRTUAL SERVERS 2 HARDWARE DATABASES 1 WEEK BEFORE HURRICANE

    SANDY
  24. None
  25. None
  26. DEVTOOLS & HOTPOT

  27. None
  28. THE HIRING SPREE

  29. None
  30. ~$ pp chef create_node earth pickles —cpu 2 —ram 2

    ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4
  31. UBUNTU GO CENTOS RUBY

  32. None
  33. ~$ pp rails deploy earth 1974_cant_hide_love ~$ pp renderer deploy

    wind 1977_fantasy ~$ pp chef deploy fire 1977_serpentine_fire
  34. PAPERLESS APPLICATION DEPLOY WORKFLOW local git local devtools ON LOCAL:

    pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs database migrations * deploys code to each node returned by Chef API query * restarts application on each node application servers SSH deployparty application servers application servers
  35. None
  36. PAPERLESS DOCKER 1.0

  37. None
  38. ~$ pp noderenderer deploy production fix_api_url

  39. PAPERLESS DOCKER DEPLOY WORKFLOW local git ON LOCAL: git push

    origin shining_star pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins local devtools jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs docker build * pushes tagged docker build to quay.io * SSH into each node returned by Chef search and: * run docker pull * restart docker container via SystemD script application servers SSH deployparty application servers application servers quay.io
  40. PAPERLESS DOCKER 1.0

  41. PAPERLESS DOCKER 1.0

  42. /etc/systemd/system/experiments.service PAPERLESS DOCKER 1.0 /etc/systemd/system/logspout.service /etc/systemd/system/docker-gc.service # experiments - systemd

    unit [Unit] Description=experiments Requires=docker.service After=docker.service [Service] Restart=on-failure # Allow start and restart tasks to take a while as the image download # can cause timeouts. RestartSec=300s TimeoutSec=300s # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run —env-file=/root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > / etc/experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} # Execute deploy-tasks (migrations, etc) if defined on the application # Stop and remove existing container ExecStartPre=-/usr/bin/docker kill experiments ExecStartPre=-/usr/bin/docker rm experiments # s'go ExecStart=/usr/bin/docker run --name experiments --rm -p 5000:41677 --env-file /home/paperless/.experimentsrc quay.io/pp/experiments:${EXPERIMENTS_TAG} # Don't remove the container automatically when stopping, leave it available for inspection ExecStop=-/usr/bin/docker stop experiments [Install] WantedBy=multi-user.target
  43. USES DEPLOYPARTY AS LOCAL TAG REGISTRY CURRENT_TAG SERVICE DeployParty CurrentTag

    app SystemD/Upstart script for application container … # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run --env-file=/ root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > /etc/ experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} …
  44. • WORKS WITH pp chef create_node • WORKS WITH pp

    <application> deploy • GIVES DEVS POWER • BUYS OPS TIME PAPERLESS DOCKER 1.0
  45. docker_application CHEF DEFINITION INSTALLS DOCKER CONFIGURES LOGSPOUT CONFIGURES FILEBEAT MANAGES

    STARTUP SCRIPTS FOR EACH CONTAINER NGINX: FORWARDING AND HTTPS
  46. SENSU CHECKS: 1. Is the application container running? (check_process) 2.

    Is /health API OK? (check_http) docker_application CHEF DEFINITION
  47. docker_application IN A COOKBOOK

  48. pp chef create_node heart simple-docker-app --attributes '{"paperless-docker": {"simple_application_name": "rainbozos"}}' docker_application

    IN A CHEF ROLE
  49. IN PRODUCTION! %%%

  50. None
  51. THE SPAMMERS

  52. None
  53. RAINBOZOS PYTHON + DOCKER + CHEF

  54. None
  55. THE HOSTING PROVIDER WOES

  56. None
  57. None
  58. None
  59. FULL ORCHESTRATION!

  60. None
  61. None
  62. PAPERLESS DOCKER 2.0 • docker_app module in Terraform: • EC2

    instances; Chef Bootstrap • Internal and external Elastic Load Balancers • DNS
  63. AMAZON RDS

  64. ~$ terraform destroy

  65. None
  66. OLD WAY ~$ knife data bag create config keepalived_pickles_production_carpathia ~$

    pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2
  67. ~$ bin/tf --application price-gun --environment staging plan ~$ bin/tf --application

    price-gun --environment staging apply
  68. ~$ pp chef create_node earth pickles —cpu 2 — ram

    2 ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp — cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ bin/tf -a pickles -e earth apply ~$ bin/tf -a pickles -e production apply
  69. ~$ bin/tf -a chefserver -e testing apply

  70. PAPERLESS-TERRAFORM DEPLOY WORKFLOW Deploy to staging Pull Request Code Review!

    Deploy to Production
  71. None
  72. None
  73. None
  74. GROWING PAINS

  75. TERRAFORM + DEVTOOLS + JENKINS?

  76. THE ALERTS

  77. None
  78. ALERTSGALORE@PAPERLESSPOST.COM** ** not an actual email address

  79. None
  80. Ruby on Rails Javascript Postgres Redis Memcache Haproxy Elasticsearch Postgres

    Sensu Graphite & Grafana Filebeat/Elasticsearch/Logstash/Kibana Go Python NodeJS Vphere EC2 S3 Amazon RDS ELB Docker Terraform Hopes Dreams 2016
  81. 2016 Terraformed full-stack services utilizing AWS, Docker and Chef (

    vSphere nodes created with `pp chef create_node`
  82. WHERE WE ARE NOT • Mature CHEF cookbook and role

    testing • Continuous Provisioning / Autoscaling • Continuous Terraform module testing
  83. All the Way, Soon

  84. ? GantryD

  85. (NYC-BASED / REMOTE FRIENDLY!) WE’RE HIRING

  86. (NOT THERE YET!)

  87. IN SUMMARY

  88. None
  89. INFRASTRUCTURE AS CODE

  90. None
  91. None
  92. None
  93. None
  94. TWITTER: @skymob bethany@paperlesspost.com www.paperlesspost.com/jobs THANKS!