Don’t Stop Believing: Our Infrastructure as Code Journey at Paperless Post

http://www.automacon.io/presentation/bethany-erskine/

1. DON’T STOP BELIEVING Our Infrastructure as Code Journey at Paperless Post Bethany Erskine
2. A STARTUP IS BORN
4. TODAY 500 VIRTUAL SERVERS 135 EMPLOYEES 60 ENGINEERS 2 OFFICES IN 2 COUNTRIES 500K DAILY WWW VISITORS
5. PAPERLESS POST OPS 2016
6. Immutable Infrastructure
7. ZERO to INFRASTRUCTURE AS CODE
8. AN OPS TEAM IS BORN # Employee #35 Employee #36 (me!)
10. Ruby on Rails Postgres Redis Memcache Nagios and Munin A whole bunch of BASH scripts Gorilla Glue Hopes and Dreams 2011
12. PAPERLESS-CHEF DEPLOY WORKFLOW local git local devtools ON WORKSTATION pp chef deploy earth shining_star * tests cookbooks * bumps cookbook versions * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS pp chef deploy earth shining_star —local * checks out tag * uploads any changed cookbooks * uploads updated environment file chefserver
13. PAPERLESS-CHEF DEPLOY WORKFLOW Deploy to staging Pull Request Code Review! Deploy to Production
14. ~$ bundle exec knife vsphere vm clone "production-september001.pp" --folder "templates" --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production-september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json-attributes='{"dc":"carpathia","provider":"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
19. NEW MANAGED HOSTING PROVIDER
21. ~$ pp chef create_node ~$ bundle exec knife vsphere vm clone "production-september001.pp" --folder "templates" --template "centos63-08072012" --dest-folder "production" --datastore "tier2b" --ccpu "4" --cram "10" --cspec "paperless" --resource-pool "Paperless" --bootstrap -N "production- september001.pp" --bootstrap-version 11.16.4 -r "role[september]" -E “production" --distro "pp-el6" --json- attributes='{"dc":"carpathia","provider" :"vsphere"}' --cips 10.52.13.136/27 --cvlan DB --cgw 10.52.13.129
22. ~$ pp chef create_node ~$ pp chef cleanup_node
23. 200+ VIRTUAL SERVERS 2 HARDWARE DATABASES 1 WEEK BEFORE HURRICANE SANDY
26. DEVTOOLS & HOTPOT
28. THE HIRING SPREE
30. ~$ pp chef create_node earth pickles —cpu 2 —ram 2 ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles —cpu 4 —ram 4
31. UBUNTU GO CENTOS RUBY
33. ~$ pp rails deploy earth 1974_cant_hide_love ~$ pp renderer deploy wind 1977_fantasy ~$ pp chef deploy fire 1977_serpentine_fire
34. PAPERLESS APPLICATION DEPLOY WORKFLOW local git local devtools ON LOCAL: pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs database migrations * deploys code to each node returned by Chef API query * restarts application on each node application servers SSH deployparty application servers application servers
36. PAPERLESS DOCKER 1.0
38. ~$ pp noderenderer deploy production fix_api_url
39. PAPERLESS DOCKER DEPLOY WORKFLOW local git ON LOCAL: git push origin shining_star pp application deploy earth shining_star * tags branch & pushes back to github * triggers deploy on Jenkins local devtools jenkins devtools ON JENKINS: pp chef deploy earth shining_star —local * checks out tag * runs docker build * pushes tagged docker build to quay.io * SSH into each node returned by Chef search and: * run docker pull * restart docker container via SystemD script application servers SSH deployparty application servers application servers quay.io
40. PAPERLESS DOCKER 1.0
41. PAPERLESS DOCKER 1.0
42. /etc/systemd/system/experiments.service PAPERLESS DOCKER 1.0 /etc/systemd/system/logspout.service /etc/systemd/system/docker-gc.service # experiments - systemd unit [Unit] Description=experiments Requires=docker.service After=docker.service [Service] Restart=on-failure # Allow start and restart tasks to take a while as the image download # can cause timeouts. RestartSec=300s TimeoutSec=300s # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run —env-file=/root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > / etc/experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} # Execute deploy-tasks (migrations, etc) if defined on the application # Stop and remove existing container ExecStartPre=-/usr/bin/docker kill experiments ExecStartPre=-/usr/bin/docker rm experiments # s'go ExecStart=/usr/bin/docker run --name experiments --rm -p 5000:41677 --env-file /home/paperless/.experimentsrc quay.io/pp/experiments:${EXPERIMENTS_TAG} # Don't remove the container automatically when stopping, leave it available for inspection ExecStop=-/usr/bin/docker stop experiments [Install] WantedBy=multi-user.target
43. USES DEPLOYPARTY AS LOCAL TAG REGISTRY CURRENT_TAG SERVICE DeployParty CurrentTag app SystemD/Upstart script for application container … # Fetch current tag from pushparty ExecStartPre=/bin/bash -c 'echo "EXPERIMENTS_TAG=`/usr/bin/docker run --env-file=/ root/.pp_token quay.io/pp/current_tag -app experiments -env production`" > /etc/ experiments_tag' EnvironmentFile=/etc/experiments_tag # Pull container if necessary ExecStartPre=/usr/bin/docker pull quay.io/pp/experiments:${EXPERIMENTS_TAG} …
44. • WORKS WITH pp chef create_node • WORKS WITH pp <application> deploy • GIVES DEVS POWER • BUYS OPS TIME PAPERLESS DOCKER 1.0
45. docker_application CHEF DEFINITION INSTALLS DOCKER CONFIGURES LOGSPOUT CONFIGURES FILEBEAT MANAGES STARTUP SCRIPTS FOR EACH CONTAINER NGINX: FORWARDING AND HTTPS
46. SENSU CHECKS: 1. Is the application container running? (check_process) 2. Is /health API OK? (check_http) docker_application CHEF DEFINITION
47. docker_application IN A COOKBOOK
48. pp chef create_node heart simple-docker-app --attributes '{"paperless-docker": {"simple_application_name": "rainbozos"}}' docker_application IN A CHEF ROLE
49. IN PRODUCTION! %%%
51. THE SPAMMERS
53. RAINBOZOS PYTHON + DOCKER + CHEF
55. THE HOSTING PROVIDER WOES
59. FULL ORCHESTRATION!
62. PAPERLESS DOCKER 2.0 • docker_app module in Terraform: • EC2 instances; Chef Bootstrap • Internal and external Elastic Load Balancers • DNS
63. AMAZON RDS
64. ~$ terraform destroy
66. OLD WAY ~$ knife data bag create config keepalived_pickles_production_carpathia ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2
67. ~$ bin/tf --application price-gun --environment staging plan ~$ bin/tf --application price-gun --environment staging apply
68. ~$ pp chef create_node earth pickles —cpu 2 — ram 2 ~$ pp chef create_node earth pickles_balancer —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_database —cpu 4 —ram 4 ~$ pp chef create_node earth pickles_webapp — cpu 4 —ram 4 ~$ pp chef create_node earth pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ pp chef create_node production pickles_balancer —cpu 2 —ram 2 ~$ pp chef create_node production pickles_database —cpu 4 —ram 4 ~$ pp chef create_node production pickles_webapp —cpu 4 —ram 4 ~$ pp chef create_node production pickles_backend —cpu 2 —ram 4 ~$ pp chef create_node production pickles — cpu 4 —ram 4 ~$ bin/tf -a pickles -e earth apply ~$ bin/tf -a pickles -e production apply
69. ~$ bin/tf -a chefserver -e testing apply
70. PAPERLESS-TERRAFORM DEPLOY WORKFLOW Deploy to staging Pull Request Code Review! Deploy to Production
74. GROWING PAINS
75. TERRAFORM + DEVTOOLS + JENKINS?
76. THE ALERTS
78. ALERTSGALORE@PAPERLESSPOST.COM** ** not an actual email address
80. Ruby on Rails Javascript Postgres Redis Memcache Haproxy Elasticsearch Postgres Sensu Graphite & Grafana Filebeat/Elasticsearch/Logstash/Kibana Go Python NodeJS Vphere EC2 S3 Amazon RDS ELB Docker Terraform Hopes Dreams 2016
81. 2016 Terraformed full-stack services utilizing AWS, Docker and Chef ( vSphere nodes created with `pp chef create_node`
82. WHERE WE ARE NOT • Mature CHEF cookbook and role testing • Continuous Provisioning / Autoscaling • Continuous Terraform module testing
83. All the Way, Soon
84. ? GantryD
85. (NYC-BASED / REMOTE FRIENDLY!) WE’RE HIRING
86. (NOT THERE YET!)
87. IN SUMMARY
89. INFRASTRUCTURE AS CODE
94. TWITTER: @skymob bethany@paperlesspost.com www.paperlesspost.com/jobs THANKS!