Cooking Up Drama

Presenters:
- Peter Shannon
- Bridget Kromhout

“What do you mean, we don’t have configuration management?” Joining DramaFever’s ops team last year, we both came from orgs that used Chef, but initially we disagreed about which flavor of CM to use here. We cooperated to build awesome instead of vying for the upper hand. (Go go gadget devops.) We’ll discuss the initial concerns around going with Chef and how we addressed them.

DramaFever uses Docker extensively, but we found Dockerfiles aren’t enough to keep All The Things consistent. We need to predictably create and maintain the host instances and then launch containers in the right environment for our main Django app or any of our golang microservices. Using Chef along with other tools such as Packer, we’re beginning to bring order to dev, qa, monitoring, CI, and our AMI creation process as well as our entire application infrastructure.

We’ll detail how we’re reducing deployment overhead and increasing maintainability for our AWS-based video platform, which streams international content, documentaries, and horror to a rapidly growing worldwide audience. We will also share some of the pitfalls and corner cases we are working through in order to create our desired infrastructure.

Transcript

1. Cooking Up Drama Bridget Kromhout & Peter Shannon
2. @bridgetkromhout @pietroshannon Bridget Kromhout Peter Shannon @devopsdaysMSP @arresteddevops devopsdays.org Go Data Ops & Zip Ties
3. @pietroshannon @bridgetkromhout DramaFever: streaming international content
4. peak load: tens of thousands of requests per second traffic variance: swings 10-20x throughout the week @bridgetkromhout @pietroshannon
5. @bridgetkromhout @pietroshannon
6. @bridgetkromhout @pietroshannon
7. Architecture • All services in AWS • Python (Django) main website • Go microservices (video analytics ingest, on- the-fly image processing, bookmarking…) • Upstreams routed via nginx • Celery + SQS for async tasks • Streaming delivery through Akamai @bridgetkromhout @pietroshannon
8. What do you mean, NO CM!? Docker Jenkins Graphite & ELK AWS Config Management @bridgetkromhout @pietroshannon
9. CONFIG MANAGEMENT EVERYWHERE @bridgetkromhout @pietroshannon
10. Why did we need CM? • Single source of truth to build AMIs and provision AWS instances. • Consistent configuration across ephemeral instances. • Hand-crafted, longer-lived instances are hard to reproduce. @bridgetkromhout @pietroshannon
11. @bridgetkromhout @pietroshannon CM selection showdown: fight!
12. @bridgetkromhout @pietroshannon cooperation, not competition
13. @bridgetkromhout @pietroshannon decisions: hosted chef
14. @bridgetkromhout @pietroshannon decisions: hosted chef (but not “Starter Kit”)
15. @bridgetkromhout @pietroshannon decisions: environments
16. @bridgetkromhout @pietroshannon decisions: environments roles
17. @bridgetkromhout @pietroshannon decisions: environments roles minimal notifies
18. @bridgetkromhout @pietroshannon workflow
19. @bridgetkromhout @pietroshannon workflow: chef-dk
20. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen
21. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins
22. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins via github
23. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins via github rubocop foodcritic
24. @bridgetkromhout @pietroshannon Docker images built from Dockerfiles in Jenkins jobs deployment via fabric & upstart
25. pain points? @bridgetkromhout @pietroshannon where to start...
26. pain points? visible wins? @bridgetkromhout @pietroshannon where to start...
27. @bridgetkromhout @pietroshannon where to start... production!
28. @bridgetkromhout @pietroshannon autoscaling: scorched earth
29. #!/bin/bash cat <<EOF > /etc/init/django.conf description "Run Django containers for www" start on started docker-reg stop on runlevel [!2345] or stopped docker respawn limit 5 30 [...] replacing 100s of lines of userdata... @bridgetkromhout @pietroshannon
30. #!/bin/bash # upstart configs are now created by chef rm /etc/chef/client.pem mkdir -p /var/log/chef chef-client -r 'role[rolename]' -E 'environment' -L /var/log/chef/chef-client.log ...with a chef-client run. @bridgetkromhout @pietroshannon
31. deregistering nodes (knife) /usr/bin/knife node delete -y -c /etc/chef/knife.rb <%= node['base']['chefname'] %> /usr/bin/knife client delete -y -c /etc/chef/knife.rb <%= node['base']['chefname'] %> @bridgetkromhout @pietroshannon
32. deregistering nodes (rc script) template '/etc/init.d/unregister_chef_instance' do source 'default/unregister_chef_instance.erb' end link '/etc/rc0.d/K99unregister_chef_instance' do to '/etc/init.d/unregister_chef_instance' end @bridgetkromhout @pietroshannon
33. ami factory @bridgetkromhout @pietroshannon Autoscaling Packer AMI EC2 Instances Jenkins GitHub Chef
34. @bridgetkromhout @pietroshannon apt-get install: only for AMI...
35. @bridgetkromhout @pietroshannon apt-get install: only for AMI… ...and test kitchen
36. Docker: what to chef? not Docker images... not application settings... LWRP: upstarts admin wrappers @bridgetkromhout @pietroshannon
37. Docker & LWRPs @bridgetkromhout @pietroshannon docker run \ -e DJANGO_ENVIRON=PROD \ -e HAPROXY=df/haproxy-prod.cfg \ -p 8000:8000 \ -v /var/log/containers:/var/log \ --name django \ localhost-alias.com:5000/www:prod \ /var/www/bin/start-django
38. docker run \ <% if @docker_rm == true -%> --rm \ <% end %> <% @docker_env.each do |k, v| -%> -e <%= k %>=<%= v %> \ <% end %> <% @docker_port.each do |p| -%> -p <%= p %> \ <% end %> @bridgetkromhout @pietroshannon upstart template
39. <% @docker_volume.each do |v| -%> -v <%= v %> \ <% end %> --name <%= @application_name %> \ localhost-alias.com:<%= @registry_port %>/<%= @docker_image %>:<%= @docker_tag %> \ <%= @docker_command %> @bridgetkromhout @pietroshannon upstart template (cont)
40. @bridgetkromhout @pietroshannon attribute :docker_command, :kind_of => String, :required => true attribute :docker_env, :kind_of => Hash, :default => {} attribute :docker_port, :kind_of => Array, :default => [] attribute :docker_volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :docker_rm, :kind_of => [TrueClass, FalseClass], : default => false attribute :docker_name_override, :kind_of => String, :default => nil attribute :docker_image_override, :kind_of => String, :default => nil attribute :docker_tag_override, :kind_of => String, :default => nil so many overrides
41. @bridgetkromhout @pietroshannon if new_resource.docker_name_override.nil? name = new_resource.name upstart = name else name = new_resource.docker_name_override upstart = "#{new_resource.name}-#{name}" end if this, then just don’t
42. @bridgetkromhout @pietroshannon attribute :command, :kind_of => String, :required => true attribute :env, :kind_of => Hash, :default => {} attribute :port, :kind_of => Array, :default => [] attribute :volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :rm, :kind_of => [TrueClass, FalseClass], :default => false attribute :image, :kind_of => String, :required => true attribute :tag, :kind_of => String, :required => true attribute :type, :kind_of => String, :required => true attribute :cron, :kind_of => [TrueClass, FalseClass], :default => false using attributes
43. recipe using LWRP base_docker node['www']['django']['name'] do command node['www']['django']['command'] env node['www'][service]['django'][env]['env'] image node['www']['django']['image'] port node['www'][service]['django'][env]['port'] tag node['www'][service]['django'][env]['tag'] type node['www']['django']['type'] end @bridgetkromhout @pietroshannon
44. Wrapping Community Cookbooks @bridgetkromhout @pietroshannon cookbook_path ["#{ENV['DFHOME']} /chef/cookbooks", "#{ENV['DFHOME']} /chef/community_cookbooks"] clear delineation no forking of community cookbooks
45. wrapper cookbook - leroy • depends on community cookbooks (jenkins, etc) • recipes include: builds, packer, plugins • fun with FC001 • variations on our base (fstab, docker registry) @bridgetkromhout @pietroshannon
46. private docker registry # this goes in /etc/default/docker to control docker's upstart config DOCKER_OPTS="--graph=/mnt/docker --insecure- registry=localhost-alias.com:5000" • localhost-alias.com in DNS with A record to 127.0.0.1 • OS X /etc/hosts: use the boot2docker host-only network IP @bridgetkromhout @pietroshannon
47. registry upstart docker pull public_registry_image docker run -p 5000:5000 --name registry \ -v /etc/docker-reg:/registry-conf \ -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml \ public_registry_image @bridgetkromhout @pietroshannon
48. config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true storage: s3 storage_path: /registry @bridgetkromhout @pietroshannon
49. packer $HOME/packer/packer build \ -var "account_id=$AWS_ACCOUNT_ID" \ -var "aws_access_key_id=$AWS_ACCESS_KEY_ID" \ -var "aws_secret_key=$AWS_SECRET_ACCESS_KEY" \ -var "x509_cert_path=$AWS_X509_CERT_PATH" \ -var "x509_key_path=$AWS_X509_KEY_PATH" \ -var "s3_bucket=bucketname" \ -var "ami_name=$AMI_NAME" \ -var "source_ami=$SOURCE_AMI" \ -var "chef_validation=$CHEF_VAL" \ -var "chef_client=$HOME/packer/client.rb" \ -only=amazon-instance \ $HOME/packer/prod.json @bridgetkromhout @pietroshannon
50. packer for ami building { "type": "chef-client", "server_url": "https://api.opscode. com/organizations/dramafever", "run_list": [ "base::ami" ], "validation_key_path": "{{user `chef_validation`}}", "validation_client_name": "dramafever-validator", "node_name": "packer-ami" } @bridgetkromhout @pietroshannon
51. limiting packer IAM permissions "Action":[ "ec2:TerminateInstances", "ec2:StopInstances", "ec2:DeleteSnapshot", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:ModifyImageAttribute" ], "Effect":"Allow", "Resource":"*", "Condition":{ "StringEquals":{ "ec2:ResourceTag/name":"Packer Builder" } } @bridgetkromhout @pietroshannon
52. so, about that Jenkins server... @bridgetkromhout @pietroshannon
53. @bridgetkromhout @pietroshannon (un-)containerize some of the things
54. sentry base_docker_upstart 'sentry-udp' do docker_env( 'DJANGO_ENVIRON' => 'DRAMAFEVER' ) docker_image_override 'sentry' docker_tag_override 'dev' docker_port ['9001:9001/udp'] docker_command 'sentry --config=/config/sentry.conf.py start udp' end @bridgetkromhout @pietroshannon
55. chef for devs @bridgetkromhout @pietroshannon
56. self-service visibility @bridgetkromhout @pietroshannon
57. (re)cycle of life @bridgetkromhout @pietroshannon
58. @bridgetkromhout @pietroshannon
59. @bridgetkromhout @pietroshannon thanks!