Slide 1

Slide 1 text

Kevin Reedy @kevinreedy https://tech.bellycard.com Load Balancing a Dynamic Infrastructure with Nginx, Chef, and Consul

Slide 2

Slide 2 text

Load Balancing a Dynamic Infrastructure @kevinreedy Overview •Belly’s infrastructure stack over time •Using Chef to configure Nginx •Using data from Consul’s service discovery to configure Nginx •Pro Tips (time permitting)

Slide 3

Slide 3 text

Load Balancing a Dynamic Infrastructure @kevinreedy Disclaimer •Nothing in this talk is specific to a single tool •Chef, Puppet, Ansible, Salt, CFEngine, etc. are all great! •Consul, Etcd, Zookeeper, etc. are all great! •Use what works for you !

Slide 4

Slide 4 text

Load Balancing a Dynamic Infrastructure @kevinreedy What is Belly?

Slide 5

Slide 5 text

Event Logging at Belly @kevinreedy

Slide 6

Slide 6 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack: The Evolution

Slide 7

Slide 7 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v0 the dark age •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed to Heroku

Slide 8

Slide 8 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v1 a new hope •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed using Capistrano •~6 Servers on AWS EC2 •AWS Elastic Load Balancer

Slide 9

Slide 9 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 buzzword ahead •Separate the Homepage from the API

Slide 10

Slide 10 text

Load Balancing a Dynamic Infrastructure @kevinreedy Service Oriented Architecture! ! (SOA)

Slide 11

Slide 11 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •Separate the Homepage from the API •Simply separate www.bellycard.com from api.bellycard.com

Slide 12

Slide 12 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •Some routes on www.bellycard.com still need to route to our API servers • www.bellycard.com/api • www.bellycard.com/api-assets • www.bellycard.com/eo • www.bellycard.com/ec • www.bellycard.com/facebook/callback

Slide 13

Slide 13 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •AWS Elastic Load Balancer no longer gave us enough flexibility •Nginx to the rescue! •Multiple Upstreams •Rewrite Rules ! !

Slide 14

Slide 14 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA upstream api { least_conn; server 10.0.0.101:8080; server 10.0.0.102:8080; server 10.0.0.103:8080; server 10.0.0.104:8080; server 10.0.0.105:8080; server 10.0.0.106:8080; } ! upstream homepage { least_conn; server 10.0.0.201:8080; server 10.0.0.202:8080; }

Slide 15

Slide 15 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; } } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }

Slide 16

Slide 16 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •This worked great! •So we deployed a third service •…and a fourth •…and a fifth… that was Node.js ! !

Slide 17

Slide 17 text

Load Balancing a Dynamic Infrastructure @kevinreedy Configuration with Chef

Slide 18

Slide 18 text

Load Balancing a Dynamic Infrastructure @kevinreedy Intro to Chef •Configuration Management with Ruby •http://learnchef.com !

Slide 19

Slide 19 text

Load Balancing a Dynamic Infrastructure @kevinreedy Recipes package 'nginx' ! service 'nginx' do action [:start, :enable] end

Slide 20

Slide 20 text

Load Balancing a Dynamic Infrastructure @kevinreedy Files package 'nginx' ! service 'nginx' do action [:start, :enable] end ! file '/usr/share/nginx/html/index.html' do content '

Hello World!

' end !

Slide 21

Slide 21 text

Load Balancing a Dynamic Infrastructure @kevinreedy Templates package 'nginx' ! service 'nginx' do action [:start, :enable] end ! g = 'Hello' w = 'Kevin' ! template '/usr/share/nginx/html/index.html' do source 'index.html.erb' variables ( 'greeting' => g, 'who' => w ) end

Slide 22

Slide 22 text

Load Balancing a Dynamic Infrastructure @kevinreedy Templates

<%= @greeting %> <%= @who %>!

Slide 23

Slide 23 text

Load Balancing a Dynamic Infrastructure @kevinreedy Cookbooks •Collection of Recipes, Files, Templates •And Resources, Providers, and Libraries •Tons of cookbooks at https://supermarket.getchef.com/ • nginx • mysql • java • apt/yum • ssh_known_hosts !

Slide 24

Slide 24 text

Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example •Install Nginx •Configure Nginx using Chef •Use Chef to discover where our application is running •Configure Nginx to route traffic to our application

Slide 25

Slide 25 text

Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example # Tune Nginx node.set['nginx']['worker_connections'] = 8192 node.set['nginx']['worker_rlimit_nofile'] = 32768 node.set['nginx']['event'] = 'epoll' node.set['nginx']['client_max_body_size'] = '4m' ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Find application nodes nodes = search(:node, 'recipe:belly-api') ! # Generate site config template "#{node['nginx']['dir']}/sites-enabled/api" do source 'api.erb' variables( 'servers' => nodes ) notifies :reload, 'service[nginx]' end

Slide 26

Slide 26 text

Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example upstream api { least_conn; <% @servers.each do |server| %> server <%= server['ipaddress'] %>:8080; # <%= server['name'] %> <% end %> } ! server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }

Slide 27

Slide 27 text

Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example # Set staging and production specific attributes if node.chef_environment == 'production' node.set['belly']['lb']['domain'] = 'bellycard.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellycard_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-assets/sitemap.xml.gz' elsif node.chef_environment == 'staging' node.set['belly']['lb']['domain'] = 'bellystaging.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellystaging_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-staging-assets/sitemap.xml.gz' end ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Configure Nginx include_recipe 'belly-lb::_tuning' include_recipe 'belly-lb::_logging' include_recipe 'belly-lb::_security' include_recipe 'belly-lb::_ssl-certs' ! # Add Sites include_recipe 'belly-lb::_default-site' include_recipe 'belly-lb::_api' include_recipe 'belly-lb::_platform' include_recipe 'belly-lb::_ssl-termination' include_recipe 'belly-lb::_other-sites' ! # Post-install Configuration include_recipe 'belly-lb::_route53' include_recipe 'belly-lb::_monitoring' include_recipe 'belly-lb::_ngxtop' ! # Misc # Set chef interval to 5 minutes (or keep at lessor value) if node['chef_client'] && node['chef_client']['interval'] node.set['chef_client']['interval'] = [Integer(node['chef_client']['interval']), 300].min node.set['chef_client']['splay'] = [Integer(node['chef_client']['splay']), 30].min end

Slide 28

Slide 28 text

Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example if Chef::Config[:solo] Chef::Log.warn('This recipe uses search. Chef Solo does not support search.') else services = Chef::DataBagItem.load('belly-platform', 'portmap') services['services'].each do |service_name, service_attr| servers = Array.new ! search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! partial_search(:node, "recipe:belly-service\\:\\:#{ service_name } AND chef_environment:#{ node.chef_environment }", keys: search_keys).each do |result| servers.push( 'host' => result['ipaddress'], 'port' => service_attr['port'], 'comment' => result['name'] ) end ! unless servers.empty? # Generate site config template "#{ node['nginx']['dir'] }/sites-available/#{ service_name }" do source "simple-site.erb" mode 0444 owner "root" group "root" variables( 'service_name' => service_name, 'service_comment' => service_attr['comment'], 'domain_name' => "#{ service_name }.#{ node['belly']['lb']['domain'] }" 'servers' => servers, 'ssl_cert' => "#{ node['belly']['lb']['cert_name'] }.crt", 'ssl_key' => "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[nginx]' end ! # Enable site nginx_site service_name end end end

Slide 29

Slide 29 text

Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example # <%= @service_comment %> nginx config # Generated by Chef for <%= node.name %> # Changes will be overwritten ! upstream <%= @service_name %> { least_conn; <% @servers.each do |server| %> server <%= server['host'] %>:<%= server['port'] %> max_fails=3 fail_timeout=60 weight=1; # <%= server['comment'] %> <% end %> } ! server { listen 443 ssl; server_name <%= @domain_name %>; ssl on; ssl_certificate /etc/nginx/ssl/<%= @ssl_cert %>; ssl_certificate_key /etc/nginx/ssl/<%= @ssl_key %>; ! # enable session resumption to improve https performance # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; ! # enables server-side protection from BEAST attacks # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html ssl_prefer_server_ciphers on; ! ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ! # ciphers chosen for forward secrecy and compatibility # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html ssl_ciphers 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 AES128-SHA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS'; ! location / { proxy_pass http://<%= @service_name %>; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }

Slide 30

Slide 30 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •We deployed around 40 ruby apps to our servers using Chef •All traffic to all apps was routed by Nginx, dynamically configured with Chef ! !

Slide 31

Slide 31 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 5 on Heroku ! !

Slide 32

Slide 32 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 4 on Heroku ! !

Slide 33

Slide 33 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SSL All The Things •All traffic to bellycard.com and its subdomains now passes through our load balancers •Assets are served on a cloudfront domain for performance !

Slide 34

Slide 34 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4

Slide 35

Slide 35 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker •Greatly simplifies deploying applications •Separation of responsibility from development and operations •Allows developers to easily spin up dependent services on their own laptop •Speeds up deployment !

Slide 36

Slide 36 text

Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker •Deploying Containers •CoreOS / Fleet •Deis •Mesos •Kubernetes •Flynn ! !

Slide 37

Slide 37 text

Load Balancing a Dynamic Infrastructure @kevinreedy Deployment Strategies •Zero downtime deploys •Deploy in place •Blue / Green •Canary ! ! ! !

Slide 38

Slide 38 text

Load Balancing a Dynamic Infrastructure @kevinreedy Service Discovery •Zookeeper •Etcd •Consul ! ! !

Slide 39

Slide 39 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul

Slide 40

Slide 40 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul •Distributed key value store •Service discovery as a first class citizen •Distributed health checks •Multiple Datacenters •Long polling for instant notification of changes •Scale out agents !

Slide 41

Slide 41 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday •Subscribes to changes in: •services •health check •key/value store !

Slide 42

Slide 42 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday •Subscribes to changes •Generates files from templates •Optionally calls a command on change !

Slide 43

Slide 43 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template $ consul-template -consul consul-prod.example.com:8500 \ -template "/etc/consul-templates/api:/etc/nginx/sites- enabled/api:service nginx reload"

Slide 44

Slide 44 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template upstream api { least_conn; {{range service "api"}}server {{.Address}}:{{.Port}}; {{end}} } ! upstream homepage { least_conn; {{range service "homepage"}}server {{.Address}}:{{.Port}}; {{end}} } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }

Slide 45

Slide 45 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Convergence on service changes is fast •Configuration Management is still needed to generate templates !

Slide 46

Slide 46 text

Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template services = ( JSON.parse( open("http://#{consul_host}:8500/v1/catalog/services") .read ) || {} ).keys ! # Generate nginx template for consul template template '/etc/consul-templates/services' do source 'consul-upstream.erb' variables( services: services, ssl_cert: "#{ node['belly']['lb']['cert_name'] }.crt", ssl_key: "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[consul-haproxy]', :delayed end ! # Configure Consul Template template '/etc/consul-template.conf' do source 'consul-conf.erb' variables( host: "#{ consul_host }:8500" source: '/etc/consul-templates/services' destination: '/etc/nginx/sites-enabled/services' reload_command: 'service nginx reload' ) notifies :reload, 'service[consul-template]', :delayed end

Slide 47

Slide 47 text

Load Balancing a Dynamic Infrastructure @kevinreedy Pro Tips!

Slide 48

Slide 48 text

Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search •Chef search makes a copy of the entire node, including all attributes •Partial search allows you to return only the keys you care about

Slide 49

Slide 49 text

Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search # search nodes = search(:node, 'recipe:belly-api') ! # partial search search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! nodes = partial_search(:node, 'recipe:belly-api', :keys => search_keys)

Slide 50

Slide 50 text

Load Balancing a Dynamic Infrastructure @kevinreedy DNS •Nginx resolves DNS in upstream blocks at config reload •Don’t do this if DNS resolution changes regularly • s3.amazonaws.com • *.herokuapp.com •Resolve in Chef and trigger Nginx reload on changes

Slide 51

Slide 51 text

Load Balancing a Dynamic Infrastructure @kevinreedy Avoid Stale Config •We need a way to clean up stale sites •add :delete functions to your Chef cookbooks •render all site configs to a single file

Slide 52

Slide 52 text

Load Balancing a Dynamic Infrastructure @kevinreedy Questions?

Slide 53

Slide 53 text

Load Balancing a Dynamic Infrastructure @kevinreedy Thanks! •Belly is hiring! •Slides will be posted •Keep in touch! •Twitter @kevinreedy •[email protected] •https://tech.bellycard.com !