Load Balancing a Dynamic Infrastructure with NGINX, Chef, and Consul

Load balancing and routing traffic to a single application is easy, but sending traffic to a always-changing number of applications is quite a challenge. In the last year, Belly has migrated from a monolithic Rails app to a service-oriented architecture with over fifty applications.

In this session, we’ll talk about how to use Chef and Consul to dynamically configure NGINX to route and load balance traffic across applications. These applications can be deployed in a variety of ways, including Chef, Docker, and S3. Additionally, we’ll discuss the importance of service discovery and monitoring inter-service traffic in a service-oriented architecture.

Transcript

1. Kevin Reedy @kevinreedy https://tech.bellycard.com Load Balancing a Dynamic Infrastructure with Nginx, Chef, and Consul
2. Load Balancing a Dynamic Infrastructure @kevinreedy Overview •Belly’s infrastructure stack over time •Using Chef to configure Nginx •Using data from Consul’s service discovery to configure Nginx •Pro Tips (time permitting)
3. Load Balancing a Dynamic Infrastructure @kevinreedy Disclaimer •Nothing in this talk is specific to a single tool •Chef, Puppet, Ansible, Salt, CFEngine, etc. are all great! •Consul, Etcd, Zookeeper, etc. are all great! •Use what works for you !
4. Load Balancing a Dynamic Infrastructure @kevinreedy What is Belly?
5. Event Logging at Belly @kevinreedy
6. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack: The Evolution
7. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v0 the dark age •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed to Heroku
8. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v1 a new hope •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed using Capistrano •~6 Servers on AWS EC2 •AWS Elastic Load Balancer
9. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 buzzword ahead •Separate the Homepage from the API
10. Load Balancing a Dynamic Infrastructure @kevinreedy Service Oriented Architecture! ! (SOA)
11. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •Separate the Homepage from the API •Simply separate www.bellycard.com from api.bellycard.com
12. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •Some routes on www.bellycard.com still need to route to our API servers • www.bellycard.com/api • www.bellycard.com/api-assets • www.bellycard.com/eo • www.bellycard.com/ec • www.bellycard.com/facebook/callback
13. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •AWS Elastic Load Balancer no longer gave us enough flexibility •Nginx to the rescue! •Multiple Upstreams •Rewrite Rules ! !
14. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA upstream api { least_conn; server 10.0.0.101:8080; server 10.0.0.102:8080; server 10.0.0.103:8080; server 10.0.0.104:8080; server 10.0.0.105:8080; server 10.0.0.106:8080; } ! upstream homepage { least_conn; server 10.0.0.201:8080; server 10.0.0.202:8080; }
15. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; } } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }
16. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •This worked great! •So we deployed a third service •…and a fourth •…and a fifth… that was Node.js ! !
17. Load Balancing a Dynamic Infrastructure @kevinreedy Configuration with Chef
18. Load Balancing a Dynamic Infrastructure @kevinreedy Intro to Chef •Configuration Management with Ruby •http://learnchef.com !
19. Load Balancing a Dynamic Infrastructure @kevinreedy Recipes package 'nginx' ! service 'nginx' do action [:start, :enable] end
20. Load Balancing a Dynamic Infrastructure @kevinreedy Files package 'nginx' ! service 'nginx' do action [:start, :enable] end ! file '/usr/share/nginx/html/index.html' do content '<html> <body> <h1>Hello World!</h1> </body> </html>' end !
21. Load Balancing a Dynamic Infrastructure @kevinreedy Templates package 'nginx' ! service 'nginx' do action [:start, :enable] end ! g = 'Hello' w = 'Kevin' ! template '/usr/share/nginx/html/index.html' do source 'index.html.erb' variables ( 'greeting' => g, 'who' => w ) end
22. Load Balancing a Dynamic Infrastructure @kevinreedy Templates <html> <body> <h1><%= @greeting %> <%= @who %>!</h1> </body> </html>
23. Load Balancing a Dynamic Infrastructure @kevinreedy Cookbooks •Collection of Recipes, Files, Templates •And Resources, Providers, and Libraries •Tons of cookbooks at https://supermarket.getchef.com/ • nginx • mysql • java • apt/yum • ssh_known_hosts !
24. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example •Install Nginx •Configure Nginx using Chef •Use Chef to discover where our application is running •Configure Nginx to route traffic to our application
25. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example # Tune Nginx node.set['nginx']['worker_connections'] = 8192 node.set['nginx']['worker_rlimit_nofile'] = 32768 node.set['nginx']['event'] = 'epoll' node.set['nginx']['client_max_body_size'] = '4m' ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Find application nodes nodes = search(:node, 'recipe:belly-api') ! # Generate site config template "#{node['nginx']['dir']}/sites-enabled/api" do source 'api.erb' variables( 'servers' => nodes ) notifies :reload, 'service[nginx]' end
26. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example upstream api { least_conn; <% @servers.each do |server| %> server <%= server['ipaddress'] %>:8080; # <%= server['name'] %> <% end %> } ! server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }
27. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example # Set staging and production specific attributes if node.chef_environment == 'production' node.set['belly']['lb']['domain'] = 'bellycard.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellycard_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-assets/sitemap.xml.gz' elsif node.chef_environment == 'staging' node.set['belly']['lb']['domain'] = 'bellystaging.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellystaging_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-staging-assets/sitemap.xml.gz' end ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Configure Nginx include_recipe 'belly-lb::_tuning' include_recipe 'belly-lb::_logging' include_recipe 'belly-lb::_security' include_recipe 'belly-lb::_ssl-certs' ! # Add Sites include_recipe 'belly-lb::_default-site' include_recipe 'belly-lb::_api' include_recipe 'belly-lb::_platform' include_recipe 'belly-lb::_ssl-termination' include_recipe 'belly-lb::_other-sites' ! # Post-install Configuration include_recipe 'belly-lb::_route53' include_recipe 'belly-lb::_monitoring' include_recipe 'belly-lb::_ngxtop' ! # Misc # Set chef interval to 5 minutes (or keep at lessor value) if node['chef_client'] && node['chef_client']['interval'] node.set['chef_client']['interval'] = [Integer(node['chef_client']['interval']), 300].min node.set['chef_client']['splay'] = [Integer(node['chef_client']['splay']), 30].min end
28. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example if Chef::Config[:solo] Chef::Log.warn('This recipe uses search. Chef Solo does not support search.') else services = Chef::DataBagItem.load('belly-platform', 'portmap') services['services'].each do |service_name, service_attr| servers = Array.new ! search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! partial_search(:node, "recipe:belly-service\\:\\:#{ service_name } AND chef_environment:#{ node.chef_environment }", keys: search_keys).each do |result| servers.push( 'host' => result['ipaddress'], 'port' => service_attr['port'], 'comment' => result['name'] ) end ! unless servers.empty? # Generate site config template "#{ node['nginx']['dir'] }/sites-available/#{ service_name }" do source "simple-site.erb" mode 0444 owner "root" group "root" variables( 'service_name' => service_name, 'service_comment' => service_attr['comment'], 'domain_name' => "#{ service_name }.#{ node['belly']['lb']['domain'] }" 'servers' => servers, 'ssl_cert' => "#{ node['belly']['lb']['cert_name'] }.crt", 'ssl_key' => "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[nginx]' end ! # Enable site nginx_site service_name end end end
29. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example # <%= @service_comment %> nginx config # Generated by Chef for <%= node.name %> # Changes will be overwritten ! upstream <%= @service_name %> { least_conn; <% @servers.each do |server| %> server <%= server['host'] %>:<%= server['port'] %> max_fails=3 fail_timeout=60 weight=1; # <%= server['comment'] %> <% end %> } ! server { listen 443 ssl; server_name <%= @domain_name %>; ssl on; ssl_certificate /etc/nginx/ssl/<%= @ssl_cert %>; ssl_certificate_key /etc/nginx/ssl/<%= @ssl_key %>; ! # enable session resumption to improve https performance # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; ! # enables server-side protection from BEAST attacks # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html ssl_prefer_server_ciphers on; ! ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ! # ciphers chosen for forward secrecy and compatibility # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html ssl_ciphers 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 AES128-SHA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS'; ! location / { proxy_pass http://<%= @service_name %>; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }
30. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA •We deployed around 40 ruby apps to our servers using Chef •All traffic to all apps was routed by Nginx, dynamically configured with Chef ! !
31. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 5 on Heroku ! !
32. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 4 on Heroku ! !
33. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SSL All The Things •All traffic to bellycard.com and its subdomains now passes through our load balancers •Assets are served on a cloudfront domain for performance !
34. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4
35. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker •Greatly simplifies deploying applications •Separation of responsibility from development and operations •Allows developers to easily spin up dependent services on their own laptop •Speeds up deployment !
36. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker •Deploying Containers •CoreOS / Fleet •Deis •Mesos •Kubernetes •Flynn ! !
37. Load Balancing a Dynamic Infrastructure @kevinreedy Deployment Strategies •Zero downtime deploys •Deploy in place •Blue / Green •Canary ! ! ! !
38. Load Balancing a Dynamic Infrastructure @kevinreedy Service Discovery •Zookeeper •Etcd •Consul ! ! !
39. Load Balancing a Dynamic Infrastructure @kevinreedy Consul
40. Load Balancing a Dynamic Infrastructure @kevinreedy Consul •Distributed key value store •Service discovery as a first class citizen •Distributed health checks •Multiple Datacenters •Long polling for instant notification of changes •Scale out agents !
41. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday •Subscribes to changes in: •services •health check •key/value store !
42. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday •Subscribes to changes •Generates files from templates •Optionally calls a command on change !
43. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template $ consul-template -consul consul-prod.example.com:8500 \ -template "/etc/consul-templates/api:/etc/nginx/sites- enabled/api:service nginx reload"
44. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template upstream api { least_conn; {{range service "api"}}server {{.Address}}:{{.Port}}; {{end}} } ! upstream homepage { least_conn; {{range service "homepage"}}server {{.Address}}:{{.Port}}; {{end}} } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }
45. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Convergence on service changes is fast •Configuration Management is still needed to generate templates !
46. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template services = ( JSON.parse( open("http://#{consul_host}:8500/v1/catalog/services") .read ) || {} ).keys ! # Generate nginx template for consul template template '/etc/consul-templates/services' do source 'consul-upstream.erb' variables( services: services, ssl_cert: "#{ node['belly']['lb']['cert_name'] }.crt", ssl_key: "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[consul-haproxy]', :delayed end ! # Configure Consul Template template '/etc/consul-template.conf' do source 'consul-conf.erb' variables( host: "#{ consul_host }:8500" source: '/etc/consul-templates/services' destination: '/etc/nginx/sites-enabled/services' reload_command: 'service nginx reload' ) notifies :reload, 'service[consul-template]', :delayed end
47. Load Balancing a Dynamic Infrastructure @kevinreedy Pro Tips!
48. Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search •Chef search makes a copy of the entire node, including all attributes •Partial search allows you to return only the keys you care about
49. Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search # search nodes = search(:node, 'recipe:belly-api') ! # partial search search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! nodes = partial_search(:node, 'recipe:belly-api', :keys => search_keys)
50. Load Balancing a Dynamic Infrastructure @kevinreedy DNS •Nginx resolves DNS in upstream blocks at config reload •Don’t do this if DNS resolution changes regularly • s3.amazonaws.com • *.herokuapp.com •Resolve in Chef and trigger Nginx reload on changes
51. Load Balancing a Dynamic Infrastructure @kevinreedy Avoid Stale Config •We need a way to clean up stale sites •add :delete functions to your Chef cookbooks •render all site configs to a single file
52. Load Balancing a Dynamic Infrastructure @kevinreedy Questions?
53. Load Balancing a Dynamic Infrastructure @kevinreedy Thanks! •Belly is hiring! •Slides will be posted •Keep in touch! •Twitter @kevinreedy •kevin@bellycard.com •https://tech.bellycard.com !