Load Balancing a Dynamic Infrastructure with NGINX, Chef, and Consul

Load balancing and routing traffic to a single application is easy, but sending traffic to a always-changing number of applications is quite a challenge. In the last year, Belly has migrated from a monolithic Rails app to a service-oriented architecture with over fifty applications.

In this session, we’ll talk about how to use Chef and Consul to dynamically configure NGINX to route and load balance traffic across applications. These applications can be deployed in a variety of ways, including Chef, Docker, and S3. Additionally, we’ll discuss the importance of service discovery and monitoring inter-service traffic in a service-oriented architecture.

Transcript

1. Kevin Reedy
   @kevinreedy
   https://tech.bellycard.com
   Load Balancing a
   Dynamic
   Infrastructure
   with Nginx, Chef, and Consul
   

   View Slide

2. Load Balancing a Dynamic Infrastructure @kevinreedy
   Overview
   •Belly’s infrastructure stack over time
   •Using Chef to configure Nginx
   •Using data from Consul’s service
   discovery to configure Nginx
   •Pro Tips (time permitting)
   

   View Slide

3. Load Balancing a Dynamic Infrastructure @kevinreedy
   Disclaimer
   •Nothing in this talk is specific to a
   single tool
   •Chef, Puppet, Ansible, Salt, CFEngine,
   etc. are all great!
   •Consul, Etcd, Zookeeper, etc. are all
   great!
   •Use what works for you
   !
   

   View Slide

4. Load Balancing a Dynamic Infrastructure @kevinreedy
   What is Belly?
   

   View Slide

5. Event Logging at Belly @kevinreedy
   

   View Slide

6. Load Balancing a Dynamic Infrastructure @kevinreedy
   Belly’s Stack:
   The Evolution
   

   View Slide

7. Load Balancing a Dynamic Infrastructure @kevinreedy
   Belly’s Stack v0 the dark age
   •Monolithic Ruby on Rails Application
   •MySQL, MongoDB, Memcache
   •Deployed to Heroku
   

   View Slide

8. Load Balancing a Dynamic Infrastructure @kevinreedy
   Belly’s Stack v1 a new hope
   •Monolithic Ruby on Rails Application
   •MySQL, MongoDB, Memcache
   •Deployed using Capistrano
   •~6 Servers on AWS EC2
   •AWS Elastic Load Balancer
   

   View Slide

9. Load Balancing a Dynamic Infrastructure @kevinreedy
   Belly’s Stack v2 buzzword ahead
   •Separate the Homepage from the API
   

   View Slide

10. Load Balancing a Dynamic Infrastructure @kevinreedy
    Service Oriented
    Architecture!
    !
    (SOA)
    

    View Slide

11. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •Separate the Homepage from the API
    •Simply separate www.bellycard.com
    from api.bellycard.com
    

    View Slide

12. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •Some routes on www.bellycard.com
    still need to route to our API servers
    • www.bellycard.com/api
    • www.bellycard.com/api-assets
    • www.bellycard.com/eo
    • www.bellycard.com/ec
    • www.bellycard.com/facebook/callback
    

    View Slide

13. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •AWS Elastic Load Balancer no longer
    gave us enough flexibility
    •Nginx to the rescue!
    •Multiple Upstreams
    •Rewrite Rules
    !
    !
    

    View Slide

14. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    upstream api {
    least_conn;
    server 10.0.0.101:8080;
    server 10.0.0.102:8080;
    server 10.0.0.103:8080;
    server 10.0.0.104:8080;
    server 10.0.0.105:8080;
    server 10.0.0.106:8080;
    }
    !
    upstream homepage {
    least_conn;
    server 10.0.0.201:8080;
    server 10.0.0.202:8080;
    }
    

    View Slide

15. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    server {
    listen 80;
    server_name api.bellycard.com;
    !
    location / {
    proxy_pass http://api;
    }
    }
    !
    server {
    listen 80;
    server_name www.bellycard.com;
    !
    location / {
    proxy_pass http://homepage;
    }
    !
    location /api {
    proxy_pass http://api;
    }
    }
    

    View Slide

16. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •This worked great!
    •So we deployed a third service
    •…and a fourth
    •…and a fifth… that was Node.js
    !
    !
    

    View Slide

17. Load Balancing a Dynamic Infrastructure @kevinreedy
    Configuration
    with Chef
    

    View Slide

18. Load Balancing a Dynamic Infrastructure @kevinreedy
    Intro to Chef
    •Configuration Management with Ruby
    •http://learnchef.com
    !
    

    View Slide

19. Load Balancing a Dynamic Infrastructure @kevinreedy
    Recipes
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end
    

    View Slide

20. Load Balancing a Dynamic Infrastructure @kevinreedy
    Files
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end
    !
    file '/usr/share/nginx/html/index.html' do
    content '
    
    Hello World!
    
    '
    end
    !
    

    View Slide

21. Load Balancing a Dynamic Infrastructure @kevinreedy
    Templates
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end
    !
    g = 'Hello'
    w = 'Kevin'
    !
    template '/usr/share/nginx/html/index.html' do
    source 'index.html.erb'
    variables (
    'greeting' => g,
    'who' => w
    )
    end
    

    View Slide

22. Load Balancing a Dynamic Infrastructure @kevinreedy
    Templates
    
    
    !
    
    
    

    View Slide

23. Load Balancing a Dynamic Infrastructure @kevinreedy
    Cookbooks
    •Collection of Recipes, Files, Templates
    •And Resources, Providers, and Libraries
    •Tons of cookbooks at
    https://supermarket.getchef.com/
    • nginx
    • mysql
    • java
    • apt/yum
    • ssh_known_hosts
    !
    

    View Slide

24. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    •Install Nginx
    •Configure Nginx using Chef
    •Use Chef to discover where our
    application is running
    •Configure Nginx to route traffic to our
    application
    

    View Slide

25. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    # Tune Nginx
    node.set['nginx']['worker_connections'] = 8192
    node.set['nginx']['worker_rlimit_nofile'] = 32768
    node.set['nginx']['event'] = 'epoll'
    node.set['nginx']['client_max_body_size'] = '4m'
    !
    # Install Nginx
    include_recipe 'nginx::repo'
    include_recipe 'nginx'
    !
    # Find application nodes
    nodes = search(:node, 'recipe:belly-api')
    !
    # Generate site config
    template "#{node['nginx']['dir']}/sites-enabled/api" do
    source 'api.erb'
    variables(
    'servers' => nodes
    )
    notifies :reload, 'service[nginx]'
    end
    

    View Slide

26. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    upstream api {
    least_conn;
    
    server :8080; #
    
    }
    !
    server {
    listen 80;
    server_name api.bellycard.com;
    !
    location / {
    proxy_pass http://api;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    }
    }
    

    View Slide

27. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    # Set staging and production specific attributes
    if node.chef_environment == 'production'
    node.set['belly']['lb']['domain'] = 'bellycard.com'
    node.set['belly']['lb']['cert_name'] = 'wildcard_bellycard_com'
    node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-assets/sitemap.xml.gz'
    elsif node.chef_environment == 'staging'
    node.set['belly']['lb']['domain'] = 'bellystaging.com'
    node.set['belly']['lb']['cert_name'] = 'wildcard_bellystaging_com'
    node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-staging-assets/sitemap.xml.gz'
    end
    !
    # Install Nginx
    include_recipe 'nginx::repo'
    include_recipe 'nginx'
    !
    # Configure Nginx
    include_recipe 'belly-lb::_tuning'
    include_recipe 'belly-lb::_logging'
    include_recipe 'belly-lb::_security'
    include_recipe 'belly-lb::_ssl-certs'
    !
    # Add Sites
    include_recipe 'belly-lb::_default-site'
    include_recipe 'belly-lb::_api'
    include_recipe 'belly-lb::_platform'
    include_recipe 'belly-lb::_ssl-termination'
    include_recipe 'belly-lb::_other-sites'
    !
    # Post-install Configuration
    include_recipe 'belly-lb::_route53'
    include_recipe 'belly-lb::_monitoring'
    include_recipe 'belly-lb::_ngxtop'
    !
    # Misc
    # Set chef interval to 5 minutes (or keep at lessor value)
    if node['chef_client'] && node['chef_client']['interval']
    node.set['chef_client']['interval'] = [Integer(node['chef_client']['interval']), 300].min
    node.set['chef_client']['splay'] = [Integer(node['chef_client']['splay']), 30].min
    end
    

    View Slide

28. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    if Chef::Config[:solo]
    Chef::Log.warn('This recipe uses search. Chef Solo does not support search.')
    else
    services = Chef::DataBagItem.load('belly-platform', 'portmap')
    services['services'].each do |service_name, service_attr|
    servers = Array.new
    !
    search_keys = {
    'name' => ['name'],
    'ipaddress' => ['ipaddress']
    }
    !
    partial_search(:node, "recipe:belly-service\\:\\:#{ service_name } AND chef_environment:#{ node.chef_environment }", keys: search_keys).each
    do |result|
    servers.push(
    'host' => result['ipaddress'],
    'port' => service_attr['port'],
    'comment' => result['name']
    )
    end
    !
    unless servers.empty?
    # Generate site config
    template "#{ node['nginx']['dir'] }/sites-available/#{ service_name }" do
    source "simple-site.erb"
    mode 0444
    owner "root"
    group "root"
    variables(
    'service_name' => service_name,
    'service_comment' => service_attr['comment'],
    'domain_name' => "#{ service_name }.#{ node['belly']['lb']['domain'] }"
    'servers' => servers,
    'ssl_cert' => "#{ node['belly']['lb']['cert_name'] }.crt",
    'ssl_key' => "#{ node['belly']['lb']['cert_name'] }.key"
    )
    notifies :reload, 'service[nginx]'
    end
    !
    # Enable site
    nginx_site service_name
    end
    end
    end
    

    View Slide

29. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    # nginx config
    # Generated by Chef for
    # Changes will be overwritten
    !
    upstream {
    least_conn;
    
    server : max_fails=3 fail_timeout=60 weight=1; #
    
    }
    !
    server {
    listen 443 ssl;
    server_name ;
    ssl on;
    ssl_certificate /etc/nginx/ssl/;
    ssl_certificate_key /etc/nginx/ssl/;
    !
    # enable session resumption to improve https performance
    # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    !
    # enables server-side protection from BEAST attacks
    # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html
    ssl_prefer_server_ciphers on;
    !
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    !
    # ciphers chosen for forward secrecy and compatibility
    # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
    ssl_ciphers 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4
    EECDH EDH+aRSA RC4 AES128-SHA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS';
    !
    location / {
    proxy_pass http://;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    }
    }
    

    View Slide

30. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •We deployed around 40 ruby apps to
    our servers using Chef
    •All traffic to all apps was routed by
    Nginx, dynamically configured with
    Chef
    !
    !
    

    View Slide

31. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SOA All The Things
    •We also had around 10 Frontend
    Javascript applications deployed onto
    S3 / Cloudfront
    •And another 5 on Heroku
    !
    !
    

    View Slide

32. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SOA All The Things
    •We also had around 10 Frontend
    Javascript applications deployed onto
    S3 / Cloudfront
    •And another 4 on Heroku
    !
    !
    

    View Slide

33. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SSL All The Things
    •All traffic to bellycard.com and its
    subdomains now passes through our
    load balancers
    •Assets are served on a cloudfront
    domain for performance
    !
    

    View Slide

34. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4
    

    View Slide

35. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4 Docker
    •Greatly simplifies deploying
    applications
    •Separation of responsibility from
    development and operations
    •Allows developers to easily spin up
    dependent services on their own
    laptop
    •Speeds up deployment
    !
    

    View Slide

36. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4 Docker
    •Deploying Containers
    •CoreOS / Fleet
    •Deis
    •Mesos
    •Kubernetes
    •Flynn
    !
    !
    

    View Slide

37. Load Balancing a Dynamic Infrastructure @kevinreedy
    Deployment Strategies
    •Zero downtime deploys
    •Deploy in place
    •Blue / Green
    •Canary
    !
    !
    !
    !
    

    View Slide

38. Load Balancing a Dynamic Infrastructure @kevinreedy
    Service Discovery
    •Zookeeper
    •Etcd
    •Consul
    !
    !
    !
    

    View Slide

39. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul
    

    View Slide

40. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul
    •Distributed key value store
    •Service discovery as a first class citizen
    •Distributed health checks
    •Multiple Datacenters
    •Long polling for instant notification of
    changes
    •Scale out agents
    !
    

    View Slide

41. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Released yesterday
    •Subscribes to changes in:
    •services
    •health check
    •key/value store
    !
    

    View Slide

42. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Released yesterday
    •Subscribes to changes
    •Generates files from templates
    •Optionally calls a command on change
    !
    

    View Slide

43. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    $ consul-template -consul consul-prod.example.com:8500 \
    -template "/etc/consul-templates/api:/etc/nginx/sites-
    enabled/api:service nginx reload"
    

    View Slide

44. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    upstream api {
    least_conn;
    {{range service "api"}}server {{.Address}}:{{.Port}};
    {{end}}
    }
    !
    upstream homepage {
    least_conn;
    {{range service "homepage"}}server {{.Address}}:{{.Port}};
    {{end}}
    }
    !
    server {
    listen 80;
    server_name www.bellycard.com;
    !
    location / {
    proxy_pass http://homepage;
    }
    !
    location /api {
    proxy_pass http://api;
    }
    }
    

    View Slide

45. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Convergence on service changes is fast
    •Configuration Management is still
    needed to generate templates
    !
    

    View Slide

46. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    services = (
    JSON.parse(
    open("http://#{consul_host}:8500/v1/catalog/services")
    .read
    ) || {}
    ).keys
    !
    # Generate nginx template for consul template
    template '/etc/consul-templates/services' do
    source 'consul-upstream.erb'
    variables(
    services: services,
    ssl_cert: "#{ node['belly']['lb']['cert_name'] }.crt",
    ssl_key: "#{ node['belly']['lb']['cert_name'] }.key"
    )
    notifies :reload, 'service[consul-haproxy]', :delayed
    end
    !
    # Configure Consul Template
    template '/etc/consul-template.conf' do
    source 'consul-conf.erb'
    variables(
    host: "#{ consul_host }:8500"
    source: '/etc/consul-templates/services'
    destination: '/etc/nginx/sites-enabled/services'
    reload_command: 'service nginx reload'
    )
    notifies :reload, 'service[consul-template]', :delayed
    end
    

    View Slide

47. Load Balancing a Dynamic Infrastructure @kevinreedy
    Pro Tips!
    

    View Slide

48. Load Balancing a Dynamic Infrastructure @kevinreedy
    Chef Partial Search
    •Chef search makes a copy of the entire
    node, including all attributes
    •Partial search allows you to return only
    the keys you care about
    

    View Slide

49. Load Balancing a Dynamic Infrastructure @kevinreedy
    Chef Partial Search
    # search
    nodes = search(:node, 'recipe:belly-api')
    !
    # partial search
    search_keys = {
    'name' => ['name'],
    'ipaddress' => ['ipaddress']
    }
    !
    nodes = partial_search(:node, 'recipe:belly-api', :keys => search_keys)
    

    View Slide

50. Load Balancing a Dynamic Infrastructure @kevinreedy
    DNS
    •Nginx resolves DNS in upstream blocks
    at config reload
    •Don’t do this if DNS resolution changes
    regularly
    • s3.amazonaws.com
    • *.herokuapp.com
    •Resolve in Chef and trigger Nginx
    reload on changes
    

    View Slide

51. Load Balancing a Dynamic Infrastructure @kevinreedy
    Avoid Stale Config
    •We need a way to clean up stale sites
    •add :delete functions to your Chef
    cookbooks
    •render all site configs to a single file
    

    View Slide

52. Load Balancing a Dynamic Infrastructure @kevinreedy
    Questions?
    

    View Slide

53. Load Balancing a Dynamic Infrastructure @kevinreedy
    Thanks!
    •Belly is hiring!
    •Slides will be posted
    •Keep in touch!
    •Twitter @kevinreedy
    •kevin@bellycard.com
    •https://tech.bellycard.com
    !
    

    View Slide