Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load Balancing a Dynamic Infrastructure with NGINX, Chef, and Consul

Kevin Reedy
October 22, 2014

Load Balancing a Dynamic Infrastructure with NGINX, Chef, and Consul

Load balancing and routing traffic to a single application is easy, but sending traffic to a always-changing number of applications is quite a challenge. In the last year, Belly has migrated from a monolithic Rails app to a service-oriented architecture with over fifty applications.

In this session, we’ll talk about how to use Chef and Consul to dynamically configure NGINX to route and load balance traffic across applications. These applications can be deployed in a variety of ways, including Chef, Docker, and S3. Additionally, we’ll discuss the importance of service discovery and monitoring inter-service traffic in a service-oriented architecture.

Kevin Reedy

October 22, 2014
Tweet

More Decks by Kevin Reedy

Other Decks in Technology

Transcript

  1. Kevin Reedy
    @kevinreedy
    https://tech.bellycard.com
    Load Balancing a
    Dynamic
    Infrastructure
    with Nginx, Chef, and Consul

    View Slide

  2. Load Balancing a Dynamic Infrastructure @kevinreedy
    Overview
    •Belly’s infrastructure stack over time
    •Using Chef to configure Nginx
    •Using data from Consul’s service
    discovery to configure Nginx
    •Pro Tips (time permitting)

    View Slide

  3. Load Balancing a Dynamic Infrastructure @kevinreedy
    Disclaimer
    •Nothing in this talk is specific to a
    single tool
    •Chef, Puppet, Ansible, Salt, CFEngine,
    etc. are all great!
    •Consul, Etcd, Zookeeper, etc. are all
    great!
    •Use what works for you
    !

    View Slide

  4. Load Balancing a Dynamic Infrastructure @kevinreedy
    What is Belly?

    View Slide

  5. Event Logging at Belly @kevinreedy

    View Slide

  6. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack:
    The Evolution

    View Slide

  7. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v0 the dark age
    •Monolithic Ruby on Rails Application
    •MySQL, MongoDB, Memcache
    •Deployed to Heroku

    View Slide

  8. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v1 a new hope
    •Monolithic Ruby on Rails Application
    •MySQL, MongoDB, Memcache
    •Deployed using Capistrano
    •~6 Servers on AWS EC2
    •AWS Elastic Load Balancer

    View Slide

  9. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 buzzword ahead
    •Separate the Homepage from the API

    View Slide

  10. Load Balancing a Dynamic Infrastructure @kevinreedy
    Service Oriented
    Architecture!
    !
    (SOA)

    View Slide

  11. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •Separate the Homepage from the API
    •Simply separate www.bellycard.com
    from api.bellycard.com

    View Slide

  12. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •Some routes on www.bellycard.com
    still need to route to our API servers
    • www.bellycard.com/api
    • www.bellycard.com/api-assets
    • www.bellycard.com/eo
    • www.bellycard.com/ec
    • www.bellycard.com/facebook/callback

    View Slide

  13. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •AWS Elastic Load Balancer no longer
    gave us enough flexibility
    •Nginx to the rescue!
    •Multiple Upstreams
    •Rewrite Rules
    !
    !

    View Slide

  14. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    upstream api {
    least_conn;
    server 10.0.0.101:8080;
    server 10.0.0.102:8080;
    server 10.0.0.103:8080;
    server 10.0.0.104:8080;
    server 10.0.0.105:8080;
    server 10.0.0.106:8080;
    }
    !
    upstream homepage {
    least_conn;
    server 10.0.0.201:8080;
    server 10.0.0.202:8080;
    }

    View Slide

  15. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    server {
    listen 80;
    server_name api.bellycard.com;
    !
    location / {
    proxy_pass http://api;
    }
    }
    !
    server {
    listen 80;
    server_name www.bellycard.com;
    !
    location / {
    proxy_pass http://homepage;
    }
    !
    location /api {
    proxy_pass http://api;
    }
    }

    View Slide

  16. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •This worked great!
    •So we deployed a third service
    •…and a fourth
    •…and a fifth… that was Node.js
    !
    !

    View Slide

  17. Load Balancing a Dynamic Infrastructure @kevinreedy
    Configuration
    with Chef

    View Slide

  18. Load Balancing a Dynamic Infrastructure @kevinreedy
    Intro to Chef
    •Configuration Management with Ruby
    •http://learnchef.com
    !

    View Slide

  19. Load Balancing a Dynamic Infrastructure @kevinreedy
    Recipes
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end

    View Slide

  20. Load Balancing a Dynamic Infrastructure @kevinreedy
    Files
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end
    !
    file '/usr/share/nginx/html/index.html' do
    content '

    Hello World!

    '
    end
    !

    View Slide

  21. Load Balancing a Dynamic Infrastructure @kevinreedy
    Templates
    package 'nginx'
    !
    service 'nginx' do
    action [:start, :enable]
    end
    !
    g = 'Hello'
    w = 'Kevin'
    !
    template '/usr/share/nginx/html/index.html' do
    source 'index.html.erb'
    variables (
    'greeting' => g,
    'who' => w
    )
    end

    View Slide

  22. Load Balancing a Dynamic Infrastructure @kevinreedy
    Templates


    !


    View Slide

  23. Load Balancing a Dynamic Infrastructure @kevinreedy
    Cookbooks
    •Collection of Recipes, Files, Templates
    •And Resources, Providers, and Libraries
    •Tons of cookbooks at
    https://supermarket.getchef.com/
    • nginx
    • mysql
    • java
    • apt/yum
    • ssh_known_hosts
    !

    View Slide

  24. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    •Install Nginx
    •Configure Nginx using Chef
    •Use Chef to discover where our
    application is running
    •Configure Nginx to route traffic to our
    application

    View Slide

  25. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    # Tune Nginx
    node.set['nginx']['worker_connections'] = 8192
    node.set['nginx']['worker_rlimit_nofile'] = 32768
    node.set['nginx']['event'] = 'epoll'
    node.set['nginx']['client_max_body_size'] = '4m'
    !
    # Install Nginx
    include_recipe 'nginx::repo'
    include_recipe 'nginx'
    !
    # Find application nodes
    nodes = search(:node, 'recipe:belly-api')
    !
    # Generate site config
    template "#{node['nginx']['dir']}/sites-enabled/api" do
    source 'api.erb'
    variables(
    'servers' => nodes
    )
    notifies :reload, 'service[nginx]'
    end

    View Slide

  26. Load Balancing a Dynamic Infrastructure @kevinreedy
    Nginx Example
    upstream api {
    least_conn;

    server :8080; #

    }
    !
    server {
    listen 80;
    server_name api.bellycard.com;
    !
    location / {
    proxy_pass http://api;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    }
    }

    View Slide

  27. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    # Set staging and production specific attributes
    if node.chef_environment == 'production'
    node.set['belly']['lb']['domain'] = 'bellycard.com'
    node.set['belly']['lb']['cert_name'] = 'wildcard_bellycard_com'
    node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-assets/sitemap.xml.gz'
    elsif node.chef_environment == 'staging'
    node.set['belly']['lb']['domain'] = 'bellystaging.com'
    node.set['belly']['lb']['cert_name'] = 'wildcard_bellystaging_com'
    node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-staging-assets/sitemap.xml.gz'
    end
    !
    # Install Nginx
    include_recipe 'nginx::repo'
    include_recipe 'nginx'
    !
    # Configure Nginx
    include_recipe 'belly-lb::_tuning'
    include_recipe 'belly-lb::_logging'
    include_recipe 'belly-lb::_security'
    include_recipe 'belly-lb::_ssl-certs'
    !
    # Add Sites
    include_recipe 'belly-lb::_default-site'
    include_recipe 'belly-lb::_api'
    include_recipe 'belly-lb::_platform'
    include_recipe 'belly-lb::_ssl-termination'
    include_recipe 'belly-lb::_other-sites'
    !
    # Post-install Configuration
    include_recipe 'belly-lb::_route53'
    include_recipe 'belly-lb::_monitoring'
    include_recipe 'belly-lb::_ngxtop'
    !
    # Misc
    # Set chef interval to 5 minutes (or keep at lessor value)
    if node['chef_client'] && node['chef_client']['interval']
    node.set['chef_client']['interval'] = [Integer(node['chef_client']['interval']), 300].min
    node.set['chef_client']['splay'] = [Integer(node['chef_client']['splay']), 30].min
    end

    View Slide

  28. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    if Chef::Config[:solo]
    Chef::Log.warn('This recipe uses search. Chef Solo does not support search.')
    else
    services = Chef::DataBagItem.load('belly-platform', 'portmap')
    services['services'].each do |service_name, service_attr|
    servers = Array.new
    !
    search_keys = {
    'name' => ['name'],
    'ipaddress' => ['ipaddress']
    }
    !
    partial_search(:node, "recipe:belly-service\\:\\:#{ service_name } AND chef_environment:#{ node.chef_environment }", keys: search_keys).each
    do |result|
    servers.push(
    'host' => result['ipaddress'],
    'port' => service_attr['port'],
    'comment' => result['name']
    )
    end
    !
    unless servers.empty?
    # Generate site config
    template "#{ node['nginx']['dir'] }/sites-available/#{ service_name }" do
    source "simple-site.erb"
    mode 0444
    owner "root"
    group "root"
    variables(
    'service_name' => service_name,
    'service_comment' => service_attr['comment'],
    'domain_name' => "#{ service_name }.#{ node['belly']['lb']['domain'] }"
    'servers' => servers,
    'ssl_cert' => "#{ node['belly']['lb']['cert_name'] }.crt",
    'ssl_key' => "#{ node['belly']['lb']['cert_name'] }.key"
    )
    notifies :reload, 'service[nginx]'
    end
    !
    # Enable site
    nginx_site service_name
    end
    end
    end

    View Slide

  29. Load Balancing a Dynamic Infrastructure @kevinreedy
    Real World Example
    # nginx config
    # Generated by Chef for
    # Changes will be overwritten
    !
    upstream {
    least_conn;

    server : max_fails=3 fail_timeout=60 weight=1; #

    }
    !
    server {
    listen 443 ssl;
    server_name ;
    ssl on;
    ssl_certificate /etc/nginx/ssl/;
    ssl_certificate_key /etc/nginx/ssl/;
    !
    # enable session resumption to improve https performance
    # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    !
    # enables server-side protection from BEAST attacks
    # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html
    ssl_prefer_server_ciphers on;
    !
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    !
    # ciphers chosen for forward secrecy and compatibility
    # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
    ssl_ciphers 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4
    EECDH EDH+aRSA RC4 AES128-SHA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS';
    !
    location / {
    proxy_pass http://;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    }
    }

    View Slide

  30. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v2 SOA
    •We deployed around 40 ruby apps to
    our servers using Chef
    •All traffic to all apps was routed by
    Nginx, dynamically configured with
    Chef
    !
    !

    View Slide

  31. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SOA All The Things
    •We also had around 10 Frontend
    Javascript applications deployed onto
    S3 / Cloudfront
    •And another 5 on Heroku
    !
    !

    View Slide

  32. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SOA All The Things
    •We also had around 10 Frontend
    Javascript applications deployed onto
    S3 / Cloudfront
    •And another 4 on Heroku
    !
    !

    View Slide

  33. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v3 SSL All The Things
    •All traffic to bellycard.com and its
    subdomains now passes through our
    load balancers
    •Assets are served on a cloudfront
    domain for performance
    !

    View Slide

  34. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4

    View Slide

  35. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4 Docker
    •Greatly simplifies deploying
    applications
    •Separation of responsibility from
    development and operations
    •Allows developers to easily spin up
    dependent services on their own
    laptop
    •Speeds up deployment
    !

    View Slide

  36. Load Balancing a Dynamic Infrastructure @kevinreedy
    Belly’s Stack v4 Docker
    •Deploying Containers
    •CoreOS / Fleet
    •Deis
    •Mesos
    •Kubernetes
    •Flynn
    !
    !

    View Slide

  37. Load Balancing a Dynamic Infrastructure @kevinreedy
    Deployment Strategies
    •Zero downtime deploys
    •Deploy in place
    •Blue / Green
    •Canary
    !
    !
    !
    !

    View Slide

  38. Load Balancing a Dynamic Infrastructure @kevinreedy
    Service Discovery
    •Zookeeper
    •Etcd
    •Consul
    !
    !
    !

    View Slide

  39. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul

    View Slide

  40. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul
    •Distributed key value store
    •Service discovery as a first class citizen
    •Distributed health checks
    •Multiple Datacenters
    •Long polling for instant notification of
    changes
    •Scale out agents
    !

    View Slide

  41. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Released yesterday
    •Subscribes to changes in:
    •services
    •health check
    •key/value store
    !

    View Slide

  42. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Released yesterday
    •Subscribes to changes
    •Generates files from templates
    •Optionally calls a command on change
    !

    View Slide

  43. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    $ consul-template -consul consul-prod.example.com:8500 \
    -template "/etc/consul-templates/api:/etc/nginx/sites-
    enabled/api:service nginx reload"

    View Slide

  44. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    upstream api {
    least_conn;
    {{range service "api"}}server {{.Address}}:{{.Port}};
    {{end}}
    }
    !
    upstream homepage {
    least_conn;
    {{range service "homepage"}}server {{.Address}}:{{.Port}};
    {{end}}
    }
    !
    server {
    listen 80;
    server_name www.bellycard.com;
    !
    location / {
    proxy_pass http://homepage;
    }
    !
    location /api {
    proxy_pass http://api;
    }
    }

    View Slide

  45. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    •Convergence on service changes is fast
    •Configuration Management is still
    needed to generate templates
    !

    View Slide

  46. Load Balancing a Dynamic Infrastructure @kevinreedy
    Consul Template
    services = (
    JSON.parse(
    open("http://#{consul_host}:8500/v1/catalog/services")
    .read
    ) || {}
    ).keys
    !
    # Generate nginx template for consul template
    template '/etc/consul-templates/services' do
    source 'consul-upstream.erb'
    variables(
    services: services,
    ssl_cert: "#{ node['belly']['lb']['cert_name'] }.crt",
    ssl_key: "#{ node['belly']['lb']['cert_name'] }.key"
    )
    notifies :reload, 'service[consul-haproxy]', :delayed
    end
    !
    # Configure Consul Template
    template '/etc/consul-template.conf' do
    source 'consul-conf.erb'
    variables(
    host: "#{ consul_host }:8500"
    source: '/etc/consul-templates/services'
    destination: '/etc/nginx/sites-enabled/services'
    reload_command: 'service nginx reload'
    )
    notifies :reload, 'service[consul-template]', :delayed
    end

    View Slide

  47. Load Balancing a Dynamic Infrastructure @kevinreedy
    Pro Tips!

    View Slide

  48. Load Balancing a Dynamic Infrastructure @kevinreedy
    Chef Partial Search
    •Chef search makes a copy of the entire
    node, including all attributes
    •Partial search allows you to return only
    the keys you care about

    View Slide

  49. Load Balancing a Dynamic Infrastructure @kevinreedy
    Chef Partial Search
    # search
    nodes = search(:node, 'recipe:belly-api')
    !
    # partial search
    search_keys = {
    'name' => ['name'],
    'ipaddress' => ['ipaddress']
    }
    !
    nodes = partial_search(:node, 'recipe:belly-api', :keys => search_keys)

    View Slide

  50. Load Balancing a Dynamic Infrastructure @kevinreedy
    DNS
    •Nginx resolves DNS in upstream blocks
    at config reload
    •Don’t do this if DNS resolution changes
    regularly
    • s3.amazonaws.com
    • *.herokuapp.com
    •Resolve in Chef and trigger Nginx
    reload on changes

    View Slide

  51. Load Balancing a Dynamic Infrastructure @kevinreedy
    Avoid Stale Config
    •We need a way to clean up stale sites
    •add :delete functions to your Chef
    cookbooks
    •render all site configs to a single file

    View Slide

  52. Load Balancing a Dynamic Infrastructure @kevinreedy
    Questions?

    View Slide

  53. Load Balancing a Dynamic Infrastructure @kevinreedy
    Thanks!
    •Belly is hiring!
    •Slides will be posted
    •Keep in touch!
    •Twitter @kevinreedy
    [email protected]
    •https://tech.bellycard.com
    !

    View Slide