$30 off During Our Annual Pro Sale. View Details »

Practical Service Discovery 
with Consul

Practical Service Discovery 
with Consul

Presented at Gluecon 2016

Chris Stevens

May 26, 2016
Tweet

More Decks by Chris Stevens

Other Decks in Technology

Transcript

  1. Practical Service Discovery
 with Consul Chris Stevens @stevenscg 
 Gluecon

    2016 This work is licensed under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
  2. Chris Stevens @stevenscg CTO @Traxo Developer. Entrepreneur. Pilot

  3. @stevenscg What is Service Discovery?

  4. @stevenscg – Jeff Lindsay @progrium “Service discovery is about knowing

    when any process in the cluster is listening on a TCP or UDP port, and being able to look up and connect to that port by name.” http://progrium.com/blog/2014/07/29/understanding-modern-service-discovery-with-docker/
  5. @stevenscg Location

  6. @stevenscg Health

  7. @stevenscg Speed

  8. @stevenscg Why do we need it?

  9. @stevenscg Massive Networks

  10. @stevenscg Industry-Wide Shift

  11. @stevenscg No Cloud Primitives

  12. @stevenscg Not a Solved Problem

  13. @stevenscg Why now? Cloud. Containerization. [Mono | Mini | Micro]services.

    Best practices (12 Factor apps, etc).
  14. @stevenscg Configuration A Retrospective

  15. @stevenscg Static Configuration // config.php $db = '192.168.1.50:3306'; $api =

    '192.168.1.51:443'; $cache = [ '192.168.1.60:11211', '192.168.1.61:11211', ]; Pets. Not Cattle. ______ < Meow > ------ \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
  16. @stevenscg Configuration Management // ansible-playbook: web-servers.php - hosts: localhost connection:

    local tasks: - name: AWS - Gather RabbitMQ server instances shell: > aws ec2 describe-instances --region {{ region }} --filters "Name=tag:class,Values=rabbitmq" --query "Reservations[].Instances[].PrivateIpAddress" --output json register: rabbitmq_instances_raw changed_when: false Ansible + EC2 Tags => Templating
  17. @stevenscg Service Discovery // config.php (using environment variables) // via

    DNS (load-balanced) $db = dns_get_record(getenv('db.service.consul'), SRV); $api = dns_get_record(getenv('api.service.consul'), SRV); // via API $cacheServers = getServers(getenv('cache.service.consul')); function getServers($name) { $url = 'http://localhost:8500/v1/catalog/service/$name'; $resp = json_decode(file_get_contents($url), true); $servers = array_map(function($item) { // format to host, port, weight, etc. }, $resp); return $servers; } Real-time. Healthy Services. Load Balancing.
  18. @stevenscg A turning point.

  19. @stevenscg A turning point.

  20. @stevenscg Consul https://www.consul.io

  21. @stevenscg Consul Features

  22. @stevenscg Consul Feature PowerIndex Service Registry + Health Checks Distributed

    Key / Value Store First-class DNS Interface Distributed Locks Vault / Nomad
 Backing Store
  23. @stevenscg Implementing Consul

  24. @stevenscg Implementing Consul

  25. @stevenscg Implementing Consul • Planning • Deployment • High Availability

    • Cluster / Instance sizing • Service naming conventions • Configuring DNS • DNS Recursors • Monitoring • Upgrades
  26. @stevenscg Planning Register external services (RDS, etc). Register batch services.

    Feature flags with KV. Locks / Semaphores. App Integration.
  27. @stevenscg Deployment Consul Servers
 Odd number required for quorum. Consul

    Agent
 One per instance or Docker host. Security Groups - TCP/UDP ports. Plan for attended provisioning.
  28. @stevenscg Detect consul servers
 via EC2 tags // ansible playbook:

    consul.yml // consul_servers_via_tag: consul - name: Gather consul server instances run_once: true shell: > aws ec2 describe-instances --region {{ region }} --filters "Name=tag:class,Values={{ consul_servers_via_tag }}" --query "Reservations[].Instances[].PrivateIpAddress" --output json register: consul_instances_raw changed_when: false when: consul_servers_via_tag is defined
  29. @stevenscg Security Groups
 Consul Agents // ansible playbook: security-groups.yml -

    name: ConsulAgentSG description: Consul agent security group rules: # Server RPC - proto: tcp from_port: 8300 to_port: 8300 cidr_ip: 10.0.0.0/8 # Serf gossip LAN - proto: tcp from_port: 8301 to_port: 8301 cidr_ip: 10.0.0.0/8 - proto: udp from_port: 8301 to_port: 8301 cidr_ip: 10.0.0.0/8 # CLI RPC - proto: tcp from_port: 8400 to_port: 8400 cidr_ip: 10.0.0.0/8 # HTTP API - proto: tcp from_port: 8500 to_port: 8500 cidr_ip: 10.0.0.0/8
  30. @stevenscg Security Groups
 Consul Servers // ansible playbook: security-groups.yml -

    name: ConsulServerSG description: Consul server security group rules: # Server RPC - proto: tcp from_port: 8300 to_port: 8300 cidr_ip: 10.0.0.0/8 # Serf gossip LAN - proto: tcp from_port: 8301 to_port: 8301 cidr_ip: 10.0.0.0/8 - proto: udp from_port: 8301 to_port: 8301 cidr_ip: 10.0.0.0/8 # CLI RPC - proto: tcp from_port: 8400 to_port: 8400 cidr_ip: 10.0.0.0/8 # HTTP API - proto: tcp from_port: 8500 to_port: 8500 cidr_ip: 10.0.0.0/8 # DNS Interface - proto: tcp from_port: 8600 to_port: 8600 cidr_ip: 10.0.0.0/8 - proto: udp from_port: 8600 to_port: 8600 cidr_ip: 10.0.0.0/8
  31. @stevenscg Availability Consul is a distributed,
 highly available system. Consul

    nodes within a datacenter
 participate in a gossip protocol. Consul servers within a datacenter
 are part of a single Raft peer set. The Raft Paper [PDF] https://www.consul.io/intro https://www.consul.io/docs/internals/architecture.html
  32. @stevenscg Cluster / Instance Sizing Odd number of servers. Use

    t2.medium or larger. Monitor for leader transitions.
  33. None
  34. None
  35. @stevenscg Service Naming Conventions • DNS-compatible from the start •

    Service name: • api • DNS names: • api.service.consul • api.service.dc1.consul
  36. @stevenscg Configuring DNS “Everything is a freaking DNS problem.” –

    Kris Buytaert
  37. @stevenscg Configuring DNS “Everything is a freaking DNS problem.” –

    Everybody
  38. @stevenscg Configuring DNS for Consul dnsmasq resolv.conf https://www.consul.io/docs/guides/forwarding.html

  39. @stevenscg dnsmasq // ansible playbook: dnsmasq.yml - name: Configure dnsmasq

    to listen only on loopback lineinfile: dest: /etc/dnsmasq.conf regexp: "^#?interface=" line: "interface=lo" state: present - name: Delegate consul DNS requests to the consul DNS port copy: > content='server=/{{ consul_domain }}/ {{ consul_client_address }}#{{ consul_port_dns }}' dest=/etc/dnsmasq.d/10-consul notify: - Restart dnsmasq
  40. @stevenscg resolv.conf // ansible playbook: dnsmasq.yml - name: Add localhost

    nameserver to resolv.conf lineinfile: dest: /etc/resolv.conf line: 'nameserver 127.0.0.1' insertbefore: "^nameserver" state: present - name: Ensure dhclient maintains the localhost nameserver lineinfile: dest: /etc/dhcp/dhclient.conf line: 'prepend domain-name-servers 127.0.0.1;' state: present when: dh.stat.exists
  41. @stevenscg Protip: DNS Recursors Required for AWS managed services (RDS).

    AWS provides DNS for VPCs (i.e. 10.0.0.2) Alert on...
 
 syslog.appName:"consul" "dns: all resolvers failed"
  42. @stevenscg Consul DNS Configuration // ansible group_vars consul_dns_config: allow_stale: true

    max_stale: 2s service_ttl: "*": 0s rds: 5s elasticache: 5s consul_dns_recursors: # - 169.254.169.253 - 10.0.0.2
  43. @stevenscg Monitoring

  44. @stevenscg Monitoring

  45. @stevenscg Monitoring

  46. @stevenscg Monitoring Built-in telemetry support. StatsD or Statsite. Log to

    syslog / Loggly / etc. consul.consul.rpc.request consul.consul.leader.reconcile consul.serf.events.consul_new-leader consul.serf.member.failed
  47. @stevenscg Upgrades

  48. @stevenscg Upgrades

  49. @stevenscg Upgrades Dedicated documentation section. Servers first. Then agents. 4

    upgrades so far
 (0.6.0 - 0.6.4). Zero downtime. https://www.consul.io/docs/upgrading.html
  50. @stevenscg Demos

  51. @stevenscg Demos

  52. @stevenscg Bootstrapping a cluster 1. Launch Consul in a container

    ("consul1"). 2. Join 2 additional server containers.
 "consul2" and "consul3". 3. Kill off "consul2". 4. Check cluster health. Just Released! Official docker image https://hub.docker.com/_/consul/
  53. None
  54. @stevenscg Companion Project • Vagrant VM • Single-node Consul Cluster

    • Interactive demo app + services • Consul Agent, UI, and API • Ansible playbooks • Curl examples https://github.com/stevenscg/service-discovery-with-consul
  55. @stevenscg Companion Project

  56. @stevenscg Companion Project

  57. @stevenscg Register a service Consul is already running in our

    VM. Run a playbook to register services:
 statsd. demo-app (web). memcached. Lookup services via DNS and API.
  58. @stevenscg Register a service $ ansible-playbook 01-register-service.yml -i .vagrant/provisioners/ansible/inventory PLAY

    *************************************************************************** TASK [setup] ******************************************************************* ok: [default] TASK [Fetch existing services from consul API] ********************************* ok: [default] TASK [Existing services] ******************************************************* ok: [default] => { "msg": { "consul": [] } } TASK [Register statsd with consul] ********************************************* changed: [default] TASK [Register demo-app with consul] ******************************************* changed: [default] TASK [Register memcached with consul] ****************************************** changed: [default] TASK [Wait for the checks to be evaluated (3 seconds)] ************************* Pausing for 3 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [default] TASK [Fetch updated services from consul API] ********************************** ok: [default] TASK [Updated services] ******************************************************** ok: [default] => { "msg": { "cache": [], "consul": [], "statsd": [], "web": [] } } PLAY RECAP ********************************************************************* default : ok=9 changed=3 unreachable=0 failed=0
  59. @stevenscg Register a service Lookup "statsd" service via DNS

  60. @stevenscg Register a service See all services via API

  61. @stevenscg Health checks Consul is already running in our VM.

    Run a playbook to register new checks for existing services. Evaluate the checks via the Consul API.
  62. @stevenscg Health checks $ ansible-playbook 02-register-health-checks.yml -i .vagrant/provisioners/ansible/inventory PLAY ***************************************************************************

    TASK [setup] ******************************************************************* ok: [default] TASK [Fetch existing health checks from consul API] **************************** ok: [default] TASK [Existing health checks] ************************************************** ok: [default] => { "msg": { "service:web": { "CheckID": "service:web", "CreateIndex": 0, "ModifyIndex": 0, "Name": "Service 'web' check", "Node": "demo", "Notes": "", "Output": "HTTP GET http://localhost:8001/health_check: 200 OK Output: ", "ServiceID": "web", "ServiceName": "web", "Status": "passing" } } }
  63. @stevenscg Health checks TASK [Register a status code check for

    the demo-app] *************************** changed: [default] TASK [Wait for the checks to be evaluated (3 seconds)] ************************* Pausing for 3 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [default] TASK [Fetch updated health checks from consul API] ***************************** ok: [default] TASK [Updated health checks] *************************************************** ok: [default] => { "msg": { "service:web": { "CheckID": "service:web", "CreateIndex": 0, "ModifyIndex": 0, "Name": "Service 'web' check", "Node": "demo", "Notes": "", "Output": "HTTP GET http://localhost:8001/health_check: 200 OK Output: ", "ServiceID": "web", "ServiceName": "web", "Status": "passing" }, "service:web:status_code": { "CheckID": "service:web:status_code", "CreateIndex": 0, "ModifyIndex": 0, "Name": "service:web:status_code", "Node": "demo", "Notes": "", "Output": "HTTP GET http://localhost:8001: 200 OK Output:", "ServiceID": "", "ServiceName": "", "Status": "passing" } } } PLAY RECAP ********************************************************************* default : ok=7 changed=1 unreachable=0 failed=0
  64. @stevenscg Health checks Lookup "web" service checks via API

  65. @stevenscg DNS Interface • Standard lookup
 [tag.]<service>.service[.datacenter].<domain>
 dig @127.0.0.1 -p

    8600 consul.service.consul SRV
 dig consul.service.consul SRV
 dig consul.service.dc1.consul SRV • Prepared query lookup
 <query or name>.query[.datacenter].<domain>
 dig redis-primary.service.consul SRV • Randomized (healthy) results for load balancing https://www.consul.io/docs/agent/dns.html
  66. @stevenscg Key / Value Store Consul is already running in

    our VM. Consul-PHP-SDK. Curl. Any HTTP client.
  67. @stevenscg Key / Value Store Set key "test/foo/bar" to a

    string
  68. @stevenscg Key / Value Store Set key "test/foo/bazz" to a

    JSON object
  69. @stevenscg Key / Value Store • Manage KV items with

    a
 dedicated git repository. • git2consul detects
 changes and updates
 KV paths. • Full change history. • Familiar workflows. Recommendations
  70. @stevenscg Key / Value Store • App config managed
 by

    consul-template. • KV changes detected
 by consul-template. • App config updated
 and restarted. • Works with Ansible,
 Chef, Puppet, etc. Recommendations
  71. @stevenscg Distributed Locks Uses Consul leader election client-side. Ensures only

    a single client process can execute. Scheduled Jobs. One-off tasks. https://www.consul.io/docs/guides/leader-election.html
  72. @stevenscg Distributed Locks Consul is already running in our VM.

    LockHandler from Consul-PHP-SDK. Consul KV + Session. https://github.com/stevenscg/consul-php-sdk
  73. @stevenscg Distributed Locks

  74. @stevenscg Feature Flags Consul KV with a simple JSON structure.

    Feature Helper from Consul-PHP-SDK. Top level "feature" key path.
  75. @stevenscg Feature Flags

  76. @stevenscg Feature Flags

  77. @stevenscg Feature Flags

  78. @stevenscg Feature Flags

  79. @stevenscg Advanced Topics • ACLs
 https://www.consul.io/docs/internals/acl.html • Multi-Datacenter
 https://www.consul.io/docs/guides/datacenters.html •

    Prepared Queries
 https://www.consul.io/docs/agent/http/query.html • Network Tomography
 https://www.consul.io/docs/internals/coordinates.html
  80. @stevenscg Works with Consul A tool for managing secrets. VAULT

    https://www.vaultproject.io
  81. @stevenscg Works with Consul NOMAD A Distributed, Highly Available, Datacenter-Aware

    Scheduler. https://www.nomadproject.io
  82. Practical Service Discovery
 with Consul This work is licensed under

    a
 Creative Commons Attribution-ShareAlike 3.0 United States License Chris Stevens @stevenscg 
 Gluecon 2016
  83. @stevenscg Image Credits 1 Flickr / Shane Gorski https://flic.kr/p/4TsYxV 2

    Traxo 5 Flickr / Xaf https://flic.kr/p/8bWWrR 6 Flickr / Tony Hisgett https://flic.kr/p/5aGNRj 7 Flickr / Tony Hisgett https://flic.kr/p/ovbczG 9 Github / @adrianco https://github.com/adrianco/spigo 19 Flickr / Xaf https://flic.kr/p/74gWVS 20-21 HashiCorp 24 Flickr / Xaf https://flic.kr/p/74gWVS 37 Flickr / Sasha G https://flic.kr/p/purZ4 44-45 Dean Mouhtaropoulos 48 Flickr / Paul Williams https://flic.kr/p/5spYAJ 51 Flickr / Timelapsed https://flic.kr/p/oVyUnH 52 Docker 80-81 HashiCorp 82 Flickr / Shane Gorski https://flic.kr/p/4TsYxV