Load balancer management with Consul

1. Tatsuhiko Kubo@cubicdaiya Consul Casual Talks#1 2016/08/01 Load balancer management with Cunsul
2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.
4. Load balancers in • nginx • L7 (HTTP or HTTPS) • L4 (TCP) • OpenResty • Dynamic behavior with Lua
5. nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 SPDY/HTTP2 SPDY/HTTP2 EC2 EC2 Load balancer for App
6. Load balancer for internal API requests global network private network Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
7. Load balancer for Search app app app OpenResty Latest Indices All Indices Contents cache & Dynamic Balancing HTTP HTTP
8. Load balancer for Push Gaurun Gaurun Gaurun
9. in • Service discovery • Load balancer, SMTP, etc… • Configuration deployment • TLS Session Tickets, IP black list • Distributed lock • Ensure running only 1 process always in clusters
10. Service discovery
11. Service discovery with • Use case in • Internal DNS as • Endpoint of internal API, DNS-RR • Via HTTP APIs • Listing nodes in service
12. Service discovery with 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “service”: { “name”: “api-internal”, “tags”: [“production”], … } } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
13. Listing all nodes in api-internal service $ curl -s \ consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
14. DNS-RR in api-internal service $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.6 production.api-internal.service.consul. 0 IN A 10.0.1.4 production.api-internal.service.consul. 0 IN A 10.0.1.1 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.5 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.1 $
15. Endpoint of internal API requests • There are subsystems in • Admin tool, Batch, Worker, Web, … • Each subsystem calls Mercari API via HTTP • e.g. production.api-internal.service.consul
16. Load balancer for internal API requests global network private network Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
17. Load balancer for internal API requests private network Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP production.api-internal.service.consul production.api-internal.service.consul global network
18. Listing nodes in service • Use case in • Gathering load balancer nodes in application deployment
19. Mercari deployment ChatOps with Slack yes EFQMPZCPU App App App nginx nginx nginx
20. ChatOps with Slack yes EFQMPZCPU ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
21. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down Mercari deployment App App App nginx nginx nginx
22. ChatOps with Slack yes EFQMPZCPU rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
23. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up Mercari deployment App App App nginx nginx nginx
24. Repeat…
25. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down or up Turn of App App App nginx nginx nginx
26. Listing all nodes in api-internal service $ curl -s \ consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
27. Mercari deployment details https://speakerdeck.com/kazeburo/continuous-improvement- applications-and-mercari-sre-number-retty-tech-cafe
28. Configuration deployment
29. Configuration deployment with • Via • consul event • consul-template • Stretcher • Now uses consul event & Stretcher
30. Load balancer configuration with • Dynamic upstream with consul-template • Instead uses ngx_dynamic_upstream • External files • TLS Session Tickets • IP black list
31. TLS Session Tickets # TLS Session Tickets # openssl rand 48 > ssl_session_ticket ssl_session_tickets on; ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket; • Client side session cache for TLS • nginx loads from local file on startup • Periodic update is required for forward secrecy • updates automatically with consul event
32. Configuration 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “watches”: [ { “type”: “event”, “name”: “tls-session-ticket-refresh”, “handler”: “/path/to/tls_session_ticket_refresh” ] } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
33. Firing event with consul event $ consul event \ -name=“tls-session-ticket-refresh” \ $(openssl rand 48 | base64) consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent Consul server # real processing is # more complicated cat $body | jq -r ‘.Payload’ | \ base64 -d | base64 -d > \ /path/to/tls_session_ticket service nginx reload tls_session_ticket_refresh Automated job Transfer payload
34. Payload should be small? https://www.consul.io/docs/commands/event.html Transfer large distribution?
35. Stretcher • A deployment tool with Consul / Serf event • github.com/fujiwara/strecher • Payload is expressed as file-path or URL $ consul event \ -name deploy-xxx \ “s3://example.com/distribution.tar.gz” { “watches”: [ { “type”: “event”, “name”: “deploy-xxx”, “handler”: “/path/to/stretcher” ] } ▪ Trigger ▪ Configuration $ consul event \ -name deploy-xxx \ “/path/to/distribution.tar.gz” or
36. Thanksʂ
37. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System Engineer • https://www.mercari.com/jp/jobs/backend/