Load balancer management with Consul

Load balancer management with Consul

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

August 02, 2016
Tweet

Transcript

  1. Tatsuhiko Kubo@cubicdaiya Consul Casual Talks#1 2016/08/01 Load balancer management with

    Cunsul
  2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

  3. None
  4. Load balancers in • nginx • L7 (HTTP or HTTPS)

    • L4 (TCP) • OpenResty • Dynamic behavior with Lua
  5. nginx nginx nginx ©2011 Amazon Web Services LLC or its

    affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 SPDY/HTTP2 SPDY/HTTP2 EC2 EC2 Load balancer for App
  6. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  7. Load balancer for Search app app app OpenResty Latest Indices

    All Indices Contents cache & Dynamic Balancing HTTP HTTP
  8. Load balancer for Push Gaurun Gaurun Gaurun

  9. in • Service discovery • Load balancer, SMTP, etc… •

    Configuration deployment • TLS Session Tickets, IP black list • Distributed lock • Ensure running only 1 process always in clusters
  10. Service discovery

  11. Service discovery with • Use case in • Internal DNS

    as • Endpoint of internal API, DNS-RR • Via HTTP APIs • Listing nodes in service
  12. Service discovery with 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent

    { “service”: { “name”: “api-internal”, “tags”: [“production”], … } } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  13. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  14. DNS-RR in api-internal service $ dig production.api-internal.service.consul | egrep ‘^production’

    production.api-internal.service.consul. 0 IN A 10.0.1.6 production.api-internal.service.consul. 0 IN A 10.0.1.4 production.api-internal.service.consul. 0 IN A 10.0.1.1 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.5 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.1 $
  15. Endpoint of internal API requests • There are subsystems in

    • Admin tool, Batch, Worker, Web, … • Each subsystem calls Mercari API via HTTP • e.g. production.api-internal.service.consul
  16. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  17. Load balancer for internal API requests private network Multimedia Corporate

    data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP production.api-internal.service.consul production.api-internal.service.consul global network
  18. Listing nodes in service • Use case in • Gathering

    load balancer nodes in application deployment
  19. Mercari deployment ChatOps with Slack yes EFQMPZCPU App App App

    nginx nginx nginx
  20. ChatOps with Slack yes EFQMPZCPU ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  21. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down Mercari deployment App App App nginx nginx nginx
  22. ChatOps with Slack yes EFQMPZCPU rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  23. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up Mercari deployment App App App nginx nginx nginx
  24. Repeat…

  25. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down or up Turn of App App App nginx nginx nginx
  26. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  27. Mercari deployment details https://speakerdeck.com/kazeburo/continuous-improvement- applications-and-mercari-sre-number-retty-tech-cafe

  28. Configuration deployment

  29. Configuration deployment with • Via • consul event • consul-template

    • Stretcher • Now uses consul event & Stretcher
  30. Load balancer configuration with • Dynamic upstream with consul-template •

    Instead uses ngx_dynamic_upstream • External files • TLS Session Tickets • IP black list
  31. TLS Session Tickets # TLS Session Tickets # openssl rand

    48 > ssl_session_ticket ssl_session_tickets on; ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket; • Client side session cache for TLS • nginx loads from local file on startup • Periodic update is required for forward secrecy • updates automatically with consul event
  32. Configuration 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “watches”:

    [ { “type”: “event”, “name”: “tls-session-ticket-refresh”, “handler”: “/path/to/tls_session_ticket_refresh” ] } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  33. Firing event with consul event $ consul event \ -name=“tls-session-ticket-refresh”

    \ $(openssl rand 48 | base64) consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent Consul server # real processing is # more complicated cat $body | jq -r ‘.Payload’ | \ base64 -d | base64 -d > \ /path/to/tls_session_ticket service nginx reload tls_session_ticket_refresh Automated job Transfer payload
  34. Payload should be small? https://www.consul.io/docs/commands/event.html Transfer large distribution?

  35. Stretcher • A deployment tool with Consul / Serf event

    • github.com/fujiwara/strecher • Payload is expressed as file-path or URL $ consul event \ -name deploy-xxx \ “s3://example.com/distribution.tar.gz” { “watches”: [ { “type”: “event”, “name”: “deploy-xxx”, “handler”: “/path/to/stretcher” ] } ▪ Trigger ▪ Configuration $ consul event \ -name deploy-xxx \ “/path/to/distribution.tar.gz” or
  36. Thanksʂ

  37. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System

    Engineer • https://www.mercari.com/jp/jobs/backend/