Load balancer management with Consul

by Tatsuhiko Kubo

Slide 1

Slide 1 text

Tatsuhiko Kubo@cubicdaiya Consul Casual Talks#1 2016/08/01 Load balancer management with Cunsul

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Load balancers in • nginx • L7 (HTTP or HTTPS) • L4 (TCP) • OpenResty • Dynamic behavior with Lua

Slide 5

Slide 5 text

nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 SPDY/HTTP2 SPDY/HTTP2 EC2 EC2 Load balancer for App

Slide 6

Slide 6 text

Load balancer for internal API requests global network private network Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP

Slide 7

Slide 7 text

Load balancer for Search app app app OpenResty Latest Indices All Indices Contents cache & Dynamic Balancing HTTP HTTP

Slide 8

Slide 8 text

Load balancer for Push Gaurun Gaurun Gaurun

Slide 9

Slide 9 text

in • Service discovery • Load balancer, SMTP, etc… • Conﬁguration deployment • TLS Session Tickets, IP black list • Distributed lock • Ensure running only 1 process always in clusters

Slide 10

Slide 10 text

Service discovery

Slide 11

Slide 11 text

Service discovery with • Use case in • Internal DNS as • Endpoint of internal API, DNS-RR • Via HTTP APIs • Listing nodes in service

Slide 12

Slide 12 text

Service discovery with 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “service”: { “name”: “api-internal”, “tags”: [“production”], … } } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6

Slide 13

Slide 13 text

Listing all nodes in api-internal service $ curl -s \ consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”

Slide 14

Slide 14 text

DNS-RR in api-internal service $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.6 production.api-internal.service.consul. 0 IN A 10.0.1.4 production.api-internal.service.consul. 0 IN A 10.0.1.1 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.5 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.1 $

Slide 15

Slide 15 text

Endpoint of internal API requests • There are subsystems in • Admin tool, Batch, Worker, Web, … • Each subsystem calls Mercari API via HTTP • e.g. production.api-internal.service.consul

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Load balancer for internal API requests private network Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP production.api-internal.service.consul production.api-internal.service.consul global network

Slide 18

Slide 18 text

Listing nodes in service • Use case in • Gathering load balancer nodes in application deployment

Slide 19

Slide 19 text

Mercari deployment ChatOps with Slack yes EFQMPZCPU App App App nginx nginx nginx

Slide 20

Slide 20 text

ChatOps with Slack yes EFQMPZCPU ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx

Slide 21

Slide 21 text

ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down Mercari deployment App App App nginx nginx nginx

Slide 22

Slide 22 text

ChatOps with Slack yes EFQMPZCPU rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx

Slide 23

Slide 23 text

ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up Mercari deployment App App App nginx nginx nginx

Slide 24

Slide 24 text

Repeat…

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Mercari deployment details https://speakerdeck.com/kazeburo/continuous-improvement- applications-and-mercari-sre-number-retty-tech-cafe

Slide 28

Slide 28 text

Conﬁguration deployment

Slide 29

Slide 29 text

Conﬁguration deployment with • Via • consul event • consul-template • Stretcher • Now uses consul event & Stretcher

Slide 30

Slide 30 text

Load balancer conﬁguration with • Dynamic upstream with consul-template • Instead uses ngx_dynamic_upstream • External ﬁles • TLS Session Tickets • IP black list

Slide 31

Slide 31 text

TLS Session Tickets # TLS Session Tickets # openssl rand 48 > ssl_session_ticket ssl_session_tickets on; ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket; • Client side session cache for TLS • nginx loads from local ﬁle on startup • Periodic update is required for forward secrecy • updates automatically with consul event

Slide 32

Slide 32 text

Conﬁguration 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “watches”: [ { “type”: “event”, “name”: “tls-session-ticket-refresh”, “handler”: “/path/to/tls_session_ticket_refresh” ] } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6

Slide 33

Slide 33 text

Firing event with consul event $ consul event \ -name=“tls-session-ticket-refresh” \ $(openssl rand 48 | base64) consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent Consul server # real processing is # more complicated cat $body | jq -r ‘.Payload’ | \ base64 -d | base64 -d > \ /path/to/tls_session_ticket service nginx reload tls_session_ticket_refresh Automated job Transfer payload

Slide 34

Slide 34 text

Payload should be small? https://www.consul.io/docs/commands/event.html Transfer large distribution?

Slide 35

Slide 35 text

Stretcher • A deployment tool with Consul / Serf event • github.com/fujiwara/strecher • Payload is expressed as ﬁle-path or URL $ consul event \ -name deploy-xxx \ “s3://example.com/distribution.tar.gz” { “watches”: [ { “type”: “event”, “name”: “deploy-xxx”, “handler”: “/path/to/stretcher” ] } ■ Trigger ■ Conﬁguration $ consul event \ -name deploy-xxx \ “/path/to/distribution.tar.gz” or

Slide 36

Slide 36 text

Thanksʂ

Slide 37

Slide 37 text

We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System Engineer • https://www.mercari.com/jp/jobs/backend/