Load balancer management with Consul

Transcript

1. Tatsuhiko Kubo@cubicdaiya
   Consul Casual Talks#1 2016/08/01
   Load balancer management with Cunsul
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Engineer, SRE @ Mercari, Inc.
   

   View Slide

3. View Slide

4. Load balancers in
   • nginx
   • L7 (HTTP or HTTPS)
   • L4 (TCP)
   • OpenResty
   • Dynamic behavior with Lua
   

   View Slide

5. nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   IAM Add-on Example:
   IAM Add-on
   ence
   )
   Assignment/
   Task
   Requester
   Workers
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   JP US
   nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia C
   d
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   EC2
   EC2 EC2 EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   SPDY/HTTP2
   SPDY/HTTP2
   EC2 EC2
   Load balancer for App
   

   View Slide

6. Load balancer for internal API requests
   global network private network
   Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Example:
   Requester
   Workers
   SPDY/HTTP2
   mercari API
   HTTP
   subsystems
   subsystems
   HTTP
   HTTP
   

   View Slide

7. Load balancer for Search
   app
   app
   app
   OpenResty
   Latest Indices
   All Indices
   Contents cache
   &
   Dynamic Balancing
   HTTP
   HTTP
   

   View Slide

8. Load balancer for Push
   Gaurun
   Gaurun
   Gaurun
   

   View Slide

9. in
   • Service discovery
   • Load balancer, SMTP, etc…
   • Configuration deployment
   • TLS Session Tickets, IP black list
   • Distributed lock
   • Ensure running only 1 process always in clusters
   

   View Slide

10. Service discovery
    

    View Slide

11. Service discovery with
    • Use case in
    • Internal DNS as
    • Endpoint of internal API, DNS-RR
    • Via HTTP APIs
    • Listing nodes in service
    

    View Slide

12. Service discovery with
    10.0.1.1
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    {
    “service”: {
    “name”: “api-internal”,
    “tags”: [“production”],
    …
    }
    }
    10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
    

    View Slide

13. Listing all nodes in api-internal service
    $ curl -s \
    consul-server:8500/v1/catalog/service/api-internal \
    jq ‘.[].Address’
    “10.0.1.1”
    “10.0.1.2”
    “10.0.1.3”
    “10.0.1.4”
    “10.0.1.5”
    “10.0.1.6”
    

    View Slide

14. DNS-RR in api-internal service
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.6
    production.api-internal.service.consul. 0 IN A 10.0.1.4
    production.api-internal.service.consul. 0 IN A 10.0.1.1
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.2
    production.api-internal.service.consul. 0 IN A 10.0.1.3
    production.api-internal.service.consul. 0 IN A 10.0.1.5
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.3
    production.api-internal.service.consul. 0 IN A 10.0.1.2
    production.api-internal.service.consul. 0 IN A 10.0.1.1
    $
    

    View Slide

15. Endpoint of internal API requests
    • There are subsystems in
    • Admin tool, Batch, Worker, Web, …
    • Each subsystem calls Mercari API via HTTP
    • e.g. production.api-internal.service.consul
    

    View Slide

16. Load balancer for internal API requests
    global network private network
    Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Example:
    Requester
    Workers
    SPDY/HTTP2
    mercari API
    HTTP
    subsystems
    subsystems
    HTTP
    HTTP
    

    View Slide

17. Load balancer for internal API requests
    private network
    Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Example:
    Requester
    Workers
    SPDY/HTTP2
    mercari API
    HTTP
    subsystems
    subsystems
    HTTP
    HTTP
    production.api-internal.service.consul
    production.api-internal.service.consul
    global network
    

    View Slide

18. Listing nodes in service
    • Use case in
    • Gathering load balancer nodes in
    application deployment
    

    View Slide

19. Mercari deployment
    ChatOps with Slack
    yes
    EFQMPZCPU
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

20. ChatOps with Slack
    yes
    EFQMPZCPU
    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

21. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

22. ChatOps with Slack
    yes
    EFQMPZCPU
    rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

23. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    up
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

24. Repeat…
    

    View Slide

25. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down or up
    Turn of
    App
    App
    App
    nginx
    nginx
    nginx
    

    View Slide

26. Listing all nodes in api-internal service
    $ curl -s \
    consul-server:8500/v1/catalog/service/api-internal \
    jq ‘.[].Address’
    “10.0.1.1”
    “10.0.1.2”
    “10.0.1.3”
    “10.0.1.4”
    “10.0.1.5”
    “10.0.1.6”
    

    View Slide

27. Mercari deployment details
    https://speakerdeck.com/kazeburo/continuous-improvement-
    applications-and-mercari-sre-number-retty-tech-cafe
    

    View Slide

28. Configuration deployment
    

    View Slide

29. Configuration deployment with
    • Via
    • consul event
    • consul-template
    • Stretcher
    • Now uses consul event & Stretcher
    

    View Slide

30. Load balancer configuration with
    • Dynamic upstream with consul-template
    • Instead uses ngx_dynamic_upstream
    • External files
    • TLS Session Tickets
    • IP black list
    

    View Slide

31. TLS Session Tickets
    # TLS Session Tickets
    # openssl rand 48 > ssl_session_ticket
    ssl_session_tickets on;
    ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket;
    • Client side session cache for TLS
    • nginx loads from local file on startup
    • Periodic update is required for forward secrecy
    • updates automatically with consul event
    

    View Slide

32. Configuration
    10.0.1.1
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    {
    “watches”: [
    {
    “type”: “event”,
    “name”: “tls-session-ticket-refresh”,
    “handler”: “/path/to/tls_session_ticket_refresh”
    ]
    }
    10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
    

    View Slide

33. Firing event with consul event
    $ consul event \
    -name=“tls-session-ticket-refresh” \
    $(openssl rand 48 | base64)
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    Consul server
    # real processing is
    # more complicated
    cat $body | jq -r ‘.Payload’ | \
    base64 -d | base64 -d > \
    /path/to/tls_session_ticket
    service nginx reload
    tls_session_ticket_refresh
    Automated job
    Transfer payload
    

    View Slide

34. Payload should be small?
    https://www.consul.io/docs/commands/event.html
    Transfer large distribution?
    

    View Slide

35. Stretcher
    • A deployment tool with Consul / Serf event
    • github.com/fujiwara/strecher
    • Payload is expressed as file-path or URL
    $ consul event \
    -name deploy-xxx \
    “s3://example.com/distribution.tar.gz”
    {
    “watches”: [
    {
    “type”: “event”,
    “name”: “deploy-xxx”,
    “handler”: “/path/to/stretcher”
    ]
    }
    ■ Trigger ■ Configuration
    $ consul event \
    -name deploy-xxx \
    “/path/to/distribution.tar.gz”
    or
    

    View Slide

36. Thanksʂ
    

    View Slide

37. We are hiring!
    • SRE
    • https://www.mercari.com/jp/jobs/sre/
    • Backend System Engineer
    • https://www.mercari.com/jp/jobs/backend/
    

    View Slide