$30 off During Our Annual Pro Sale. View Details »

Load balancer management with Consul

Load balancer management with Consul

Tatsuhiko Kubo

August 02, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Tatsuhiko Kubo@cubicdaiya
    Consul Casual Talks#1 2016/08/01
    Load balancer management with Cunsul

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Engineer, SRE @ Mercari, Inc.

    View Slide

  3. View Slide

  4. Load balancers in
    • nginx
    • L7 (HTTP or HTTPS)
    • L4 (TCP)
    • OpenResty
    • Dynamic behavior with Lua

    View Slide

  5. nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    IAM Add-on Example:
    IAM Add-on
    ence
    )
    Assignment/
    Task
    Requester
    Workers
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    JP US
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia C
    d
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    EC2
    EC2 EC2 EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    SPDY/HTTP2
    SPDY/HTTP2
    EC2 EC2
    Load balancer for App

    View Slide

  6. Load balancer for internal API requests
    global network private network
    Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Example:
    Requester
    Workers
    SPDY/HTTP2
    mercari API
    HTTP
    subsystems
    subsystems
    HTTP
    HTTP

    View Slide

  7. Load balancer for Search
    app
    app
    app
    OpenResty
    Latest Indices
    All Indices
    Contents cache
    &
    Dynamic Balancing
    HTTP
    HTTP

    View Slide

  8. Load balancer for Push
    Gaurun
    Gaurun
    Gaurun

    View Slide

  9. in
    • Service discovery
    • Load balancer, SMTP, etc…
    • Configuration deployment
    • TLS Session Tickets, IP black list
    • Distributed lock
    • Ensure running only 1 process always in clusters

    View Slide

  10. Service discovery

    View Slide

  11. Service discovery with
    • Use case in
    • Internal DNS as
    • Endpoint of internal API, DNS-RR
    • Via HTTP APIs
    • Listing nodes in service

    View Slide

  12. Service discovery with
    10.0.1.1
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    {
    “service”: {
    “name”: “api-internal”,
    “tags”: [“production”],

    }
    }
    10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6

    View Slide

  13. Listing all nodes in api-internal service
    $ curl -s \
    consul-server:8500/v1/catalog/service/api-internal \
    jq ‘.[].Address’
    “10.0.1.1”
    “10.0.1.2”
    “10.0.1.3”
    “10.0.1.4”
    “10.0.1.5”
    “10.0.1.6”

    View Slide

  14. DNS-RR in api-internal service
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.6
    production.api-internal.service.consul. 0 IN A 10.0.1.4
    production.api-internal.service.consul. 0 IN A 10.0.1.1
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.2
    production.api-internal.service.consul. 0 IN A 10.0.1.3
    production.api-internal.service.consul. 0 IN A 10.0.1.5
    $ dig production.api-internal.service.consul | egrep ‘^production’
    production.api-internal.service.consul. 0 IN A 10.0.1.3
    production.api-internal.service.consul. 0 IN A 10.0.1.2
    production.api-internal.service.consul. 0 IN A 10.0.1.1
    $

    View Slide

  15. Endpoint of internal API requests
    • There are subsystems in
    • Admin tool, Batch, Worker, Web, …
    • Each subsystem calls Mercari API via HTTP
    • e.g. production.api-internal.service.consul

    View Slide

  16. Load balancer for internal API requests
    global network private network
    Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Example:
    Requester
    Workers
    SPDY/HTTP2
    mercari API
    HTTP
    subsystems
    subsystems
    HTTP
    HTTP

    View Slide

  17. Load balancer for internal API requests
    private network
    Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Example:
    Requester
    Workers
    SPDY/HTTP2
    mercari API
    HTTP
    subsystems
    subsystems
    HTTP
    HTTP
    production.api-internal.service.consul
    production.api-internal.service.consul
    global network

    View Slide

  18. Listing nodes in service
    • Use case in
    • Gathering load balancer nodes in
    application deployment

    View Slide

  19. Mercari deployment
    ChatOps with Slack
    yes
    EFQMPZCPU
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  20. ChatOps with Slack
    yes
    EFQMPZCPU
    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  21. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  22. ChatOps with Slack
    yes
    EFQMPZCPU
    rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  23. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    up
    Mercari deployment
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  24. Repeat…

    View Slide

  25. ChatOps with Slack
    yes
    EFQMPZCPU
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down or up
    Turn of
    App
    App
    App
    nginx
    nginx
    nginx

    View Slide

  26. Listing all nodes in api-internal service
    $ curl -s \
    consul-server:8500/v1/catalog/service/api-internal \
    jq ‘.[].Address’
    “10.0.1.1”
    “10.0.1.2”
    “10.0.1.3”
    “10.0.1.4”
    “10.0.1.5”
    “10.0.1.6”

    View Slide

  27. Mercari deployment details
    https://speakerdeck.com/kazeburo/continuous-improvement-
    applications-and-mercari-sre-number-retty-tech-cafe

    View Slide

  28. Configuration deployment

    View Slide

  29. Configuration deployment with
    • Via
    • consul event
    • consul-template
    • Stretcher
    • Now uses consul event & Stretcher

    View Slide

  30. Load balancer configuration with
    • Dynamic upstream with consul-template
    • Instead uses ngx_dynamic_upstream
    • External files
    • TLS Session Tickets
    • IP black list

    View Slide

  31. TLS Session Tickets
    # TLS Session Tickets
    # openssl rand 48 > ssl_session_ticket
    ssl_session_tickets on;
    ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket;
    • Client side session cache for TLS
    • nginx loads from local file on startup
    • Periodic update is required for forward secrecy
    • updates automatically with consul event

    View Slide

  32. Configuration
    10.0.1.1
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    {
    “watches”: [
    {
    “type”: “event”,
    “name”: “tls-session-ticket-refresh”,
    “handler”: “/path/to/tls_session_ticket_refresh”
    ]
    }
    10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6

    View Slide

  33. Firing event with consul event
    $ consul event \
    -name=“tls-session-ticket-refresh” \
    $(openssl rand 48 | base64)
    consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent
    Consul server
    # real processing is
    # more complicated
    cat $body | jq -r ‘.Payload’ | \
    base64 -d | base64 -d > \
    /path/to/tls_session_ticket
    service nginx reload
    tls_session_ticket_refresh
    Automated job
    Transfer payload

    View Slide

  34. Payload should be small?
    https://www.consul.io/docs/commands/event.html
    Transfer large distribution?

    View Slide

  35. Stretcher
    • A deployment tool with Consul / Serf event
    • github.com/fujiwara/strecher
    • Payload is expressed as file-path or URL
    $ consul event \
    -name deploy-xxx \
    “s3://example.com/distribution.tar.gz”
    {
    “watches”: [
    {
    “type”: “event”,
    “name”: “deploy-xxx”,
    “handler”: “/path/to/stretcher”
    ]
    }
    ■ Trigger ■ Configuration
    $ consul event \
    -name deploy-xxx \
    “/path/to/distribution.tar.gz”
    or

    View Slide

  36. Thanksʂ

    View Slide

  37. We are hiring!
    • SRE
    • https://www.mercari.com/jp/jobs/sre/
    • Backend System Engineer
    • https://www.mercari.com/jp/jobs/backend/

    View Slide