$30 off During Our Annual Pro Sale. View Details »

Thalassa - DenverJS

Mike Brevoort
September 12, 2013

Thalassa - DenverJS

Thalassa is a system of components geared to enable continuous deployment scenarios through dynamic configuration of HAProxy load balancers.

http://www.meetup.com/Denver-JS/events/129876632/

Mike Brevoort

September 12, 2013
Tweet

More Decks by Mike Brevoort

Other Decks in Technology

Transcript

  1. Thalassa
    A system for continuous delivery
    Mike Brevoort
    DenverJS
    September 12, 2013
    Friday, September 13, 13

    View Slide

  2. Mike Brevoort
    @mbrevoort
    I work at Pearson!
    Friday, September 13, 13

    View Slide

  3. Background
    Friday, September 13, 13

    View Slide

  4. 18 months ago
    Friday, September 13, 13

    View Slide

  5. Ultimatum
    Friday, September 13, 13

    View Slide

  6. Continuous
    Deployment
    Friday, September 13, 13

    View Slide

  7. • no downtime
    • no user impact
    • no connection drops
    • repeatable
    • reliable
    Continuos Deployment
    Friday, September 13, 13

    View Slide

  8. deploy(n+1) → 0 effort
    Friday, September 13, 13

    View Slide

  9. But How?
    Friday, September 13, 13

    View Slide

  10. Typically handled at a
    load balancer
    Friday, September 13, 13

    View Slide

  11. A/B style deployment
    • Minimize disruption
    • Reliable fast rollbacks
    • Deployments don’t cripple capacity
    Friday, September 13, 13

    View Slide

  12. The Problem
    How do we automate the load
    balancer and A/B style deployments?
    for heterogeneous apps?
    (node.js, Java, etc.)
    in heterogenous environments
    (public/private cloud, bare metal)
    Friday, September 13, 13

    View Slide

  13. “Necessity is the
    mother of taking
    chances”
    Mark Twain
    Friday, September 13, 13

    View Slide

  14. Could we use Seaport
    and a node as a
    dynamic, programmable
    proxy?
    the “itch”
    https://github.com/substack/seaport
    Friday, September 13, 13

    View Slide

  15. Why node.js?
    • Many connections
    • Streaming data
    • Relatively lightweight
    • Very programmable
    • Seaport was an excellent head-start
    Friday, September 13, 13

    View Slide

  16. but node as the proxy?
    Friday, September 13, 13

    View Slide

  17. HAProxy
    • Software router/load balancer
    • A reliable workhorse
    • Feature rich
    • Can dynamically reload configuration
    without dropping connections
    Friday, September 13, 13

    View Slide

  18. Thalassa
    • Registry of services
    • API for configuring HAProxy
    • Dynamically set HAProxy backend servers
    based on registry services name and
    version
    • Fault tolerant, do no harm
    fka “Spindrift”
    Friday, September 13, 13

    View Slide

  19. 3 Major Components
    • Thalassa Server/Client Registry
    • Aqueduct
    • Crowsnest
    Friday, September 13, 13

    View Slide

  20. Thalassa
    • Lightweight service registry
    • Formerly build on top of @substack’s
    Seaport
    • Ping and expire based approach through
    HTTP API
    • Built on Redis, axon for pub-sub
    • https://github.com/PearsonEducation/
    thalassa
    Friday, September 13, 13

    View Slide

  21. Aqueduct
    • REST API for HAProxy
    • Manage HAProxy process
    • Dynamically generates config
    • Streams HAProxy stats
    • Configuration as a CRDT
    • enables peering of Aqueduct servers
    • https://github.com/PearsonEducation/
    thalassa-aqueduct
    Friday, September 13, 13

    View Slide

  22. Crowsnest
    • Visualize what’s happening in the system
    • https://github.com/PearsonEducation/
    thalassa-crowsnest
    Friday, September 13, 13

    View Slide

  23. Thalassa
    Server
    Friday, September 13, 13

    View Slide

  24. Thalassa
    Server
    A@1
    Friday, September 13, 13

    View Slide

  25. Thalassa
    Server
    A@1
    Friday, September 13, 13

    View Slide

  26. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    Friday, September 13, 13

    View Slide

  27. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    axon
    socket
    Friday, September 13, 13

    View Slide

  28. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    80
    FE
    axon
    socket
    Friday, September 13, 13

    View Slide

  29. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13

    View Slide

  30. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13

    View Slide

  31. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13

    View Slide

  32. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13

    View Slide

  33. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13

    View Slide

  34. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  35. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  36. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  37. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  38. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  39. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13

    View Slide

  40. Demo
    Friday, September 13, 13

    View Slide

  41. Friday, September 13, 13

    View Slide

  42. Friday, September 13, 13

    View Slide

  43. Friday, September 13, 13

    View Slide

  44. Highlights
    Friday, September 13, 13

    View Slide

  45. Monday morning
    August 26th
    Friday, September 13, 13

    View Slide

  46. commutative data
    >
    Friday, September 13, 13

    View Slide

  47. Friday, September 13, 13

    View Slide

  48. Friday, September 13, 13

    View Slide

  49. Thalassa Registry
    • Redis Implementation
    • Key/Value for each registration
    • Sorted set of keys ordered by time to
    expire
    Friday, September 13, 13

    View Slide

  50. /classroom-ui/0.1.32/10.199.240.52/8080 {JSON}
    /classroom-ui/0.1.32/10.199.240.51/8080 {JSON}
    /myapp/1.0.0/10.10.10.10/8888 {JSON}
    /yourapp/8.0.1/192.168.8.10/80 {JSON}
    https://github.com/PearsonEducation/thalassa/blob/master/lib/server/RedisData.js
    key value
    Redis Registrations
    Friday, September 13, 13

    View Slide

  51. /classroom-ui/0.1.32/10.199.240.52/8080 timeToExpire
    /classroom-ui/0.1.32/10.199.240.51/8080
    /myapp/1.0.0/10.10.10.10/8888
    /yourapp/8.0.1/192.168.8.10/80
    timeToExpire
    timeToExpire
    timeToExpire
    Redis Registrations
    Sorted Set
    The Reaper
    Friday, September 13, 13

    View Slide

  52. local res =
    redis.call('ZRANGEBYSCORE','__thalassa.registrations', 0, 1,
    'LIMIT', 0, 100 )
    if #res > 0 then
    redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 )
    return res
    else
    return false
    end
    function onOnline (reg) {
    pub.send(reg.id, 'online', reg.stringify());
    }
    function onOffline(regId) {
    pub.send(regId, 'offline');
    }
    The Reaper
    Friday, September 13, 13

    View Slide

  53. Aqueduct Config
    Templates
    • haproxy.cfg.tmpl template for haproxy.cfg
    • Handlebars.js
    • Handlebars macros
    https://github.com/PearsonEducation/thalassa-aqueduct/blob/master/default.haproxycfg.tmpl
    Friday, September 13, 13

    View Slide

  54. global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    daemon
    maxconn 4096
    user haproxy
    group haproxy
    # stats socket {{haproxySocketPath}} user USER_RUNNING_NODE_PROCESS level admin
    stats socket {{haproxySocketPath}} level admin
    defaults
    log global
    option dontlognull
    option redispatch
    retries 3
    maxconn 2000
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    listen stats :1988
    mode http
    stats enable
    stats uri /
    stats refresh 2s
    stats realm Haproxy\ Stats
    stats auth showme:showme
    {{#each frontends}}
    frontend {{name}}
    {{frontendHelper this}}
    {{/each}}
    {{#each backends}}
    backend {{name}}
    {{backendHelper this}}
    {{/each}}
    Friday, September 13, 13

    View Slide

  55. Use of Streams
    • Aqueduct streams config and HAProxy
    stats to Crowsnest over a websocket
    • Crowsnest aggregates and streams to the
    browser
    • MuxDemux
    Friday, September 13, 13

    View Slide

  56. registrations
    config
    stats
    stats
    config
    stats
    config
    control
    stream
    MuxDemux
    Streams
    Friday, September 13, 13

    View Slide

  57. registrations
    config
    stats
    stats
    config
    stats
    Aqueduct Aqueduct
    config
    Crowsnest
    Server
    Aqueduct
    Thalassa
    Server
    control
    stream
    MuxDemux
    Streams
    Friday, September 13, 13

    View Slide

  58. New to streams?
    https://github.com/substack/stream-adventure
    Friday, September 13, 13

    View Slide

  59. When?
    • Modules will be officially published this
    month within the next several weeks
    • Repositories
    http://github.com/PearsonEducation
    Friday, September 13, 13

    View Slide

  60. Questions?
    Friday, September 13, 13

    View Slide