Thalassa - DenverJS

Thalassa is a system of components geared to enable continuous deployment scenarios through dynamic configuration of HAProxy load balancers.

http://www.meetup.com/Denver-JS/events/129876632/

Transcript

1. Thalassa
   A system for continuous delivery
   Mike Brevoort
   DenverJS
   September 12, 2013
   Friday, September 13, 13
   

   View Slide

2. Mike Brevoort
   @mbrevoort
   I work at Pearson!
   Friday, September 13, 13
   

   View Slide

3. Background
   Friday, September 13, 13
   

   View Slide

4. 18 months ago
   Friday, September 13, 13
   

   View Slide

5. Ultimatum
   Friday, September 13, 13
   

   View Slide

6. Continuous
   Deployment
   Friday, September 13, 13
   

   View Slide

7. • no downtime
   • no user impact
   • no connection drops
   • repeatable
   • reliable
   Continuos Deployment
   Friday, September 13, 13
   

   View Slide

8. deploy(n+1) → 0 effort
   Friday, September 13, 13
   

   View Slide

9. But How?
   Friday, September 13, 13
   

   View Slide

10. Typically handled at a
    load balancer
    Friday, September 13, 13
    

    View Slide

11. A/B style deployment
    • Minimize disruption
    • Reliable fast rollbacks
    • Deployments don’t cripple capacity
    Friday, September 13, 13
    

    View Slide

12. The Problem
    How do we automate the load
    balancer and A/B style deployments?
    for heterogeneous apps?
    (node.js, Java, etc.)
    in heterogenous environments
    (public/private cloud, bare metal)
    Friday, September 13, 13
    

    View Slide

13. “Necessity is the
    mother of taking
    chances”
    Mark Twain
    Friday, September 13, 13
    

    View Slide

14. Could we use Seaport
    and a node as a
    dynamic, programmable
    proxy?
    the “itch”
    https://github.com/substack/seaport
    Friday, September 13, 13
    

    View Slide

15. Why node.js?
    • Many connections
    • Streaming data
    • Relatively lightweight
    • Very programmable
    • Seaport was an excellent head-start
    Friday, September 13, 13
    

    View Slide

16. but node as the proxy?
    Friday, September 13, 13
    

    View Slide

17. HAProxy
    • Software router/load balancer
    • A reliable workhorse
    • Feature rich
    • Can dynamically reload configuration
    without dropping connections
    Friday, September 13, 13
    

    View Slide

18. Thalassa
    • Registry of services
    • API for configuring HAProxy
    • Dynamically set HAProxy backend servers
    based on registry services name and
    version
    • Fault tolerant, do no harm
    fka “Spindrift”
    Friday, September 13, 13
    

    View Slide

19. 3 Major Components
    • Thalassa Server/Client Registry
    • Aqueduct
    • Crowsnest
    Friday, September 13, 13
    

    View Slide

20. Thalassa
    • Lightweight service registry
    • Formerly build on top of @substack’s
    Seaport
    • Ping and expire based approach through
    HTTP API
    • Built on Redis, axon for pub-sub
    • https://github.com/PearsonEducation/
    thalassa
    Friday, September 13, 13
    

    View Slide

21. Aqueduct
    • REST API for HAProxy
    • Manage HAProxy process
    • Dynamically generates config
    • Streams HAProxy stats
    • Configuration as a CRDT
    • enables peering of Aqueduct servers
    • https://github.com/PearsonEducation/
    thalassa-aqueduct
    Friday, September 13, 13
    

    View Slide

22. Crowsnest
    • Visualize what’s happening in the system
    • https://github.com/PearsonEducation/
    thalassa-crowsnest
    Friday, September 13, 13
    

    View Slide

23. Thalassa
    Server
    Friday, September 13, 13
    

    View Slide

24. Thalassa
    Server
    A@1
    Friday, September 13, 13
    

    View Slide

25. Thalassa
    Server
    A@1
    Friday, September 13, 13
    

    View Slide

26. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    Friday, September 13, 13
    

    View Slide

27. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    axon
    socket
    Friday, September 13, 13
    

    View Slide

28. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    80
    FE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

29. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

30. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

31. Thalassa
    Server
    Aqueduct
    HAProxy
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

32. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

33. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    axon
    socket
    Friday, September 13, 13
    

    View Slide

34. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    A@1
    80
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

35. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

36. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

37. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

38. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

39. Thalassa
    Server
    Aqueduct
    HAProxy
    Crowsnest
    A@1
    A@1
    A@1
    80 A@2
    A@2
    A@2
    FE BE
    streaming stats
    axon
    socket
    Friday, September 13, 13
    

    View Slide

40. Demo
    Friday, September 13, 13
    

    View Slide

41. Friday, September 13, 13
    

    View Slide

42. Friday, September 13, 13
    

    View Slide

43. Friday, September 13, 13
    

    View Slide

44. Highlights
    Friday, September 13, 13
    

    View Slide

45. Monday morning
    August 26th
    Friday, September 13, 13
    

    View Slide

46. commutative data
    >
    Friday, September 13, 13
    

    View Slide

47. Friday, September 13, 13
    

    View Slide

48. Friday, September 13, 13
    

    View Slide

49. Thalassa Registry
    • Redis Implementation
    • Key/Value for each registration
    • Sorted set of keys ordered by time to
    expire
    Friday, September 13, 13
    

    View Slide

50. /classroom-ui/0.1.32/10.199.240.52/8080 {JSON}
    /classroom-ui/0.1.32/10.199.240.51/8080 {JSON}
    /myapp/1.0.0/10.10.10.10/8888 {JSON}
    /yourapp/8.0.1/192.168.8.10/80 {JSON}
    https://github.com/PearsonEducation/thalassa/blob/master/lib/server/RedisData.js
    key value
    Redis Registrations
    Friday, September 13, 13
    

    View Slide

51. /classroom-ui/0.1.32/10.199.240.52/8080 timeToExpire
    /classroom-ui/0.1.32/10.199.240.51/8080
    /myapp/1.0.0/10.10.10.10/8888
    /yourapp/8.0.1/192.168.8.10/80
    timeToExpire
    timeToExpire
    timeToExpire
    Redis Registrations
    Sorted Set
    The Reaper
    Friday, September 13, 13
    

    View Slide

52. local res =
    redis.call('ZRANGEBYSCORE','__thalassa.registrations', 0, 1,
    'LIMIT', 0, 100 )
    if #res > 0 then
    redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 )
    return res
    else
    return false
    end
    function onOnline (reg) {
    pub.send(reg.id, 'online', reg.stringify());
    }
    function onOffline(regId) {
    pub.send(regId, 'offline');
    }
    The Reaper
    Friday, September 13, 13
    

    View Slide

53. Aqueduct Config
    Templates
    • haproxy.cfg.tmpl template for haproxy.cfg
    • Handlebars.js
    • Handlebars macros
    https://github.com/PearsonEducation/thalassa-aqueduct/blob/master/default.haproxycfg.tmpl
    Friday, September 13, 13
    

    View Slide

54. global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    daemon
    maxconn 4096
    user haproxy
    group haproxy
    # stats socket {{haproxySocketPath}} user USER_RUNNING_NODE_PROCESS level admin
    stats socket {{haproxySocketPath}} level admin
    defaults
    log global
    option dontlognull
    option redispatch
    retries 3
    maxconn 2000
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    listen stats :1988
    mode http
    stats enable
    stats uri /
    stats refresh 2s
    stats realm Haproxy\ Stats
    stats auth showme:showme
    {{#each frontends}}
    frontend {{name}}
    {{frontendHelper this}}
    {{/each}}
    {{#each backends}}
    backend {{name}}
    {{backendHelper this}}
    {{/each}}
    Friday, September 13, 13
    

    View Slide

55. Use of Streams
    • Aqueduct streams config and HAProxy
    stats to Crowsnest over a websocket
    • Crowsnest aggregates and streams to the
    browser
    • MuxDemux
    Friday, September 13, 13
    

    View Slide

56. registrations
    config
    stats
    stats
    config
    stats
    config
    control
    stream
    MuxDemux
    Streams
    Friday, September 13, 13
    

    View Slide

57. registrations
    config
    stats
    stats
    config
    stats
    Aqueduct Aqueduct
    config
    Crowsnest
    Server
    Aqueduct
    Thalassa
    Server
    control
    stream
    MuxDemux
    Streams
    Friday, September 13, 13
    

    View Slide

58. New to streams?
    https://github.com/substack/stream-adventure
    Friday, September 13, 13
    

    View Slide

59. When?
    • Modules will be officially published this
    month within the next several weeks
    • Repositories
    http://github.com/PearsonEducation
    Friday, September 13, 13
    

    View Slide

60. Questions?
    Friday, September 13, 13
    

    View Slide