Thalassa - DenverJS

Thalassa is a system of components geared to enable continuous deployment scenarios through dynamic configuration of HAProxy load balancers.

http://www.meetup.com/Denver-JS/events/129876632/

Transcript

1. Thalassa A system for continuous delivery Mike Brevoort DenverJS September 12, 2013 Friday, September 13, 13
2. Mike Brevoort @mbrevoort I work at Pearson! Friday, September 13, 13
3. Background Friday, September 13, 13
4. 18 months ago Friday, September 13, 13
5. Ultimatum Friday, September 13, 13
6. Continuous Deployment Friday, September 13, 13
7. • no downtime • no user impact • no connection drops • repeatable • reliable Continuos Deployment Friday, September 13, 13
8. deploy(n+1) → 0 effort Friday, September 13, 13
9. But How? Friday, September 13, 13
10. Typically handled at a load balancer Friday, September 13, 13
11. A/B style deployment • Minimize disruption • Reliable fast rollbacks • Deployments don’t cripple capacity Friday, September 13, 13
12. The Problem How do we automate the load balancer and A/B style deployments? for heterogeneous apps? (node.js, Java, etc.) in heterogenous environments (public/private cloud, bare metal) Friday, September 13, 13
13. “Necessity is the mother of taking chances” Mark Twain Friday, September 13, 13
14. Could we use Seaport and a node as a dynamic, programmable proxy? the “itch” https://github.com/substack/seaport Friday, September 13, 13
15. Why node.js? • Many connections • Streaming data • Relatively lightweight • Very programmable • Seaport was an excellent head-start Friday, September 13, 13
16. but node as the proxy? Friday, September 13, 13
17. HAProxy • Software router/load balancer • A reliable workhorse • Feature rich • Can dynamically reload configuration without dropping connections Friday, September 13, 13
18. Thalassa • Registry of services • API for configuring HAProxy • Dynamically set HAProxy backend servers based on registry services name and version • Fault tolerant, do no harm fka “Spindrift” Friday, September 13, 13
19. 3 Major Components • Thalassa Server/Client Registry • Aqueduct • Crowsnest Friday, September 13, 13
20. Thalassa • Lightweight service registry • Formerly build on top of @substack’s Seaport • Ping and expire based approach through HTTP API • Built on Redis, axon for pub-sub • https://github.com/PearsonEducation/ thalassa Friday, September 13, 13
21. Aqueduct • REST API for HAProxy • Manage HAProxy process • Dynamically generates config • Streams HAProxy stats • Configuration as a CRDT • enables peering of Aqueduct servers • https://github.com/PearsonEducation/ thalassa-aqueduct Friday, September 13, 13
22. Crowsnest • Visualize what’s happening in the system • https://github.com/PearsonEducation/ thalassa-crowsnest Friday, September 13, 13
23. Thalassa Server Friday, September 13, 13
24. Thalassa Server A@1 Friday, September 13, 13
25. Thalassa Server A@1 Friday, September 13, 13
26. Thalassa Server Aqueduct HAProxy A@1 Friday, September 13, 13
27. Thalassa Server Aqueduct HAProxy A@1 axon socket Friday, September 13, 13
28. Thalassa Server Aqueduct HAProxy A@1 80 FE axon socket Friday, September 13, 13
29. Thalassa Server Aqueduct HAProxy A@1 A@1 80 FE BE axon socket Friday, September 13, 13
30. Thalassa Server Aqueduct HAProxy A@1 A@1 80 FE BE axon socket Friday, September 13, 13
31. Thalassa Server Aqueduct HAProxy A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13
32. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13
33. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13
34. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE streaming stats axon socket Friday, September 13, 13
35. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 FE BE streaming stats axon socket Friday, September 13, 13
36. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
37. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
38. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
39. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
40. Demo Friday, September 13, 13
41. Friday, September 13, 13
42. Friday, September 13, 13
43. Friday, September 13, 13
44. Highlights Friday, September 13, 13
45. Monday morning August 26th Friday, September 13, 13
46. commutative data > Friday, September 13, 13
47. Friday, September 13, 13
48. Friday, September 13, 13
49. Thalassa Registry • Redis Implementation • Key/Value for each registration • Sorted set of keys ordered by time to expire Friday, September 13, 13
50. /classroom-ui/0.1.32/10.199.240.52/8080 {JSON} /classroom-ui/0.1.32/10.199.240.51/8080 {JSON} /myapp/1.0.0/10.10.10.10/8888 {JSON} /yourapp/8.0.1/192.168.8.10/80 {JSON} https://github.com/PearsonEducation/thalassa/blob/master/lib/server/RedisData.js key value Redis Registrations Friday, September 13, 13
51. /classroom-ui/0.1.32/10.199.240.52/8080 timeToExpire /classroom-ui/0.1.32/10.199.240.51/8080 /myapp/1.0.0/10.10.10.10/8888 /yourapp/8.0.1/192.168.8.10/80 timeToExpire timeToExpire timeToExpire Redis Registrations Sorted Set The Reaper Friday, September 13, 13
52. local res = redis.call('ZRANGEBYSCORE','__thalassa.registrations', 0, 1, 'LIMIT', 0, 100 ) if #res > 0 then redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 ) return res else return false end function onOnline (reg) { pub.send(reg.id, 'online', reg.stringify()); } function onOffline(regId) { pub.send(regId, 'offline'); } The Reaper Friday, September 13, 13
53. Aqueduct Config Templates • haproxy.cfg.tmpl template for haproxy.cfg • Handlebars.js • Handlebars macros https://github.com/PearsonEducation/thalassa-aqueduct/blob/master/default.haproxycfg.tmpl Friday, September 13, 13
54. global log 127.0.0.1 local0 log 127.0.0.1 local1 notice daemon maxconn 4096 user haproxy group haproxy # stats socket {{haproxySocketPath}} user USER_RUNNING_NODE_PROCESS level admin stats socket {{haproxySocketPath}} level admin defaults log global option dontlognull option redispatch retries 3 maxconn 2000 timeout connect 5000ms timeout client 50000ms timeout server 50000ms listen stats :1988 mode http stats enable stats uri / stats refresh 2s stats realm Haproxy\ Stats stats auth showme:showme {{#each frontends}} frontend {{name}} {{frontendHelper this}} {{/each}} {{#each backends}} backend {{name}} {{backendHelper this}} {{/each}} Friday, September 13, 13
55. Use of Streams • Aqueduct streams config and HAProxy stats to Crowsnest over a websocket • Crowsnest aggregates and streams to the browser • MuxDemux Friday, September 13, 13
56. registrations config stats stats config stats config control stream MuxDemux Streams Friday, September 13, 13
57. registrations config stats stats config stats Aqueduct Aqueduct config Crowsnest Server Aqueduct Thalassa Server control stream MuxDemux Streams Friday, September 13, 13
58. New to streams? https://github.com/substack/stream-adventure Friday, September 13, 13
59. When? • Modules will be officially published this month within the next several weeks • Repositories http://github.com/PearsonEducation Friday, September 13, 13
60. Questions? Friday, September 13, 13