Thalassa - DenverJS - Speaker Deck

Slide 1

Slide 1 text

Thalassa A system for continuous delivery Mike Brevoort DenverJS September 12, 2013 Friday, September 13, 13

Slide 2

Slide 2 text

Mike Brevoort @mbrevoort I work at Pearson! Friday, September 13, 13

Slide 3

Slide 3 text

Background Friday, September 13, 13

Slide 4

Slide 4 text

18 months ago Friday, September 13, 13

Slide 5

Slide 5 text

Ultimatum Friday, September 13, 13

Slide 6

Slide 6 text

Continuous Deployment Friday, September 13, 13

Slide 7

Slide 7 text

• no downtime • no user impact • no connection drops • repeatable • reliable Continuos Deployment Friday, September 13, 13

Slide 8

Slide 8 text

deploy(n+1) → 0 effort Friday, September 13, 13

Slide 9

Slide 9 text

But How? Friday, September 13, 13

Slide 10

Slide 10 text

Typically handled at a load balancer Friday, September 13, 13

Slide 11

Slide 11 text

A/B style deployment • Minimize disruption • Reliable fast rollbacks • Deployments don’t cripple capacity Friday, September 13, 13

Slide 12

Slide 12 text

The Problem How do we automate the load balancer and A/B style deployments? for heterogeneous apps? (node.js, Java, etc.) in heterogenous environments (public/private cloud, bare metal) Friday, September 13, 13

Slide 13

Slide 13 text

“Necessity is the mother of taking chances” Mark Twain Friday, September 13, 13

Slide 14

Slide 14 text

Could we use Seaport and a node as a dynamic, programmable proxy? the “itch” https://github.com/substack/seaport Friday, September 13, 13

Slide 15

Slide 15 text

Why node.js? • Many connections • Streaming data • Relatively lightweight • Very programmable • Seaport was an excellent head-start Friday, September 13, 13

Slide 16

Slide 16 text

but node as the proxy? Friday, September 13, 13

Slide 17

Slide 17 text

HAProxy • Software router/load balancer • A reliable workhorse • Feature rich • Can dynamically reload conﬁguration without dropping connections Friday, September 13, 13

Slide 18

Slide 18 text

Thalassa • Registry of services • API for conﬁguring HAProxy • Dynamically set HAProxy backend servers based on registry services name and version • Fault tolerant, do no harm fka “Spindrift” Friday, September 13, 13

Slide 19

Slide 19 text

3 Major Components • Thalassa Server/Client Registry • Aqueduct • Crowsnest Friday, September 13, 13

Slide 20

Slide 20 text

Thalassa • Lightweight service registry • Formerly build on top of @substack’s Seaport • Ping and expire based approach through HTTP API • Built on Redis, axon for pub-sub • https://github.com/PearsonEducation/ thalassa Friday, September 13, 13

Slide 21

Slide 21 text

Aqueduct • REST API for HAProxy • Manage HAProxy process • Dynamically generates conﬁg • Streams HAProxy stats • Conﬁguration as a CRDT • enables peering of Aqueduct servers • https://github.com/PearsonEducation/ thalassa-aqueduct Friday, September 13, 13

Slide 22

Slide 22 text

Crowsnest • Visualize what’s happening in the system • https://github.com/PearsonEducation/ thalassa-crowsnest Friday, September 13, 13

Slide 23

Slide 23 text

Thalassa Server Friday, September 13, 13

Slide 24

Slide 24 text

Thalassa Server A@1 Friday, September 13, 13

Slide 25

Slide 25 text

Thalassa Server A@1 Friday, September 13, 13

Slide 26

Slide 26 text

Thalassa Server Aqueduct HAProxy A@1 Friday, September 13, 13

Slide 27

Slide 27 text

Thalassa Server Aqueduct HAProxy A@1 axon socket Friday, September 13, 13

Slide 28

Slide 28 text

Thalassa Server Aqueduct HAProxy A@1 80 FE axon socket Friday, September 13, 13

Slide 29

Slide 29 text

Thalassa Server Aqueduct HAProxy A@1 A@1 80 FE BE axon socket Friday, September 13, 13

Slide 30

Slide 30 text

Thalassa Server Aqueduct HAProxy A@1 A@1 80 FE BE axon socket Friday, September 13, 13

Slide 31

Slide 31 text

Thalassa Server Aqueduct HAProxy A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13

Slide 32

Slide 32 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13

Slide 33

Slide 33 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE axon socket Friday, September 13, 13

Slide 34

Slide 34 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80 FE BE streaming stats axon socket Friday, September 13, 13

Slide 35

Slide 35 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 FE BE streaming stats axon socket Friday, September 13, 13

Slide 36

Slide 36 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13

Slide 37

Slide 37 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13

Slide 38

Slide 38 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13

Slide 39

Slide 39 text

Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13

Slide 40

Slide 40 text

Demo Friday, September 13, 13

Slide 41

Slide 41 text

Friday, September 13, 13

Slide 42

Slide 42 text

Friday, September 13, 13

Slide 43

Slide 43 text

Friday, September 13, 13

Slide 44

Slide 44 text

Highlights Friday, September 13, 13

Slide 45

Slide 45 text

Monday morning August 26th Friday, September 13, 13

Slide 46

Slide 46 text

commutative data > Friday, September 13, 13

Slide 47

Slide 47 text

Friday, September 13, 13

Slide 48

Slide 48 text

Friday, September 13, 13

Slide 49

Slide 49 text

Thalassa Registry • Redis Implementation • Key/Value for each registration • Sorted set of keys ordered by time to expire Friday, September 13, 13

Slide 50

Slide 50 text

/classroom-ui/0.1.32/10.199.240.52/8080 {JSON} /classroom-ui/0.1.32/10.199.240.51/8080 {JSON} /myapp/1.0.0/10.10.10.10/8888 {JSON} /yourapp/8.0.1/192.168.8.10/80 {JSON} https://github.com/PearsonEducation/thalassa/blob/master/lib/server/RedisData.js key value Redis Registrations Friday, September 13, 13

Slide 51

Slide 51 text

/classroom-ui/0.1.32/10.199.240.52/8080 timeToExpire /classroom-ui/0.1.32/10.199.240.51/8080 /myapp/1.0.0/10.10.10.10/8888 /yourapp/8.0.1/192.168.8.10/80 timeToExpire timeToExpire timeToExpire Redis Registrations Sorted Set The Reaper Friday, September 13, 13

Slide 52

Slide 52 text

local res = redis.call('ZRANGEBYSCORE','__thalassa.registrations', 0, 1, 'LIMIT', 0, 100 ) if #res > 0 then redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 ) return res else return false end function onOnline (reg) { pub.send(reg.id, 'online', reg.stringify()); } function onOffline(regId) { pub.send(regId, 'offline'); } The Reaper Friday, September 13, 13

Slide 53

Slide 53 text

Aqueduct Conﬁg Templates • haproxy.cfg.tmpl template for haproxy.cfg • Handlebars.js • Handlebars macros https://github.com/PearsonEducation/thalassa-aqueduct/blob/master/default.haproxycfg.tmpl Friday, September 13, 13

Slide 54

Slide 54 text

global log 127.0.0.1 local0 log 127.0.0.1 local1 notice daemon maxconn 4096 user haproxy group haproxy # stats socket {{haproxySocketPath}} user USER_RUNNING_NODE_PROCESS level admin stats socket {{haproxySocketPath}} level admin defaults log global option dontlognull option redispatch retries 3 maxconn 2000 timeout connect 5000ms timeout client 50000ms timeout server 50000ms listen stats :1988 mode http stats enable stats uri / stats refresh 2s stats realm Haproxy\ Stats stats auth showme:showme {{#each frontends}} frontend {{name}} {{frontendHelper this}} {{/each}} {{#each backends}} backend {{name}} {{backendHelper this}} {{/each}} Friday, September 13, 13

Slide 55

Slide 55 text

Use of Streams • Aqueduct streams conﬁg and HAProxy stats to Crowsnest over a websocket • Crowsnest aggregates and streams to the browser • MuxDemux Friday, September 13, 13

Slide 56

Slide 56 text

registrations config stats stats config stats config control stream MuxDemux Streams Friday, September 13, 13

Slide 57

Slide 57 text

registrations config stats stats config stats Aqueduct Aqueduct config Crowsnest Server Aqueduct Thalassa Server control stream MuxDemux Streams Friday, September 13, 13

Slide 58

Slide 58 text

New to streams? https://github.com/substack/stream-adventure Friday, September 13, 13

Slide 59

Slide 59 text

When? • Modules will be ofﬁcially published this month within the next several weeks • Repositories http://github.com/PearsonEducation Friday, September 13, 13

Slide 60

Slide 60 text

Questions? Friday, September 13, 13