Dynamic Infrastructure Orchestration

Slide 1

Slide 1 text

freistil.it Dynamic Infrastructure Orchestration Jochen Lillich

Slide 2

Slide 2 text

Jochen Lillich @geewiz Sysadmin since 1993 CEO of freistil IT freistilbox: PaaS for Drupal & WordPress

Slide 3

Slide 3 text

Conﬁg Management Elimination of manual changes Consistent conﬁguration Single source of truth DRY

Slide 4

Slide 4 text

Weaknesses Periodic convergence runs Central database

Slide 5

Slide 5 text

Orchestration Perform changes quickly Handle failure Tolerate network partitions

Slide 6

Slide 6 text

serf

Slide 7

Slide 7 text

Basics Cluster communication tool Developed by Hashicorp Simple binary, written in Go Gossip protocol

Slide 8

Slide 8 text

serf node 1 $ serf agent -node=node1 -bind=srv1 ==> Starting Serf agent... ==> Starting Serf agent RPC... ==> Serf agent running! Node name: 'srv1' Bind addr: '192.168.0.11:7946' … ==> Log data will now stream in as it occurs: 2014/04/09 16:05:51 [INFO] agent: Serf agent starting 2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1 192.168.0.11 2014/04/09 16:05:52 [INFO] agent: Received event: member- join

Slide 9

Slide 9 text

serf node 2 $ serf agent -node=node2 -bind=srv2 … $ serf join srv1 Successfully joined cluster by contacting 1 nodes. $ serf members node2 192.168.0.12:7946 alive node1 192.168.0.11:7946 alive

Slide 10

Slide 10 text

Membership Cluster management Service conﬁguration

Slide 11

Slide 11 text

Failure detection Detects failure quickly Notiﬁes other nodes Executes handler scripts

Slide 12

Slide 12 text

Events member-join member-leave member-failed member-update member-reap user query

Slide 13

Slide 13 text

User events Deploy application Perform convergence run Query information from the cluster

Slide 14

Slide 14 text

Example handler #!/bin/bash ! echo echo "New event: ${SERF_EVENT}. Data follows..." while read line; do printf "${line}\n" done

Slide 15

Slide 15 text

User event $ serf event deploy-app ! 2014/04/09 17:06:32 [INFO] agent: Received event: user-event: deploy-app

Slide 16

Slide 16 text

etcd

Slide 17

Slide 17 text

Basics HA key-value store Developed by CoreOS Written in Go Raft consensus algorithm Pure HTTP API

Slide 18

Slide 18 text

Key-value store /release = 9607bcfeb48905d26db8f /cluster /cluster/node1 = node1.example.com /cluster/node2 = node2.example.com

Slide 19

Slide 19 text

Quorum Available

Slide 20

Slide 20 text

Quorum Available

Slide 21

Slide 21 text

Quorum Available

Slide 22

Slide 22 text

Quorum Unavailable

Slide 23

Slide 23 text

Writing $ etcdctl set /message Hello Hello $ curl -L -X PUT \  http://127.0.0.1:4001/v2/keys/message \  -d value=“Test" {"action":"set","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15},"prevNode":{"key":"/ message","value":"Hello","modifiedIndex": 2,"createdIndex":2}}

Slide 24

Slide 24 text

Reading $ etcdctl get /message Hello $ curl -L \  http://127.0.0.1:4001/v2/keys/message {"action":"get","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15}

Slide 25

Slide 25 text

Deleting $ etcdctl rm /message $ curl -L -X DELETE \  http://127.0.0.1:4001/v2/keys/message

Slide 26

Slide 26 text

Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true        

Slide 27

Slide 27 text

Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true {"action":"set","node":{"key":"/ release","value":"9607bcfe","modifiedIndex": 16,"createdIndex":16},"prevNode":{"key":"/ release","value":"18512199","modifiedIndex": 14,"createdIndex":14}}

Slide 28

Slide 28 text

Applications Locking Leader election Database master Elastic IP

Slide 29

Slide 29 text

Leader election etcd App

Slide 30

Slide 30 text

Leader election etcd App DB?

Slide 31

Slide 31 text

Leader election etcd App DB? Nope.

Slide 32

Slide 32 text

Leader election etcd App DB? Nope. I’ll wait.

Slide 33

Slide 33 text

Leader election etcd App

Slide 34

Slide 34 text

Leader election etcd App DB X

Slide 35

Slide 35 text

Leader election etcd App DB X Master me!

Slide 36

Slide 36 text

Leader election etcd App DB X Master me! Ok!

Slide 37

Slide 37 text

Leader election etcd App DB X DB X is master.

Slide 38

Slide 38 text

Leader election etcd App DB X DB X is master.

Slide 39

Slide 39 text

Leader election etcd App DB X DB X is master.

Slide 40

Slide 40 text

Leader election etcd App DB X

Slide 41

Slide 41 text

Leader election etcd App DB X DB Y

Slide 42

Slide 42 text

Leader election etcd App DB X DB Y Master me!

Slide 43

Slide 43 text

Leader election etcd App DB X DB Y Master me! No, DB X is.

Slide 44

Slide 44 text

Leader election etcd App DB X DB Y

Slide 45

Slide 45 text

Leader election etcd App DB Y

Slide 46

Slide 46 text

Leader election etcd App DB Y

Slide 47

Slide 47 text

Leader election etcd App DB Y

Slide 48

Slide 48 text

Leader election etcd App DB Y You da master!

Slide 49

Slide 49 text

Leader election etcd App DB Y DB Y is master now.

Slide 50

Slide 50 text

Leader election etcd App DB Y

Slide 51

Slide 51 text

Summary

Slide 52

Slide 52 text

Thanks! @geewiz [email protected]