Dynamic Infrastructure Orchestration

My talk at the 2014 Open Source Datacenter Conference about adding more dynamic to configuration management with tools like serf and etcd.

Transcript

1. freistil.it Dynamic Infrastructure Orchestration Jochen Lillich
2. Jochen Lillich @geewiz Sysadmin since 1993 CEO of freistil IT freistilbox: PaaS for Drupal & WordPress
3. Config Management Elimination of manual changes Consistent configuration Single source of truth DRY
4. Weaknesses Periodic convergence runs Central database
5. Orchestration Perform changes quickly Handle failure Tolerate network partitions
6. serf
7. Basics Cluster communication tool Developed by Hashicorp Simple binary, written in Go Gossip protocol
8. serf node 1 $ serf agent -node=node1 -bind=srv1 ==> Starting Serf agent... ==> Starting Serf agent RPC... ==> Serf agent running! Node name: 'srv1' Bind addr: '192.168.0.11:7946' … ==> Log data will now stream in as it occurs: 2014/04/09 16:05:51 [INFO] agent: Serf agent starting 2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1 192.168.0.11 2014/04/09 16:05:52 [INFO] agent: Received event: member- join
9. serf node 2 $ serf agent -node=node2 -bind=srv2 … $ serf join srv1 Successfully joined cluster by contacting 1 nodes. $ serf members node2 192.168.0.12:7946 alive node1 192.168.0.11:7946 alive
10. Membership Cluster management Service configuration
11. Failure detection Detects failure quickly Notifies other nodes Executes handler scripts
12. Events member-join member-leave member-failed member-update member-reap user query
13. User events Deploy application Perform convergence run Query information from the cluster
14. Example handler #!/bin/bash ! echo echo "New event: ${SERF_EVENT}. Data follows..." while read line; do printf "${line}\n" done
15. User event $ serf event deploy-app ! 2014/04/09 17:06:32 [INFO] agent: Received event: user-event: deploy-app
16. etcd
17. Basics HA key-value store Developed by CoreOS Written in Go Raft consensus algorithm Pure HTTP API
18. Key-value store /release = 9607bcfeb48905d26db8f /cluster /cluster/node1 = node1.example.com /cluster/node2 = node2.example.com
19. Quorum Available
20. Quorum Available
21. Quorum Available
22. Quorum Unavailable
23. Writing $ etcdctl set /message Hello Hello $ curl -L -X PUT \
 http://127.0.0.1:4001/v2/keys/message \
 -d value=“Test" {"action":"set","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15},"prevNode":{"key":"/ message","value":"Hello","modifiedIndex": 2,"createdIndex":2}}
24. Reading $ etcdctl get /message Hello $ curl -L \
 http://127.0.0.1:4001/v2/keys/message {"action":"get","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15}
25. Deleting $ etcdctl rm /message $ curl -L -X DELETE \
 http://127.0.0.1:4001/v2/keys/message
26. Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true 
 
 
 

27. Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true {"action":"set","node":{"key":"/ release","value":"9607bcfe","modifiedIndex": 16,"createdIndex":16},"prevNode":{"key":"/ release","value":"18512199","modifiedIndex": 14,"createdIndex":14}}
28. Applications Locking Leader election Database master Elastic IP
29. Leader election etcd App
30. Leader election etcd App DB?
31. Leader election etcd App DB? Nope.
32. Leader election etcd App DB? Nope. I’ll wait.
33. Leader election etcd App
34. Leader election etcd App DB X
35. Leader election etcd App DB X Master me!
36. Leader election etcd App DB X Master me! Ok!
37. Leader election etcd App DB X DB X is master.
38. Leader election etcd App DB X DB X is master.
39. Leader election etcd App DB X DB X is master.
40. Leader election etcd App DB X
41. Leader election etcd App DB X DB Y
42. Leader election etcd App DB X DB Y Master me!
43. Leader election etcd App DB X DB Y Master me! No, DB X is.
44. Leader election etcd App DB X DB Y
45. Leader election etcd App DB Y
46. Leader election etcd App DB Y
47. Leader election etcd App DB Y
48. Leader election etcd App DB Y You da master!
49. Leader election etcd App DB Y DB Y is master now.
50. Leader election etcd App DB Y
51. Summary
52. Thanks! @geewiz jochen@freistil.it