Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dynamic Infrastructure Orchestration

Dynamic Infrastructure Orchestration

My talk at the 2014 Open Source Datacenter Conference about adding more dynamic to configuration management with tools like serf and etcd.

3abfc9128fe1aeae3058edc36936f360?s=128

Jochen Lillich
PRO

April 10, 2014
Tweet

Transcript

  1. freistil.it Dynamic Infrastructure Orchestration Jochen Lillich

  2. Jochen Lillich @geewiz Sysadmin since 1993 CEO of freistil IT

    freistilbox: PaaS for Drupal & WordPress
  3. Config Management Elimination of manual changes Consistent configuration Single source

    of truth DRY
  4. Weaknesses Periodic convergence runs Central database

  5. Orchestration Perform changes quickly Handle failure Tolerate network partitions

  6. serf

  7. Basics Cluster communication tool Developed by Hashicorp Simple binary, written

    in Go Gossip protocol
  8. serf node 1 $ serf agent -node=node1 -bind=srv1 ==> Starting

    Serf agent... ==> Starting Serf agent RPC... ==> Serf agent running! Node name: 'srv1' Bind addr: '192.168.0.11:7946' … ==> Log data will now stream in as it occurs: 2014/04/09 16:05:51 [INFO] agent: Serf agent starting 2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1 192.168.0.11 2014/04/09 16:05:52 [INFO] agent: Received event: member- join
  9. serf node 2 $ serf agent -node=node2 -bind=srv2 … $

    serf join srv1 Successfully joined cluster by contacting 1 nodes. $ serf members node2 192.168.0.12:7946 alive node1 192.168.0.11:7946 alive
  10. Membership Cluster management Service configuration

  11. Failure detection Detects failure quickly Notifies other nodes Executes handler

    scripts
  12. Events member-join member-leave member-failed member-update member-reap user query

  13. User events Deploy application Perform convergence run Query information from

    the cluster
  14. Example handler #!/bin/bash ! echo echo "New event: ${SERF_EVENT}. Data

    follows..." while read line; do printf "${line}\n" done
  15. User event $ serf event deploy-app ! 2014/04/09 17:06:32 [INFO]

    agent: Received event: user-event: deploy-app
  16. etcd

  17. Basics HA key-value store Developed by CoreOS Written in Go

    Raft consensus algorithm Pure HTTP API
  18. Key-value store /release = 9607bcfeb48905d26db8f /cluster /cluster/node1 = node1.example.com /cluster/node2

    = node2.example.com
  19. Quorum Available

  20. Quorum Available

  21. Quorum Available

  22. Quorum Unavailable

  23. Writing $ etcdctl set /message Hello Hello $ curl -L

    -X PUT \
 http://127.0.0.1:4001/v2/keys/message \
 -d value=“Test" {"action":"set","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15},"prevNode":{"key":"/ message","value":"Hello","modifiedIndex": 2,"createdIndex":2}}
  24. Reading $ etcdctl get /message Hello $ curl -L \


    http://127.0.0.1:4001/v2/keys/message {"action":"get","node":{"key":"/ message","value":"Test","modifiedIndex": 15,"createdIndex":15}
  25. Deleting $ etcdctl rm /message $ curl -L -X DELETE

    \
 http://127.0.0.1:4001/v2/keys/message
  26. Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true 
 
 
 


  27. Waiting $ curl -L http://127.0.0.1:4001/v2/keys/ release\?wait\=true {"action":"set","node":{"key":"/ release","value":"9607bcfe","modifiedIndex": 16,"createdIndex":16},"prevNode":{"key":"/ release","value":"18512199","modifiedIndex":

    14,"createdIndex":14}}
  28. Applications Locking Leader election Database master Elastic IP

  29. Leader election etcd App

  30. Leader election etcd App DB?

  31. Leader election etcd App DB? Nope.

  32. Leader election etcd App DB? Nope. I’ll wait.

  33. Leader election etcd App

  34. Leader election etcd App DB X

  35. Leader election etcd App DB X Master me!

  36. Leader election etcd App DB X Master me! Ok!

  37. Leader election etcd App DB X DB X is master.

  38. Leader election etcd App DB X DB X is master.

  39. Leader election etcd App DB X DB X is master.

  40. Leader election etcd App DB X

  41. Leader election etcd App DB X DB Y

  42. Leader election etcd App DB X DB Y Master me!

  43. Leader election etcd App DB X DB Y Master me!

    No, DB X is.
  44. Leader election etcd App DB X DB Y

  45. Leader election etcd App DB Y

  46. Leader election etcd App DB Y

  47. Leader election etcd App DB Y

  48. Leader election etcd App DB Y You da master!

  49. Leader election etcd App DB Y DB Y is master

    now.
  50. Leader election etcd App DB Y

  51. Summary

  52. Thanks! @geewiz jochen@freistil.it