Infrastructure orchestration systems are a family of tools that allow dispatching commands against a set of remote hosts in a controlled (often ordered) fashion. MCollective, fabric, ansible etc are few of them.
In this talk we'll discuss serf and blender, as another system orchestration tool. It was born out of our need to have a similar but network tolerant tool. We maintain large, distributed clusters over the WAN which need to withstand network outages. Serf's master-less, gossip style event dispatch mechanism and ability to execute handlers upon receiving events helped us to build our own tools where serf acts as the message dispatching mechanism and we get to implement the "what to do if this event received" part. Currently we are using this to automate entire fleet wide Chef runs, periodic cassandra operations (like restores, compaction/repairs etc), and more.