$30 off During Our Annual Pro Sale. View Details »

The Raft Protocol: Distributed Consensus for Dummies

The Raft Protocol: Distributed Consensus for Dummies

An introduction to Raft and its implementation in java, Barge.

Arnaud Bailly

June 16, 2014
Tweet

More Decks by Arnaud Bailly

Other Decks in Programming

Transcript

  1. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    The Raft Protocol Distributed Consensus for
    Dummies
    Arnaud Bailly @abailly
    2014-06

    View Slide

  2. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Who am I?
    ▶ Writing code since 1986

    View Slide

  3. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Who am I?
    ▶ Writing code since 1986
    ▶ Developping software since 1994

    View Slide

  4. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Who am I?
    ▶ Writing code since 1986
    ▶ Developping software since 1994
    ▶ Lead developer, Java/XP consultant at Murex since 2009

    View Slide

  5. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Who am I?
    ▶ Writing code since 1986
    ▶ Developping software since 1994
    ▶ Lead developer, Java/XP consultant at Murex since 2009
    ▶ Fascinated with distributed computing since …

    View Slide

  6. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Who am I?
    ▶ Writing code since 1986
    ▶ Developping software since 1994
    ▶ Lead developer, Java/XP consultant at Murex since 2009
    ▶ Fascinated with distributed computing since …
    ▶ By the way, Murex is hiring!

    View Slide

  7. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)

    View Slide

  8. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!

    View Slide

  9. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often

    View Slide

  10. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance

    View Slide

  11. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably

    View Slide

  12. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably
    ▶ high-availabilty

    View Slide

  13. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably
    ▶ high-availabilty
    ▶ Consensus is a basic building block for all kind of distributed
    systems features

    View Slide

  14. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters

    View Slide

  15. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters
    ▶ Provide consistent configuration for all docker containers
    hosted on CoreOS

    View Slide

  16. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters
    ▶ Provide consistent configuration for all docker containers
    hosted on CoreOS
    ▶ Uses on Raft Distributed Consensus implemented in Go

    View Slide

  17. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store

    View Slide

  18. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution

    View Slide

  19. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:

    View Slide

  20. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts

    View Slide

  21. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK

    View Slide

  22. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster

    View Slide

  23. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster
    4. The service then can address other services using “dynamic”
    HAProxy-ed address

    View Slide

  24. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster
    4. The service then can address other services using “dynamic”
    HAProxy-ed address
    ▶ zab ensures distributed consensus across ZK nodes

    View Slide

  25. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Distributed Consensus is A Very Old Problem…

    View Slide

  26. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…

    View Slide

  27. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…

    View Slide

  28. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…

    View Slide

  29. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…

    View Slide

  30. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…

    View Slide

  31. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…

    View Slide

  32. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…
    7. You need to feed horses…

    View Slide

  33. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…
    7. You need to feed horses…
    8. Not all horses are created equal.

    View Slide

  34. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.

    View Slide

  35. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.

    View Slide

  36. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.

    View Slide

  37. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.

    View Slide

  38. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.

    View Slide

  39. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.

    View Slide

  40. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.
    7. Transport cost is zero.

    View Slide

  41. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.
    7. Transport cost is zero.
    8. The network is homogeneous.

    View Slide

  42. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Fundamental Impossibility Results
    Figure : The Fischer-Lynch-Paterson Theorem (aka. FLP)

    View Slide

  43. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In an Asynchronous Network…
    It is not possible to reach distributed consensus with
    arbitrary communication failures
    Distributed Algorithms, Nancy Lynch, 1997,
    Morkan-Kaufmann

    View Slide

  44. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In a Partially Synchronous Network…
    It is possible to reach consensus assuming f processes fail
    and there is an upper bound d on delivery time for all
    messages, provided the number of processes is greater
    than 2f
    Nancy Lynch, op.cit.

    View Slide

  45. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In Practice

    View Slide

  46. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport

    View Slide

  47. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby

    View Slide

  48. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:

    View Slide

  49. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:
    While Paxos can be described with a page of
    pseudo-code, our complete implementation contains
    several thousand lines of C++ code. Converting the
    algorithm into a practical system involved
    implementing many features some published in the
    literature and some not.

    View Slide

  50. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:
    While Paxos can be described with a page of
    pseudo-code, our complete implementation contains
    several thousand lines of C++ code. Converting the
    algorithm into a practical system involved
    implementing many features some published in the
    literature and some not.
    Paxos Made Live - An Engineering Perspective,
    T.Chandra et al.

    View Slide

  51. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013

    View Slide

  52. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind

    View Slide

  53. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind
    ▶ Dozens of implementations in various language

    View Slide

  54. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind
    ▶ Dozens of implementations in various language
    ▶ Most prominent use is currently Go version for etcd
    distributed configuration system in CoreOS

    View Slide

  55. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principle: Replicated State Machine With Persistent Log

    View Slide

  56. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster

    View Slide

  57. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster
    ▶ Each instance is a Replicated state machine whose state is
    uniquely determined by a linear persistent log

    View Slide

  58. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster
    ▶ Each instance is a Replicated state machine whose state is
    uniquely determined by a linear persistent log
    ▶ Leader orchestrates safe log replication to its followers

    View Slide

  59. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney requests being appointed leader

    View Slide

  60. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney becomes leader

    View Slide

  61. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Leader replicates own log to followers

    View Slide

  62. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney receives attack order and propagates it

    View Slide

  63. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney receives march order but is isolated

    View Slide

  64. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Lannes is appointed leader for new term

    View Slide

  65. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney comes back and tries to propagates march order

    View Slide

  66. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney fallback to follower state

    View Slide

  67. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption

    View Slide

  68. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption
    ▶ Log compaction Logs can grow very large on systems with
    high throughput, slowing down rebuild after crash and
    occupying unnecessary disk space

    View Slide

  69. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption
    ▶ Log compaction Logs can grow very large on systems with
    high throughput, slowing down rebuild after crash and
    occupying unnecessary disk space
    ▶ Snapshotting replaces history prefix with a representation of
    the state

    View Slide

  70. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly

    View Slide

  71. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP

    View Slide

  72. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP
    ▶ Feature complete w.r.t base protocol but missing cluster
    reconfiguration and log compaction

    View Slide

  73. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP
    ▶ Feature complete w.r.t base protocol but missing cluster
    reconfiguration and log compaction
    ▶ Friendly (Apache 2.0) License, Pull Requests are welcomed

    View Slide

  74. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Demo

    View Slide

  75. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements

    View Slide

  76. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus

    View Slide

  77. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level

    View Slide

  78. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable

    View Slide

  79. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet

    View Slide

  80. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet
    ▶ Strong Consistency has a cost you don’t want to pay for high
    throughput and large data sets

    View Slide

  81. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet
    ▶ Strong Consistency has a cost you don’t want to pay for high
    throughput and large data sets
    ▶ Sweet spot: Configuration data, synchronizing clients at key
    points

    View Slide

  82. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    TINSTAAFL

    View Slide

  83. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Questions?

    View Slide

  84. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems

    View Slide

  85. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems
    ▶ Napoléon à Austerlitz

    View Slide

  86. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems
    ▶ Napoléon à Austerlitz
    ▶ Nancy Lynch at CSAIL

    View Slide