The Raft Protocol: Distributed Consensus for Dummies

An introduction to Raft and its implementation in java, Barge.

Transcript

1. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   The Raft Protocol Distributed Consensus for
   Dummies
   Arnaud Bailly @abailly
   2014-06
   

   View Slide

2. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Who am I?
   ▶ Writing code since 1986
   

   View Slide

3. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Who am I?
   ▶ Writing code since 1986
   ▶ Developping software since 1994
   

   View Slide

4. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Who am I?
   ▶ Writing code since 1986
   ▶ Developping software since 1994
   ▶ Lead developer, Java/XP consultant at Murex since 2009
   

   View Slide

5. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Who am I?
   ▶ Writing code since 1986
   ▶ Developping software since 1994
   ▶ Lead developer, Java/XP consultant at Murex since 2009
   ▶ Fascinated with distributed computing since …
   

   View Slide

6. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Who am I?
   ▶ Writing code since 1986
   ▶ Developping software since 1994
   ▶ Lead developer, Java/XP consultant at Murex since 2009
   ▶ Fascinated with distributed computing since …
   ▶ By the way, Murex is hiring!
   

   View Slide

7. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Why Should I Care about Distributed Consensus?
   ▶ Real world is distributed (multicore chips, WWW)
   

   View Slide

8. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Why Should I Care about Distributed Consensus?
   ▶ Real world is distributed (multicore chips, WWW)
   ▶ Today’s applications need to take care of distribution:
   abstractions leak!
   

   View Slide

9. .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   .
   .
   ..
   .
   Why Should I Care about Distributed Consensus?
   ▶ Real world is distributed (multicore chips, WWW)
   ▶ Today’s applications need to take care of distribution:
   abstractions leak!
   ▶ Systems may fail, and large systems may fail more often
   

   View Slide

10. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    

    View Slide

11. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably
    

    View Slide

12. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably
    ▶ high-availabilty
    

    View Slide

13. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Why Should I Care about Distributed Consensus?
    ▶ Real world is distributed (multicore chips, WWW)
    ▶ Today’s applications need to take care of distribution:
    abstractions leak!
    ▶ Systems may fail, and large systems may fail more often
    ▶ fault-tolerance
    ▶ Yet we need to provide fast service reliably
    ▶ high-availabilty
    ▶ Consensus is a basic building block for all kind of distributed
    systems features
    

    View Slide

14. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters
    

    View Slide

15. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters
    ▶ Provide consistent configuration for all docker containers
    hosted on CoreOS
    

    View Slide

16. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: PaaS Configuration
    ▶ etcd is part of CoreOS, a linux distribution for clusters
    ▶ Provide consistent configuration for all docker containers
    hosted on CoreOS
    ▶ Uses on Raft Distributed Consensus implemented in Go
    

    View Slide

17. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    

    View Slide

18. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    

    View Slide

19. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    

    View Slide

20. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    

    View Slide

21. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    

    View Slide

22. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster
    

    View Slide

23. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster
    4. The service then can address other services using “dynamic”
    HAProxy-ed address
    

    View Slide

24. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Use Case: Service Discovery
    ▶ Apache’s ZooKeeper provides distributed consistent
    hierarchical key-value store
    ▶ AirBnB uses ZK to provide service discovery in their
    SmartStack solution
    ▶ Example scenario:
    1. A room registration service instance starts
    2. It registers itself as an ephemeral node in ZK
    3. This triggers reconfiguration of HAProxy to this service in the
    cluster
    4. The service then can address other services using “dynamic”
    HAProxy-ed address
    ▶ zab ensures distributed consensus across ZK nodes
    

    View Slide

25. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Distributed Consensus is A Very Old Problem…
    

    View Slide

26. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    

    View Slide

27. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    

    View Slide

28. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    

    View Slide

29. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    

    View Slide

30. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    

    View Slide

31. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…
    

    View Slide

32. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…
    7. You need to feed horses…
    

    View Slide

33. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … And it is Hard
    1. Horses and messengers can get killed…
    2. Horses can travel only so fast…
    3. You can send only so many horses at once…
    4. Enemy can setup ambushes…
    5. Army corps can move…
    6. Nobody knows everything…
    7. You need to feed horses…
    8. Not all horses are created equal.
    

    View Slide

34. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    

    View Slide

35. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    

    View Slide

36. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    

    View Slide

37. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    

    View Slide

38. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    

    View Slide

39. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.
    

    View Slide

40. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.
    7. Transport cost is zero.
    

    View Slide

41. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    … Even in Distributed Computing
    The 8 Fallacies of Distributed Computing
    1. The network is reliable.
    2. Latency is zero.
    3. Bandwidth is infinite.
    4. The network is secure.
    5. Topology doesn’t change.
    6. There is one administrator.
    7. Transport cost is zero.
    8. The network is homogeneous.
    

    View Slide

42. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Fundamental Impossibility Results
    Figure : The Fischer-Lynch-Paterson Theorem (aka. FLP)
    

    View Slide

43. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In an Asynchronous Network…
    It is not possible to reach distributed consensus with
    arbitrary communication failures
    Distributed Algorithms, Nancy Lynch, 1997,
    Morkan-Kaufmann
    

    View Slide

44. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In a Partially Synchronous Network…
    It is possible to reach consensus assuming f processes fail
    and there is an upper bound d on delivery time for all
    messages, provided the number of processes is greater
    than 2f
    Nancy Lynch, op.cit.
    

    View Slide

45. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    In Practice
    

    View Slide

46. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    

    View Slide

47. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    

    View Slide

48. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:
    

    View Slide

49. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:
    While Paxos can be described with a page of
    pseudo-code, our complete implementation contains
    several thousand lines of C++ code. Converting the
    algorithm into a practical system involved
    implementing many features some published in the
    literature and some not.
    

    View Slide

50. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Paxos
    ▶ Renowned consensus algorithm invented by Leslie Lamport
    ▶ Provides foundations for several implementations: ZooKeeper
    (kinda…), Chubby
    ▶ But it is hard to implement correctly:
    While Paxos can be described with a page of
    pseudo-code, our complete implementation contains
    several thousand lines of C++ code. Converting the
    algorithm into a practical system involved
    implementing many features some published in the
    literature and some not.
    Paxos Made Live - An Engineering Perspective,
    T.Chandra et al.
    

    View Slide

51. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    

    View Slide

52. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind
    

    View Slide

53. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind
    ▶ Dozens of implementations in various language
    

    View Slide

54. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft
    ▶ In Search of an Understandable Consensus Algorithm,
    D.Ongaro and J.Osterhout, 2013
    ▶ Novel algorithm designed with understandability in mind
    ▶ Dozens of implementations in various language
    ▶ Most prominent use is currently Go version for etcd
    distributed configuration system in CoreOS
    

    View Slide

55. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principle: Replicated State Machine With Persistent Log
    

    View Slide

56. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster
    

    View Slide

57. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster
    ▶ Each instance is a Replicated state machine whose state is
    uniquely determined by a linear persistent log
    

    View Slide

58. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Principles of Operation
    ▶ Leader-follower based algorithm: Leader is the single entry
    point for all operations on the cluster
    ▶ Each instance is a Replicated state machine whose state is
    uniquely determined by a linear persistent log
    ▶ Leader orchestrates safe log replication to its followers
    

    View Slide

59. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney requests being appointed leader
    

    View Slide

60. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney becomes leader
    

    View Slide

61. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Leader replicates own log to followers
    

    View Slide

62. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney receives attack order and propagates it
    

    View Slide

63. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney receives march order but is isolated
    

    View Slide

64. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Lannes is appointed leader for new term
    

    View Slide

65. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney comes back and tries to propagates march order
    

    View Slide

66. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Raft Algorithm
    Figure : Ney fallback to follower state
    

    View Slide

67. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption
    

    View Slide

68. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption
    ▶ Log compaction Logs can grow very large on systems with
    high throughput, slowing down rebuild after crash and
    occupying unnecessary disk space
    

    View Slide

69. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Other Features
    ▶ Cluster Reconfiguration Supports cluster membership
    changes w/o service interruption
    ▶ Log compaction Logs can grow very large on systems with
    high throughput, slowing down rebuild after crash and
    occupying unnecessary disk space
    ▶ Snapshotting replaces history prefix with a representation of
    the state
    

    View Slide

70. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    

    View Slide

71. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP
    

    View Slide

72. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP
    ▶ Feature complete w.r.t base protocol but missing cluster
    reconfiguration and log compaction
    

    View Slide

73. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Java Implementation: Barge
    https://github.com/mgodave/barge !
    ▶ OSS project started by Dave Rusek with contributions from
    Justin Santa Barbara and yours truly
    ▶ Still very young but usable, provides 2 transport methods:
    Raw TCP and HTTP
    ▶ Feature complete w.r.t base protocol but missing cluster
    reconfiguration and log compaction
    ▶ Friendly (Apache 2.0) License, Pull Requests are welcomed
    

    View Slide

74. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Demo
    

    View Slide

75. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    

    View Slide

76. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    

    View Slide

77. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    

    View Slide

78. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    

    View Slide

79. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet
    

    View Slide

80. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet
    ▶ Strong Consistency has a cost you don’t want to pay for high
    throughput and large data sets
    

    View Slide

81. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Takeaways
    ▶ Understand your consistency requirements
    ▶ Strong consistency Consensus
    ▶ Lowered barrier of entry to use consensus at applicative level
    ▶ Raft is lightweight and understandable
    ▶ Not a Silver Bullet
    ▶ Strong Consistency has a cost you don’t want to pay for high
    throughput and large data sets
    ▶ Sweet spot: Configuration data, synchronizing clients at key
    points
    

    View Slide

82. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    TINSTAAFL
    

    View Slide

83. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Questions?
    

    View Slide

84. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems
    

    View Slide

85. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems
    ▶ Napoléon à Austerlitz
    

    View Slide

86. .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    .
    .
    ..
    .
    Credits & Links
    ▶ ETH Zurich Course on Distributed Systems
    ▶ Napoléon à Austerlitz
    ▶ Nancy Lynch at CSAIL
    

    View Slide