Tweet
Tweet

Slide 1

Slide 1 text

Verifying a Distributed System with Combinatorial Topology Verónica López Sr. Software Engineer @maria_fibonacci CodeMesh 2018

Slide 2

Slide 2 text

Verifying a Distributed System with Combinatorial Topology Verónica López Sr. Software Engineer @maria_fibonacci CodeMesh 2018

Slide 3

Slide 3 text

- Academy & Industry: From Physics to Distributed Systems - Software Engineer: Go & Kubernetes, Containers, Linux - Personal preference: Elixir (BEAM) - Before: Big Latin American systems: many constraints - Technology as a means of social progress whoami

Slide 4

Slide 4 text

Agenda - Distributed Systems - Graph Theory - Topology

Slide 5

Slide 5 text

Topology: the math term, not the (pretentious) engineer term for any systems design diagram

Slide 6

Slide 6 text

All these concepts have connectivity in common

Slide 7

Slide 7 text

Distributed Systems

Slide 8

Slide 8 text

Famous -and overused- quote about distsys...

Slide 9

Slide 9 text

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer* unusable” Leslie Lamport

Slide 10

Slide 10 text

Ideal Distributed System - Fault Tolerant - Highly available - Recoverable - Consistent - Scalable - (Predictable) Performance - Secure

Slide 11

Slide 11 text

Design for Failure

Slide 12

Slide 12 text

If the probability of something happening is one in 10^13, how often will it really happen? “Real life”: never Physics: all the time Think about servers (infrastructure) at scale Or in terms of downtime

Slide 13

Slide 13 text

Verification of a Distributed System

Slide 14

Slide 14 text

Hard Problem: - Have control and visibility over all the interconnections of our systems - Solutions: Monitoring, Chaos Engineering, On-Call rotations, Testing in Production, etc. Formal Verification - Formal specification languages & model checkers - Still requires the definition of the program, possible failures, correctness definitions

Slide 15

Slide 15 text

What if we had something that allowed us to see all these possibilities at once

Slide 16

Slide 16 text

Graph Theory

Slide 17

Slide 17 text

- The mathematical structures used to model pairwise relations between objects. - Seven Bridges of Könisberg (1736, Euler) is the first paper in history of graph theory - K-connectedness: how many nodes we need to disconnect a graph (a system) - Verify points of failure

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Describing the adjacencies (interactions) of distributed systems gets messier with graphs

Slide 21

Slide 21 text

Topology

Slide 22

Slide 22 text

The study of geometric properties and spatial relations unaffected by the continuous change of shape or size of figures.

Slide 23

Slide 23 text

The paper on the Seven Bridges of Königsberg is also considered the first paper in history of Topology

Slide 24

Slide 24 text

Properties remain invariant under continuous stretching and bending of the object (different partitions)

Slide 25

Slide 25 text

Herlihy, Maurice, et al. Distributed Computing through Combinatorial Topology. Morgan Kaufmann, 2014.

Slide 26

Slide 26 text

A topologist is a person who cannot tell the difference between a coffee mug and a donut

Slide 27

Slide 27 text

A topologist is a person who cannot tell the difference between a coffee mug and a donut

Slide 28

Slide 28 text

Combinatorial (Algebraic) Topology - Studies spaces that can be constructed with discretized spaces - Allows to have all the (system) perspectives (of a node) available at the same time - Perspectives evolve with communication - Perspective = the view from a single node

Slide 29

Slide 29 text

Combinatorial (Algebraic) Topology - Branches of topology differ in the way they represent spaces and in the continuous transformations that preserve properties. - Spaces made up of simple pieces for which essential properties can be characterized by counting, such as the sum of the degrees of the nodes in a graph. - Countable items allow combinations (interactions)

Slide 30

Slide 30 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 31

Slide 31 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 32

Slide 32 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 33

Slide 33 text

Subdivisions - Not every continuous map A->B has a simplicial approximation. Herlihy, Maurice, et al. Distributed Computing through Combinatorial Topology. Morgan Kaufmann, 2014.

Slide 34

Slide 34 text

Verifying a Distributed System with Combinatorial Topology

Slide 35

Slide 35 text

Thesis Distributed systems can be formally verified by treating them as (a set of) topological entities that are subject to (valid) subdivisions, analysis of the persistence and consistency of their interconnections (paths), offering a comprehensive set of states of the world

Slide 36

Slide 36 text

Step 1 If your system can be described as a graph, it can also be described as a topological object (if the connections are preserved) Theorem: A topology on V is compatible with a graph G(V,E) if every induced subgraph of G is connected if and only if its vertex set is topologically connected (too).

Slide 37

Slide 37 text

Step 2 Describe our systems as a topological object: Every node is an elemen of our system: compute server, cluster, etc.

Slide 38

Slide 38 text

Step 3 Prove connectivity -> Verifying the system Analyze the connections and interactions (in terms of formal Connectivity) Get all the possible states of the world (use cases; paths) Once all the connections are topologically correct, we can say that the system is verified.

Slide 39

Slide 39 text

Resources 1. Algebraic topology and distributed computing a primer https://link.springer.com/chapter/10.1007%2FBFb0015245 2. The Topology of shared-memory adversaries https://dl.acm.org/citation.cfm?doid=1835698.1835724 3. Distributed Computing Through Combinatorial Topology https://www.elsevier.com/books/distributed-computing-through-combinatorial-topolo gy/herlihy/978-0-12-404578-1

Slide 40

Slide 40 text

Thank you!

Slide 41

Slide 41 text

- Academy & Industry: From Physics to Distributed Systems - Software Engineer: Go & Kubernetes, Containers, Linux - Personal preference: Elixir (BEAM) - Before: Big Latin American systems: many constraints - Technology as a means of social progress whoami

Slide 42

Slide 42 text

Agenda - Distributed Systems - Graph Theory - Topology

Slide 43

Slide 43 text

Topology: the math term, not the (pretentious) engineer term for any systems design diagram

Slide 44

Slide 44 text

All these concepts have connectivity in common

Slide 45

Slide 45 text

Distributed Systems

Slide 46

Slide 46 text

Famous -and overused- quote about distsys...

Slide 47

Slide 47 text

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer* unusable” Leslie Lamport

Slide 48

Slide 48 text

Ideal Distributed System - Fault Tolerant - Highly available - Recoverable - Consistent - Scalable - (Predictable) Performance - Secure

Slide 49

Slide 49 text

Design for Failure

Slide 50

Slide 50 text

If the probability of something happening is one in 10^13, how often will it really happen? “Real life”: never Physics: all the time Think about servers (infrastructure) at scale Or in terms of downtime

Slide 51

Slide 51 text

Verification of a Distributed System

Slide 52

Slide 52 text

Hard Problem: - Have control and visibility over all the interconnections of our systems - Solutions: Monitoring, Chaos Engineering, On-Call rotations, Testing in Production, etc. Formal Verification - Formal specification languages & model checkers - Still requires the definition of the program, possible failures, correctness definitions

Slide 53

Slide 53 text

What if we had something that allowed us to see all these possibilities at once

Slide 54

Slide 54 text

Graph Theory

Slide 55

Slide 55 text

- The mathematical structures used to model pairwise relations between objects. - Seven Bridges of Könisberg (1736, Euler) is the first paper in history of graph theory - K-connectedness: how many nodes we need to disconnect a graph (a system) - Verify points of failure

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Describing the adjacencies (interactions) of distributed systems gets messier with graphs

Slide 59

Slide 59 text

Topology

Slide 60

Slide 60 text

The study of geometric properties and spatial relations unaffected by the continuous change of shape or size of figures.

Slide 61

Slide 61 text

The paper on the Seven Bridges of Königsberg is also considered the first paper in history of Topology

Slide 62

Slide 62 text

Properties remain invariant under continuous stretching and bending of the object (different partitions)

Slide 63

Slide 63 text

Herlihy, Maurice, et al. Distributed Computing through Combinatorial Topology. Morgan Kaufmann, 2014.

Slide 64

Slide 64 text

A topologist is a person who cannot tell the difference between a coffee mug and a donut

Slide 65

Slide 65 text

A topologist is a person who cannot tell the difference between a coffee mug and a donut

Slide 66

Slide 66 text

Combinatorial (Algebraic) Topology - Studies spaces that can be constructed with discretized spaces - Allows to have all the (system) perspectives (of a node) available at the same time - Perspectives evolve with communication - Perspective = the view from a single node

Slide 67

Slide 67 text

Combinatorial (Algebraic) Topology - Branches of topology differ in the way they represent spaces and in the continuous transformations that preserve properties. - Spaces made up of simple pieces for which essential properties can be characterized by counting, such as the sum of the degrees of the nodes in a graph. - Countable items allow combinations (interactions)

Slide 68

Slide 68 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 69

Slide 69 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 70

Slide 70 text

Views: each set of interactions has its own perspective of the system. Views can be later put together to describe the system.

Slide 71

Slide 71 text

Subdivisions - Not every continuous map A->B has a simplicial approximation. Herlihy, Maurice, et al. Distributed Computing through Combinatorial Topology. Morgan Kaufmann, 2014.

Slide 72

Slide 72 text

Verifying a Distributed System with Combinatorial Topology

Slide 73

Slide 73 text

Thesis Distributed systems can be formally verified by treating them as (a set of) topological entities that are subject to (valid) subdivisions, analysis of the persistence and consistency of their interconnections (paths), offering a comprehensive set of states of the world

Slide 74

Slide 74 text

Step 1 If your system can be described as a graph, it can also be described as a topological object (if the connections are preserved) Theorem: A topology on V is compatible with a graph G(V,E) if every induced subgraph of G is connected if and only if its vertex set is topologically connected (too).

Slide 75

Slide 75 text

Step 2 Describe our systems as a topological object: Every node is an elemen of our system: compute server, cluster, etc.

Slide 76

Slide 76 text

Step 3 Prove connectivity -> Verifying the system Analyze the connections and interactions (in terms of formal Connectivity) Get all the possible states of the world (use cases; paths) Once all the connections are topologically correct, we can say that the system is verified.

Slide 77

Slide 77 text

Resources 1. Algebraic topology and distributed computing a primer https://link.springer.com/chapter/10.1007%2FBFb0015245 2. The Topology of shared-memory adversaries https://dl.acm.org/citation.cfm?doid=1835698.1835724 3. Distributed Computing Through Combinatorial Topology https://www.elsevier.com/books/distributed-computing-through-combinatorial-topolo gy/herlihy/978-0-12-404578-1

Slide 78

Slide 78 text

Thank you!