Topics in Distributed Systems Arnon Rotem-Gal-Oz 

What’s a “distributed system”? You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done. —LESLIE LAMPORT 

Your mission, should you choose to accept it: • Read data from one “place” • Write it to another “place” 

mov eax, [ebx] mov [ecx],eax (try (let [[partitioner msg] (channel/pull chan)] (kp/send-message @producer (kp/message topic (.getBytes ^String partitioner) (.getBytes ^String msg))) (counter-fn)) (catch Exception ex … 

System Event Actual Latency Scaled Latency One CPU cycle 0.4 ns 1 s Level 1 cache access 0.9 ns 2 s Level 2 cache access 2.8 ns 7 s Level 3 cache access 28 ns 1 min Main memory access (DDR DIMM) ~100 ns 4 min Intel® Optane™ DC persistent memory access ~350 ns 15 min Intel® Optane™ DC SSD I/O <10 μs 7 hrs NVMe SSD I/O ~25 μs 17 hrs SSD I/O 50–150 μs 1.5–4 days Rotational disk I/O 1–10 ms 1–9 months Internet call: San Francisco to New York City 65 ms 5 years Internet call: San Francisco to Hong Kong 141 ms 11 years Systems Performance: Enterprise and the Cloud, Brendan 

System Event Actual Latency Scaled Latency One CPU cycle 0.4 ns 1 s Level 1 cache access 0.9 ns 2 s Level 2 cache access 2.8 ns 7 s Level 3 cache access 28 ns 1 min Main memory access (DDR DIMM) ~100 ns 4 min Intel® Optane™ DC persistent memory access ~350 ns 15 min Intel® Optane™ DC SSD I/O <10 μs 7 hrs NVMe SSD I/O ~25 μs 17 hrs SSD I/O 50–150 μs 1.5–4 days Rotational disk I/O 1–10 ms 1–9 months Internet call: San Francisco to New York City 65 ms 5 years Internet call: San Francisco to Hong Kong 141 ms 11 years Systems Performance: Enterprise and the Cloud, Brendan 

System Event Actual Latency Scaled Latency One CPU cycle 0.4 ns 1 s Level 1 cache access 0.9 ns 2 s Level 2 cache access 2.8 ns 7 s Level 3 cache access 28 ns 1 min Main memory access (DDR DIMM) ~100 ns 4 min Intel® Optane™ DC persistent memory access ~350 ns 15 min Intel® Optane™ DC SSD I/O <10 μs 7 hrs NVMe SSD I/O ~25 μs 17 hrs SSD I/O 50–150 μs 1.5–4 days Rotational disk I/O 1–10 ms 1–9 months Internet call: San Francisco to New York City 65 ms 5 years Internet call: San Francisco to Hong Kong 141 ms 11 years Systems Performance: Enterprise and the Cloud, Brendan mov eax, [ebx] mov [ecx],eax (try (let [[partitioner msg] (cha (kp/send-message @pr message topic (.getBytes ^Str (.getBytes ^String msg))) 

Request network 

The network is reliable skb rides the rocket… 

Latency is zero 

System Event Actual Latency Scaled Latency One CPU cycle 0.4 ns 1 s Level 1 cache access 0.9 ns 2 s Level 2 cache access 2.8 ns 7 s Level 3 cache access 28 ns 1 min Main memory access (DDR DIMM) ~100 ns 4 min Intel® Optane™ DC persistent memory access ~350 ns 15 min Intel® Optane™ DC SSD I/O <10 μs 7 hrs NVMe SSD I/O ~25 μs 17 hrs SSD I/O 50–150 μs 1.5–4 days Rotational disk I/O 1–10 ms 1–9 months Internet call: San Francisco to New York City 65 ms 5 years Internet call: San Francisco to Hong Kong 141 ms 11 years Systems Performance: Enterprise and the Cloud, Brendan 

Bandwidth is infinite 

The network is secure 

Topology doesn’t change 

There is one administrator 

Transport cost is zero 

Network is homogeneous 

Instances are free 

Instances have 
 identities 

Latency is zero 

Latency is constant 

No content

What’s ”Happened Before”? 

Concurrent Q A B X Y Firewall blocks all traffic: P can’t communicate to Q P 

P Q A B P sends M Q receives M X Causal reation 

Q LogicalClockQ A P sends M 0 1 2 3 0 1 2 Q receives M B X P LogicalClockP 

Q computes: LogicalClockQ = max(0, 3) + 1 P LogicalClockP Q LogicalClockQ A P sends M 0 1 2 3 0 1 4 5 Q receives M B LogicalClockM = 3 X Y 

Counter 

Counter take 2 

Decrements? 

Sets ? 

• Don’t take distributed actions lightly • Be careful when using abstractions that hide distributed calls • Big data means low- probability problems are daily occurances 

Read more • Fallacies of distributed computing • Vector clocks • CRDTs - https://www.serverless.com/blog/crdt-explained- supercharge-serverless-at-edge • https://bartoszsypytkowski.com/the-state-of-a-state-based-crdts/ • Google Spanner https://static.googleusercontent.com/media/ research.google.com/en//archive/spanner-osdi2012.pdf • https://research.google/pubs/pub45855/