Share
The rsync algorithm
Modern computers are very powerful. These days, mobile phones are packed with multi core CPUs and even GPUs. Despite these advances in hardware, internet connections in most parts of the world are still surprisingly slow and unreliable. This creates a challenge: how can files be efficiently transferred between computers over a low-bandwidth, high latency network connection? In 1996, Andrew Tridgell and Paul Mackerras developed a simple solution which powers one of the most useful UNIX tools, rsync. The algorithm remains relevant 21 years after its invention because it solves a timeless problem.
This talk will take you step by step through the rsync algorithm. You will learn that PhD theses are not always scary or unapproachable. Hopefully, you will also leave with a better intuition about the inner workings of rsync, as well as ideas for how its principles can be building blocks for solving many other different problems.
Camilo Aguilar
Transcript
145 slides
-
The rsync algorithm By Andrew Tridgell and Paul Mackerras Presented
by Camilo Aguilar - @c4milo -
A B Client Server Σ Σ Σ Σ Calculating block
signatures… -
A B Client Server Σ Σ Σ Σ Σ Σ
Σ Σ Σ Σ Calculating block signatures… -
A B Client Server Σ Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Calculating block signatures… -
A B Client Server Σ Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Calculating block signatures… Sending block signatures to Client… done -
A B Client Server Σ Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Calculating block signatures… Sending block signatures to Client… done done -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server Σ C Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server Σ Lookup C Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server Σ Match! C Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server Σ Match! C Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Lookup Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Not Found! Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… Finding differences with B… -
A B Client Server C Σ Σ Σ Σ Σ
Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Reconstructing C… done Finding differences with B… done -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 strong: 1c2fb9a8 index: 2 → -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table -
Server Block Size Block Match 2 Bytes False Weak 0
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Lookup… -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table ✘ -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table ✘ -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table -
Server Block Size Block Match 2 Bytes False Weak 13d00d6
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations Lookup… -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✘ -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✘ -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes False Weak 406019b
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations Lookup… -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 0 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✓ -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✓ -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✓ -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✓ Lookup… -
Server Block Size Block Match 2 Bytes False Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations ✓ ✓ -
Server Block Size Block Match 2 Bytes True Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations -
Server Block Size Block Match 2 Bytes True Weak 12e00cb
Strong EOF False 5982e9e5 Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 0 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 0 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 0 True Lookup… -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 0 True ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 1c2fb9a8 True ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 1c2fb9a8 True ✓ Lookup… -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 1c2fb9a8 True ✓ ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 1c2fb9a8 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 13400d3 1c2fb9a8 True 2 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 0 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 0 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd 0 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd 0 True Lookup… -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd 0 True ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd ab9e1994 True ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd ab9e1994 True ✓ ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd ab9e1994 True ✓ ✓ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd ab9e1994 True -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 13900cd ab9e1994 True 3 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 0 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 0 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 13d00d6 0 Lookup… -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 True 3 13d00d6 0 ✘ -
Server Block Size Block Match 2 Bytes Weak Strong EOF
False Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 12 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 2 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 1 2 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 2 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 2 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations 2 False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations False 3 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table Operations False 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table False 13d00d6 0 C -
Server Block Size Block Match 2 Bytes Weak Strong EOF
True Client A B 12e00cb → strong: 5982e9e5 index: 1 13400d3 → strong: 1c2fb9a8 index: 2 13900cd → strong: ab9e1994 index: 3 Signature table False 13d00d6 0 -
Rolling Checksum const ( mod = 1 << 16 )
func rollingHash(block []byte) (uint32, uint32, uint32) { var a, b uint32 l := uint32(len(block)) for index, value := range block { a += uint32(value) b += (l - uint32(index)) * uint32(value) } r1 := a % mod r2 := b % mod r := r1 + (mod * r2) return r1, r2, r } -
Rolling Checksum const ( mod = 1 << 16 )
func rollingHash(block []byte) (uint32, uint32, uint32) { var a, b uint32 l := uint32(len(block)) for index, value := range block { a += uint32(value) b += (l - uint32(index)) * uint32(value) } r1 := a % mod r2 := b % mod r := r1 + (mod * r2) return r1, r2, r } -
Rolling Checksum const ( mod = 1 << 16 )
func rollingHash(block []byte) (uint32, uint32, uint32) { var a, b uint32 l := uint32(len(block)) for index, value := range block { a += uint32(value) b += (l - uint32(index)) * uint32(value) } r1 := a % mod r2 := b % mod r := r1 + (mod * r2) return r1, r2, r } -
Rolling Checksum const ( mod = 1 << 16 )
func rollingHash(block []byte) (uint32, uint32, uint32) { var a, b uint32 l := uint32(len(block)) for index, value := range block { a += uint32(value) b += (l - uint32(index)) * uint32(value) } r1 := a % mod r2 := b % mod r := r1 + (mod * r2) return r1, r2, r } -
Incremental Rolling Checksum func rollingHash2(l, r1, r2, outgoingValue, incomingValue uint32)
(uint32, uint32, uint32) { r1 = (r1 - outgoingValue + incomingValue) % mod r2 = (r2 - (l * outgoingValue) + r1) % mod r := r1 + (mod * r2) return r1, r2, r } -
Incremental Rolling Checksum func rollingHash2(l, r1, r2, outgoingValue, incomingValue uint32)
(uint32, uint32, uint32) { r1 = (r1 - outgoingValue + incomingValue) % mod r2 = (r2 - (l * outgoingValue) + r1) % mod r := r1 + (mod * r2) return r1, r2, r } -
Incremental Rolling Checksum func rollingHash2(l, r1, r2, outgoingValue, incomingValue uint32)
(uint32, uint32, uint32) { r1 = (r1 - outgoingValue + incomingValue) % mod r2 = (r2 - (l * outgoingValue) + r1) % mod r := r1 + (mod * r2) return r1, r2, r } -
Incremental Rolling Checksum func rollingHash2(l, r1, r2, outgoingValue, incomingValue uint32)
(uint32, uint32, uint32) { r1 = (r1 - outgoingValue + incomingValue) % mod r2 = (r2 - (l * outgoingValue) + r1) % mod r := r1 + (mod * r2) return r1, r2, r } -
What could be improved? • Adaptive block size based on
file size • Whole file checksum client generation and verification at the server. • Pipelining to sync multiple files in parallel over the same network connection. • Block compression -
Applications • Online data deduplication • File forensics • Incremental
backup systems • Efficient software updates … -
Acknowledgments Thanks for all the invaluable feedback on early drafts
to: • Catherine Lopez - @lopezcatherine • Elizabeth Ramírez - @eramirem • David Castillo - @castillobgr • Andrew Turley - @casio_juarez • Cameron Yick - @hydrosquall • Juan P. Osorio - @jpoo90 -
Acknowledgments For being so kind with his time answering all
my questions about his PhD thesis. Andrew Tridgell -
References • https://www.samba.org/~tridge/phd_thesis.pdf • https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf • https://speakerdeck.com/ceejbot/hash-functions-and-you • https://en.wikipedia.org/wiki/Rabin_fingerprint •
https://en.wikipedia.org/wiki/Rolling_hash • http://olstrans.sourceforge.net/release/OLS2000-rsync/OLS2000-rsync.html • http://preshing.com/20110504/hash-collision-probabilities/ -
Demo • Written in Go • Using gRPC Streaming •
LZ4 Compression • Lookup table uses dynamic arrays for chaining • Pipelining possible by using more gRPC streams on the same TCP connection, one per file. • Block size: 6kb • Strong checksum algorithm: xxhash