Diving into Merkle Trees

Introduced by Ralph C. Merkle, the Merkle Tree is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. Due to its unique content validating and performance qualities, they are especially useful in distributed, peer-to-peer systems where the same data should exist in multiple places. Let's learn how to detect inconsistencies between trees and reduce the amount of transferred data enabling peer-to-peer file sharing by building our own tree with Elixir.

Transcript

1. diving into @ordepdev Merkle Trees
2. @ordepdev @ordepdev
3. @ordepdev
4. @ordepdev
5. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf 1987
6. “[…] can sign an unlimited number of messages, and the signature size increases logarithmically.”
7. “The general idea in this new system is to use an infinite tree of one-time signatures.”
8. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf https://patents.google.com/patent/US4309569 http://www.merkle.com/papers/Thesis1979.pdf 1979
10. 1989 http://merkle.com/papers/Certified1979.pdf
11. “[…] is a data structure used for efficiently summarizing and verifying the integrity of large sets of data.”
12. “[…] it was used for the purpose of one-time signatures and authenticated public key distribution.”
14. Summarize and Verify
15. One-way Functions
16. “A one-way function f is a function that is easy to compute but difficult to invert.”
17. () () () 4a23ca438f3 065b90acc7 ccca99192c
18. Why do we need them?
19. “[…] being able to store and identify data with a fixed length output can create vast storage savings.”
20. “The person who computes y=F(x) is the only person who knows x.”
21. One-time Signatures
22. “One time signatures are practical between a single pair of users who are willing to exchange the large amount of data necessary […]”
23. “If 1000 messages are to be signed before new public authentication data is needed, over 20,000,000 bits or 2.5 megabytes must be stored as public information.”
24. Improving one- time signatures
25. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf http://www.merkle.com/papers/Thesis1979.pdf
26. How to compute a Merkle Root?
27. 9ec4 L1 L2
28. h(L1)
29. 9ec4 L1 L2
30. h(L2)
31. a871 L1 L2 7e6a 9ec4
32. h(h(L1)||h(L2))
33. aea9 a871 7e6a L1 L2 9ec4
35. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Leaf
36. defmodule MerkleTree.Leaf do defstruct [:hash, :value] @type hash :: String.t @type value :: String.t @type t :: %MerkleTree.Leaf{ hash: hash, value: value } end
37. defmodule MerkleTree.Leaf do defstruct [:hash, :value] @type hash :: String.t @type value :: String.t @type t :: %MerkleTree.Leaf{ hash: hash, value: value } end
38. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Node
39. defmodule MerkleTree.Node do defstruct [:hash, :left, :right] @type hash :: String.t @type left :: MerkleTree.Node.t | MerkleTree.Leaf.t @type right :: MerkleTree.Node.t | MerkleTree.Leaf.t @type t :: %MerkleTree.Node{ hash: hash, left: left, right: right } end
40. defmodule MerkleTree.Node do defstruct [:hash, :left, :right] @type hash :: String.t @type left :: MerkleTree.Node.t | MerkleTree.Leaf.t @type right :: MerkleTree.Node.t | MerkleTree.Leaf.t @type t :: %MerkleTree.Node{ hash: hash, left: left, right: right } end
41. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Leaf MerkleTree.Node MerkleTree[:root]
42. defmodule MerkleTree do defstruct [:root] @type root :: MerkleTree.Node.t @type t :: %MerkleTree{ root: root } end
43. defmodule MerkleTree do defstruct [:root] @type root :: MerkleTree.Node.t @type t :: %MerkleTree{ root: root } end
44. Hashing the data blocks
45. defmodule MerkleTree.Crypto do def hash(input, type \\ :sha256) do type |> :crypto.hash("#{input}") |> Base.encode16 end end
46. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
47. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
48. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
49. defmodule MerkleTree.Leaf do def build(value, hash) do %Leaf{value: value, hash: hash} end end
50. [“L1”, “L2”] |> MerkleTree.new
51. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
52. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
53. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
54. Hashing the nodes
55. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |> Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
56. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |> Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
57. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |> Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
58. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |> Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
59. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |> Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
60. defmodule MerkleTree.Node do def build(hash, [left, right]) do %Node{hash: hash, left: left, right: right} end end
61. MerkleTree.Node.new([ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ])
62. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
63. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
64. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
65. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
66. Building a bigger tree
67. 9ec4 L1 L2 L3 L4
68. 9ec4 L1 L2 L3 L4
69. 9ec4 L1 L2 L3 L4 7e6a
70. 9ec4 L1 L2 L3 L4 7e6a 842c
71. 9ec4 L1 L2 L3 L4 7e6a 842c 4aa0
72. 9ec4 842c 4aa0 L1 L2 L3 L4 7e6a aea9
73. 9ec4 4aa0 L1 L2 L3 L4 7e6a aea9 842c d886
74. d886 9ec4 4aa0 L1 L2 L3 L4 7e6a aea9 842c 4b25
75. All the way up
76. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
77. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
78. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
79. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
80. [“L1”, “L2”,“L3”, “L4”] |> Enum.chunk_every(2)
81. [[“L1”, “L2”],[“L3”, “L4”]]
82. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
83. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
84. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
85. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
86. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
87. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
88. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
89. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
90. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
91. [“L1”, “L2”] |> MerkleTree.new
92. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
93. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
94. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
95. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
96. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
97. Audit Proof
98. P2 9ec4 H3 L1 L2 L3 L4 H2 P1 H3 R
99. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3) R H3 H2 H3
100. P2 9ec4 h(L4) L1 L2 L3 L4 H2 P1 h(L3) R H3 H3
101. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3) R H2 H3 H3
102. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3) R H2 H3 H3
103. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3) R H2 H3 H3
104. How they are useful?
105. Detect Inconsistencies
106. L1 L4 Replica 1 L1 L3 L1 L4 L1 L2 L1 L4 L1 L2 Replica 2 Replica 3 Repair Coordinator
107. L1 L4 Replica 1 H1 H3 L1 L3 L1 L4 H1 H2 L1 L2 L1 L4 H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
108. L1 L4 Replica 1 B H1 H3 L1 L3 L1 L4 A H1 H2 L1 L2 L1 L4 A H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
109. L1 L4 Replica 1 B H1 H3 L1 L3 L1 L4 A H1 H2 L1 L2 L1 L4 A H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
110. B H1 H2 H1 H3 L1 L2 L1 L3 A Replica 2 Replica 3
111. B H1 H2 H1 H3 L1 L2 L1 L3 A Replica 2 Replica 3
112. B H1 H2 H1 H3 L1 L2 L1 L3 A Replica 2 Replica 3
113. B H1 H2 H1 H3 L1 L2 L1 L2 A Replica 2 Replica 3
114. B H1 H2 H1 H2 L1 L2 L1 L2 A Replica 2 Replica 3
115. A H1 H2 H1 H2 L1 L2 L1 L2 A Replica 2 Replica 3
116. Dynamo, Riak, and Cassandra use this to repair bad replicas!
117. Peer-to-peer file sharing
118. 9cee
119. 9cee L3
120. 9cee b2d0 L3
121. 9cee b2d0 L3 8f14 b2d0
122. 9cee b2d0 L3 8f14 165f b2d0
123. 165f 8f14 L3 e831 b2d0 9cee b2d0
124. 165f 8f14 L3 e831 b2d0 9cee ✅ ✅ b2d0
125. Copy-on-write
126. Copy-on-write data structures are also called persistent data structures.
127. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 8f14
128. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 8f14 L4
129. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 8f14 L4 5e42
130. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 8f14 L4 5e42 3d0b
131. a871 L1 L2 L3 L4 e4da e831 b2d0 L4 4a88 3d0b 5e42
132. Instead of taking a full copy, we can share the same tree between both the copy and the original tree.
133. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 5e42 3d0b 4a88 8f14 L4
134. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4 8f14 Version 1
135. a871 L1 L2 L3 L4 e4da e831 b2d0 L4 4a88 3d0b 5e42 Version 2
136. Wrapping-up
137. Merkle trees are binary trees containing an infinite number of cryptographic hashes.
138. Leaves contains hashes of data blocks.
139. Nodes contains hashes of their children.
140. Produce a root hash that summarizes an entire data set and it’s publicly distributed.
141. Easily prove that a given data block exists in the tree.
142. You can find them in…
143. And lots of papers…
144. @ordepdev Thank you.
145. diving into @ordepdev Merkle Trees