Diving into Merkle Trees

Diving into Merkle Trees

Introduced by Ralph C. Merkle, the Merkle Tree is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. Due to its unique content validating and performance qualities, they are especially useful in distributed, peer-to-peer systems where the same data should exist in multiple places. Let's learn how to detect inconsistencies between trees and reduce the amount of transferred data enabling peer-to-peer file sharing by building our own tree with Elixir.

074c1726aee7c73df923c3d5064ab861?s=128

Pedro Tavares

February 21, 2019
Tweet

Transcript

  1. diving into @ordepdev Merkle Trees

  2. @ordepdev @ordepdev

  3. @ordepdev

  4. @ordepdev

  5. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf 1987

  6. “[…] can sign an unlimited number of messages, and the

    signature size increases logarithmically.”
  7. “The general idea in this new system is to use

    an infinite tree of one-time signatures.”
  8. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf https://patents.google.com/patent/US4309569 http://www.merkle.com/papers/Thesis1979.pdf 1979

  9. None
  10. 1989 http://merkle.com/papers/Certified1979.pdf

  11. “[…] is a data structure used for efficiently summarizing and

    verifying the integrity of large sets of data.”
  12. “[…] it was used for the purpose of one-time signatures

    and authenticated public key distribution.”
  13. None
  14. Summarize and Verify

  15. One-way Functions

  16. “A one-way function f is a function that is easy

    to compute but difficult to invert.”
  17. () () () 4a23ca438f3 065b90acc7 ccca99192c

  18. Why do we need them?

  19. “[…] being able to store and identify data with a

    fixed length output can create vast storage savings.”
  20. “The person who computes y=F(x) is the only person who

    knows x.”
  21. One-time Signatures

  22. “One time signatures are practical between a single pair of

    users who are willing to exchange the large amount of data necessary […]”
  23. “If 1000 messages are to be signed before new public

    authentication data is needed, over 20,000,000 bits or 2.5 megabytes must be stored as public information.”
  24. Improving one- time signatures

  25. https://people.eecs.berkeley.edu/~raluca/cs261-f15/readings/merkle.pdf http://www.merkle.com/papers/Thesis1979.pdf

  26. How to compute a Merkle Root?

  27. 9ec4 L1 L2

  28. h(L1)

  29. 9ec4 L1 L2

  30. h(L2)

  31. a871 L1 L2 7e6a 9ec4

  32. h(h(L1)||h(L2))

  33. aea9 a871 7e6a L1 L2 9ec4

  34. None
  35. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Leaf

  36. defmodule MerkleTree.Leaf do defstruct [:hash, :value] @type hash :: String.t

    @type value :: String.t @type t :: %MerkleTree.Leaf{ hash: hash, value: value } end
  37. defmodule MerkleTree.Leaf do defstruct [:hash, :value] @type hash :: String.t

    @type value :: String.t @type t :: %MerkleTree.Leaf{ hash: hash, value: value } end
  38. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Node

  39. defmodule MerkleTree.Node do defstruct [:hash, :left, :right] @type hash ::

    String.t @type left :: MerkleTree.Node.t | MerkleTree.Leaf.t @type right :: MerkleTree.Node.t | MerkleTree.Leaf.t @type t :: %MerkleTree.Node{ hash: hash, left: left, right: right } end
  40. defmodule MerkleTree.Node do defstruct [:hash, :left, :right] @type hash ::

    String.t @type left :: MerkleTree.Node.t | MerkleTree.Leaf.t @type right :: MerkleTree.Node.t | MerkleTree.Leaf.t @type t :: %MerkleTree.Node{ hash: hash, left: left, right: right } end
  41. aea9 a871 7e6a L1 L2 9ec4 MerkleTree.Leaf MerkleTree.Node MerkleTree[:root]

  42. defmodule MerkleTree do defstruct [:root] @type root :: MerkleTree.Node.t @type

    t :: %MerkleTree{ root: root } end
  43. defmodule MerkleTree do defstruct [:root] @type root :: MerkleTree.Node.t @type

    t :: %MerkleTree{ root: root } end
  44. Hashing the data blocks

  45. defmodule MerkleTree.Crypto do def hash(input, type \\ :sha256) do type

    |> :crypto.hash("#{input}") |> Base.encode16 end end
  46. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
  47. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
  48. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
  49. defmodule MerkleTree.Leaf do def build(value, hash) do %Leaf{value: value, hash:

    hash} end end
  50. [“L1”, “L2”] |> MerkleTree.new

  51. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”,

    hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
  52. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”,

    hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
  53. [ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”,

    hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ]
  54. Hashing the nodes

  55. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |>

    Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
  56. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |>

    Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
  57. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |>

    Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
  58. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |>

    Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
  59. defmodule MerkleTree.Node do alias MerkleTree.Crypto def new(nodes) do nodes |>

    Enum.map_join(&(&1.hash)) |> Crypto.hash |> build(nodes) end end
  60. defmodule MerkleTree.Node do def build(hash, [left, right]) do %Node{hash: hash,

    left: left, right: right} end end
  61. MerkleTree.Node.new([ %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” }, %MerkleTree.Leaf{ value: “L2”,

    hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } ])
  62. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” },

    right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
  63. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” },

    right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
  64. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” },

    right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
  65. %MerkleTree.Node{ hash: “F5FC2A0E6D4568040ED7B8D2C59A16B73B8C9EC9...”, left: %MerkleTree.Leaf{ value: “L1”, hash: “6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EA...” },

    right: %MerkleTree.Leaf{ value: “L2”, hash: “D4735E3A265E16EEE03F59718B9B5D03019C07D8...” } }
  66. Building a bigger tree

  67. 9ec4 L1 L2 L3 L4

  68. 9ec4 L1 L2 L3 L4

  69. 9ec4 L1 L2 L3 L4 7e6a

  70. 9ec4 L1 L2 L3 L4 7e6a 842c

  71. 9ec4 L1 L2 L3 L4 7e6a 842c 4aa0

  72. 9ec4 842c 4aa0 L1 L2 L3 L4 7e6a aea9

  73. 9ec4 4aa0 L1 L2 L3 L4 7e6a aea9 842c d886

  74. d886 9ec4 4aa0 L1 L2 L3 L4 7e6a aea9 842c

    4b25
  75. All the way up

  76. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) end end
  77. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  78. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  79. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  80. [“L1”, “L2”,“L3”, “L4”] |> Enum.chunk_every(2)

  81. [[“L1”, “L2”],[“L3”, “L4”]]

  82. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  83. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  84. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  85. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  86. defmodule MerkleTree do defp build([root]) do %MerkleTree{root: root} end defp

    build(nodes) do nodes |> Enum.chunk_every(2) |> Enum.map(&Node.new(&1)) |> build end end
  87. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
  88. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
  89. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
  90. defmodule MerkleTree do alias MerkleTree.Leaf alias MerkleTree.Crypto def new(blocks) do

    blocks |> Enum.map(&%Leaf.build(&1, Crypto.hash(&1))) |> build end end
  91. [“L1”, “L2”] |> MerkleTree.new

  92. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash:

    “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
  93. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash:

    “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
  94. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash:

    “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
  95. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash:

    “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
  96. %MerkleTree{ root: %MerkleTree.Node{ hash: “8C569660D98A20D59DE10E134D81A8CE55...”, left: %MerkleTree.Leaf{ value: “L1”, hash:

    “DFFE8596427FC50E8F64654A609AF135...” }, right: %MerkleTree.Leaf{ value: “L2”, hash: “D76354D8457898445BB69E0DC0DC95FB...” } } }
  97. Audit Proof

  98. P2 9ec4 H3 L1 L2 L3 L4 H2 P1 H3

    R
  99. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3)

    R H3 H2 H3
  100. P2 9ec4 h(L4) L1 L2 L3 L4 H2 P1 h(L3)

    R H3 H3
  101. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3)

    R H2 H3 H3
  102. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3)

    R H2 H3 H3
  103. P2 9ec4 h(L4) L1 L2 L3 L4 h(L2) P1 h(L3)

    R H2 H3 H3
  104. How they are useful?

  105. Detect Inconsistencies

  106. L1 L4 Replica 1 L1 L3 L1 L4 L1 L2

    L1 L4 L1 L2 Replica 2 Replica 3 Repair Coordinator
  107. L1 L4 Replica 1 H1 H3 L1 L3 L1 L4

    H1 H2 L1 L2 L1 L4 H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
  108. L1 L4 Replica 1 B H1 H3 L1 L3 L1

    L4 A H1 H2 L1 L2 L1 L4 A H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
  109. L1 L4 Replica 1 B H1 H3 L1 L3 L1

    L4 A H1 H2 L1 L2 L1 L4 A H1 H2 L1 L2 Replica 2 Replica 3 Repair Coordinator
  110. B H1 H2 H1 H3 L1 L2 L1 L3 A

    Replica 2 Replica 3
  111. B H1 H2 H1 H3 L1 L2 L1 L3 A

    Replica 2 Replica 3
  112. B H1 H2 H1 H3 L1 L2 L1 L3 A

    Replica 2 Replica 3
  113. B H1 H2 H1 H3 L1 L2 L1 L2 A

    Replica 2 Replica 3
  114. B H1 H2 H1 H2 L1 L2 L1 L2 A

    Replica 2 Replica 3
  115. A H1 H2 H1 H2 L1 L2 L1 L2 A

    Replica 2 Replica 3
  116. Dynamo, Riak, and Cassandra use this to repair bad replicas!

  117. Peer-to-peer file sharing

  118. 9cee

  119. 9cee L3

  120. 9cee b2d0 L3

  121. 9cee b2d0 L3 8f14 b2d0

  122. 9cee b2d0 L3 8f14 165f b2d0

  123. 165f 8f14 L3 e831 b2d0 9cee b2d0

  124. 165f 8f14 L3 e831 b2d0 9cee ✅ ✅ b2d0

  125. Copy-on-write

  126. Copy-on-write data structures are also called persistent data structures.

  127. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    8f14
  128. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    8f14 L4
  129. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    8f14 L4 5e42
  130. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    8f14 L4 5e42 3d0b
  131. a871 L1 L2 L3 L4 e4da e831 b2d0 L4 4a88

    3d0b 5e42
  132. Instead of taking a full copy, we can share the

    same tree between both the copy and the original tree.
  133. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    5e42 3d0b 4a88 8f14 L4
  134. 165f a871 L1 L2 L3 e4da e831 b2d0 9cee L4

    8f14 Version 1
  135. a871 L1 L2 L3 L4 e4da e831 b2d0 L4 4a88

    3d0b 5e42 Version 2
  136. Wrapping-up

  137. Merkle trees are binary trees containing an infinite number of

    cryptographic hashes.
  138. Leaves contains hashes of data blocks.

  139. Nodes contains hashes of their children.

  140. Produce a root hash that summarizes an entire data set

    and it’s publicly distributed.
  141. Easily prove that a given data block exists in the

    tree.
  142. You can find them in…

  143. And lots of papers…

  144. @ordepdev Thank you.

  145. diving into @ordepdev Merkle Trees