Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Knee-Deep Into P2P: A Tale of Fail (PWL Porto)

Knee-Deep Into P2P: A Tale of Fail (PWL Porto)

Fernando Mendes

May 23, 2018
Tweet

More Decks by Fernando Mendes

Other Decks in Programming

Transcript

  1. Knee-Deep Into P2P
    A Tale of Fail
    @fribmendes

    View Slide

  2. Knee-Deep Into P2P
    A Tale of Fail
    @fribmendes

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. I don’t know how to
    smart office

    View Slide

  9. I don’t know how to
    smart office
    … what now?

    View Slide

  10. @fribmendes
    me failing
    at photoshop

    View Slide

  11. View Slide

  12. I don’t know how to
    smart office
    … what now?

    View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. Step 1: receive new connections

    View Slide

  18. Step 1: receive new connections
    Step 2: accept and send messages

    View Slide

  19. Step 1: receive new connections
    Step 2: accept and send messages
    Step 3: do a bunch of Steps 1 and 2

    View Slide

  20. Step 1: receive new connections

    View Slide

  21. View Slide

  22. defp accept_loop(pid, server_socket) do
    {:ok, client} = :gen_tcp.accept(server_socket)
    :inet.setopts(client, [active: true])
    :gen_tcp.controlling_process(client, pid)
    Gossip.accept(pid, client)
    accept_loop(pid, server_socket)
    end

    View Slide

  23. defp accept_loop(pid, server_socket) do
    {:ok, client} = :gen_tcp.accept(server_socket)
    :inet.setopts(client, [active: true])
    :gen_tcp.controlling_process(client, pid)
    Gossip.accept(pid, client)
    accept_loop(pid, server_socket)
    end

    View Slide

  24. Step 1: receive new connections
    Step 2: accept and send messages

    View Slide

  25. View Slide

  26. def recv_loop(pid, socket) do
    receive do
    {:tcp, _port, msg} ->
    # process an incoming message
    {:tcp_closed, port} ->
    # close the sockets
    {:send, msg} ->
    # send an outgoing message
    end
    end
    end

    View Slide

  27. Step 1: receive new connections
    Step 2: accept and send messages
    Step 3: do a bunch of Steps 1 and 2

    View Slide

  28. Raspberry Pi #1 Raspberry Pi #2

    View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. “Does it scale?”

    View Slide

  33. View Slide

  34. View Slide

  35. g

    View Slide

  36. Gnutella

    View Slide

  37. Gnutella

    View Slide

  38. Gnutella

    View Slide

  39. Gnutella

    View Slide

  40. Gnutella

    View Slide

  41. g

    View Slide

  42. g
    (gnutella2)

    View Slide

  43. Gnutella

    View Slide

  44. G2/Gnutella2

    View Slide

  45. G2/Gnutella2

    View Slide

  46. G2/Gnutella2

    View Slide

  47. G2/Gnutella2

    View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. View Slide

  52. View Slide

  53. HyParView

    View Slide

  54. View Slide

  55. View Slide

  56. View Slide

  57. View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. Plumtrees

    View Slide

  63. Optimal number of messages

    View Slide

  64. But you can’t afford to lose nodes

    View Slide

  65. View Slide

  66. View Slide

  67. View Slide

  68. View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. View Slide

  74. View Slide

  75. View Slide

  76. “Aha! It works on my computer!”

    View Slide

  77. “Aha! It works on my computer!”

    View Slide

  78. “Great but we need
    something to show”

    View Slide

  79. “Great but we need
    something to show”
    (aka Raspberry Pi time)

    View Slide

  80. “Guys… Is this
    a bomb? Are we
    going to die?”
    — @naps62

    View Slide

  81. “Hey, I can borrow™ someone else’s code”

    View Slide

  82. View Slide

  83. View Slide

  84. View Slide

  85. you shall not pass!

    View Slide

  86. Stick everything on Raspberry Pi’s

    View Slide

  87. Things running on one Raspberry Pi

    View Slide

  88. Things running on one Raspberry Pi
    ✓BEAM

    View Slide

  89. Things running on one Raspberry Pi
    ✓BEAM
    ✓thebox (sensors)

    View Slide

  90. Things running on one Raspberry Pi
    ✓BEAM
    ✓thebox (sensors)
    ✓Phoenix app

    View Slide

  91. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app

    View Slide

  92. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres

    View Slide

  93. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres
    ✓Cassandra

    View Slide

  94. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres
    ✓Cassandra
    it works!

    View Slide

  95. View Slide

  96. View Slide

  97. View Slide

  98. “Looking good! Everything’s working!”

    View Slide

  99. lol, nope

    View Slide

  100. State of each node:

    View Slide

  101. State of each node:
    • Last sensor readings

    View Slide

  102. State of each node:
    • Last sensor readings
    • Network map (MAC-IP)

    View Slide

  103. State of each node:
    • Last sensor readings
    • Network map (MAC-IP)
    • Target values

    View Slide

  104. State of each node:
    • Last sensor readings
    • Network map (MAC-IP)
    • Target values

    View Slide

  105. View Slide

  106. How do we handle concurrency?

    View Slide

  107. View Slide

  108. View Slide

  109. No database locks.
    No transactions.
    You’re on your own, kiddo.

    View Slide

  110. Vector Clocks

    View Slide

  111. View Slide

  112. View Slide

  113. View Slide

  114. View Slide

  115. View Slide

  116. View Slide

  117. View Slide

  118. View Slide

  119. Vector = (1, 0)
    Vector = (0, 1)

    View Slide

  120. CAP Theorem

    View Slide

  121. CAP Theorem
    “you’re a programmer.
    you can’t have nice things.”

    View Slide

  122. consistency
    availability partitioning

    View Slide

  123. consistency
    availability partitioning

    View Slide

  124. View Slide

  125. Eventual Consistency

    View Slide

  126. CRDTs

    View Slide

  127. View Slide

  128. Operation-Based CRDT

    View Slide

  129. Operation-Based CRDT
    commutative but not idempotent
    update exactly once

    View Slide

  130. no CRDTs

    View Slide

  131. no CRDTs

    View Slide

  132. no CRDTs

    View Slide

  133. no CRDTs

    View Slide

  134. Op-based CRDTs

    View Slide

  135. Op-based CRDTs

    View Slide

  136. Op-based CRDTs

    View Slide

  137. Op-based CRDTs

    View Slide

  138. State-Based CRDT

    View Slide

  139. State-Based CRDT
    commutative and idempotent
    heavier on the network

    View Slide

  140. State-based CRDTs

    View Slide

  141. State-based CRDTs

    View Slide

  142. State-based CRDTs

    View Slide

  143. State-based CRDTs

    View Slide

  144. View Slide

  145. View Slide

  146. View Slide

  147. View Slide

  148. View Slide

  149. View Slide

  150. View Slide

  151. View Slide

  152. Wrapping up

    View Slide

  153. System resources matter

    View Slide

  154. System resources matter
    your algorithms should
    account for them

    View Slide

  155. There are models.
    Use them.

    View Slide

  156. Distributed System Checklist

    View Slide

  157. Distributed System Checklist
    •Is the number of processes known or finite?

    View Slide

  158. Distributed System Checklist
    •Is the number of processes known or finite?
    •Is there a global notion of time?

    View Slide

  159. Distributed System Checklist
    •Is the number of processes known or finite?
    •Is there a global notion of time?
    •Is the network reliable?

    View Slide

  160. Distributed System Checklist
    •Is the number of processes known or finite?
    •Is there a global notion of time?
    •Is the network reliable?
    •Is there full connectivity?

    View Slide

  161. Distributed System Checklist
    •Is the number of processes known or finite?
    •Is there a global notion of time?
    •Is the network reliable?
    •Is there full connectivity?
    •What happens when a process crashes?

    View Slide

  162. It really doesn’t change
    that much

    View Slide

  163. CRDTs aren’t a golden hammer

    View Slide

  164. Reinventing the wheel is stupid

    View Slide

  165. View Slide

  166. Knee-Deep Into P2P
    A Tale of Fail
    @fribmendes

    View Slide