Knee-Deep Into P2P: A Tale of Fail (non-Elixir)

Transcript

1. Knee-Deep Into P2P A Tale of Fail @fribmendes

2. Knee-Deep Into P2P A Tale of Fail @fribmendes

3. “This is SOOO boring…” - me, circa last year or

   smth
4. None
5. None
6. None

7. “This is SOOO boring…” - me, 10 months ago

8. None
9. None
10. None
11. None
12. None

13. “ ” - me, whenever I get to play with

    distributed systems
14. None

15. I don’t know how to smart office

16. I know how to web development

17. I know how to web development … what now?

18. None

19. Step 1: receive new connections

20. Step 1: receive new connections Step 2: accept and send

    messages

21. Step 1: receive new connections Step 2: accept and send

    messages Step 3: do a bunch of Steps 1 and 2

22. Step 1: receive new connections

23. None

24. defmodule Gossip.Server do def listen(pid, port) do {:ok, server_socket} =

    :gen_tcp.listen(port, @socket_opts) accept_loop(pid, server_socket) end defp accept_loop(pid, server_socket) do {:ok, client} = :gen_tcp.accept(server_socket) :inet.setopts(client, [active: true]) :gen_tcp.controlling_process(client, pid) Gossip.accept(pid, client) accept_loop(pid, server_socket) end end

25. defmodule Gossip.Server do def listen(pid, port) do {:ok, server_socket} =

    :gen_tcp.listen(port, @socket_opts) accept_loop(pid, server_socket) end defp accept_loop(pid, server_socket) do {:ok, client} = :gen_tcp.accept(server_socket) :inet.setopts(client, [active: true]) :gen_tcp.controlling_process(client, pid) Gossip.accept(pid, client) accept_loop(pid, server_socket) end end

26. Step 1: receive new connections Step 2: accept and send

    messages
27. None

28. defmodule Gossip.Worker do def recv_loop(pid, socket) do continue = receive

    do {:tcp, _port, msg} -> Gossip.recv(pid, msg) true {:tcp_closed, port} -> :gen_tcp.close(port) Gossip.disconnect(pid, self()) false {:send, msg} -> :gen_tcp.send(socket, msg) true end continue and recv_loop(pid, socket) end end

29. defmodule Gossip.Worker do def recv_loop(pid, socket) do continue = receive

    do {:tcp, _port, msg} -> Gossip.recv(pid, msg) true {:tcp_closed, port} -> :gen_tcp.close(port) Gossip.disconnect(pid, self()) false {:send, msg} -> :gen_tcp.send(socket, msg) true end continue and recv_loop(pid, socket) end end

30. Step 1: receive new connections Step 2: accept and send

    messages Step 3: do a bunch of Steps 1 and 2

31. Raspberry Pi #1 Raspberry Pi #2

32. None
33. None

34. “Does it scale?”

35. None

36. Time to copy off the internet find a creative solution

37. None

38. g

39. Gnutella

40. Gnutella

41. Gnutella

42. Gnutella

43. Gnutella

44. g (gnutella2)

45. Gnutella

46. G2/Gnutella2

47. G2/Gnutella2

48. G2/Gnutella2

49. G2/Gnutella2

50. None
51. None
52. None
53. None
54. None

55. HyParView

56. None
57. None
58. None
59. None
60. None
61. None

62. Plumtrees

63. Optimal number of messages

64. But you can’t afford to lose nodes

65. None
66. None
67. None
68. None
69. None
70. None
71. None
72. None
73. None
74. None

75. “Aha! It works on my computer!”

76. “Great but we need something to show”

77. “Great but we need something to show” (aka Raspberry Pi

    time)

78. “Sure. Seems legit…” — @iampfac

79. “Wait. He works here!?” — @naps62

80. “Hey, I can borrow™ someone else’s code”

81. None
82. None
83. None

84. you shall not pass!

85. Solution: stick everything on Raspberry Pi’s

86. Things running on one Raspberry Pi

87. Things running on one Raspberry Pi ✓BEAM

88. Things running on one Raspberry Pi ✓BEAM ✓thebox (sensors)

89. Things running on one Raspberry Pi ✓BEAM ✓thebox (sensors) ✓Phoenix

    app

90. Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)

    ✓Phoenix app

91. Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)

    ✓Phoenix app ✓Postgres

92. Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)

    ✓Phoenix app ✓Postgres ✓Cassandra

93. Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)

    ✓Phoenix app ✓Postgres ✓Cassandra it works!
94. None
95. None

96. @antenna = Satellite::Antenna.new(host, port) @antenna.on(“data") do |data| if data.avg_temp >

    25 slack.send_msg(“people, it's too hot") end end @antenna.on("lights") do |payload, data| payload == "on" ? turn_lights_on : turn_lights_off end @antenna.watch

97. “Looking good! Everything’s working!”

98. None

99. State of each node:

100. State of each node: • Last sensor readings

101. State of each node: • Last sensor readings • Network

     map (MAC-IP)

102. State of each node: • Last sensor readings • Network

     map (MAC-IP) • Target values

103. State of each node: • Last sensor readings • Network

     map (MAC-IP) • Target values
104. None

105. How do we handle concurrency?

106. None
107. None

108. No database locks. No transactions. You’re on your own, kiddo.

109. Vector Clocks

110. None
111. None
112. None
113. None
114. None
115. None
116. None

117. Shared state and why it sucks

118. Vector = (1, 0) Vector = (0, 1)

119. CAP Theorem

120. CAP Theorem “you’re a programmer. you can’t have nice things.”

121. consistency availability partitioning

122. consistency availability partitioning

123. Eventual Consistency

124. CRDTs

125. Operation-Based CRDT

126. Operation-Based CRDT commutative but not idempotent update exactly once

127. no CRDTs

128. no CRDTs

129. no CRDTs

130. no CRDTs

131. Op-based CRDTs

132. Op-based CRDTs

133. Op-based CRDTs

134. Op-based CRDTs

135. State-Based CRDT

136. State-Based CRDT commutative and idempotent heavier on the network

137. State-based CRDTs

138. State-based CRDTs

139. State-based CRDTs

140. State-based CRDTs

141. None
142. None
143. None
144. None
145. None
146. None
147. None

148. Wrapping up

149. System resources matter

150. System resources matter your algorithms should account for them

151. There are models. Use them.

152. Distributed System Checklist

153. Distributed System Checklist •Is the number of processes known or

     finite?

154. Distributed System Checklist •Is the number of processes known or

     finite? •Is there a global notion of time?

155. Distributed System Checklist •Is the number of processes known or

     finite? •Is there a global notion of time? •Is the network reliable?

156. Distributed System Checklist •Is the number of processes known or

     finite? •Is there a global notion of time? •Is the network reliable? •Is there full connectivity?

157. Distributed System Checklist •Is the number of processes known or

     finite? •Is there a global notion of time? •Is the network reliable? •Is there full connectivity? •What happens when a process crashes?

158. It really doesn’t change that much

159. CRDTs aren’t a golden hammer

160. Reinventing the wheel is stupid

161. None

162. Knee-Deep Into P2P A Tale of Fail @fribmendes