Knee-Deep Into P2P: A Tale of Fail (PWL Porto)

Transcript

1. Knee-Deep Into P2P
   A Tale of Fail
   @fribmendes
   

   View Slide

2. Knee-Deep Into P2P
   A Tale of Fail
   @fribmendes
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. I don’t know how to
   smart office
   

   View Slide

9. I don’t know how to
   smart office
   … what now?
   

   View Slide

10. @fribmendes
    me failing
    at photoshop
    

    View Slide

11. View Slide

12. I don’t know how to
    smart office
    … what now?
    

    View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. Step 1: receive new connections
    

    View Slide

18. Step 1: receive new connections
    Step 2: accept and send messages
    

    View Slide

19. Step 1: receive new connections
    Step 2: accept and send messages
    Step 3: do a bunch of Steps 1 and 2
    

    View Slide

20. Step 1: receive new connections
    

    View Slide

21. View Slide

22. defp accept_loop(pid, server_socket) do
    {:ok, client} = :gen_tcp.accept(server_socket)
    :inet.setopts(client, [active: true])
    :gen_tcp.controlling_process(client, pid)
    Gossip.accept(pid, client)
    accept_loop(pid, server_socket)
    end
    

    View Slide

23. defp accept_loop(pid, server_socket) do
    {:ok, client} = :gen_tcp.accept(server_socket)
    :inet.setopts(client, [active: true])
    :gen_tcp.controlling_process(client, pid)
    Gossip.accept(pid, client)
    accept_loop(pid, server_socket)
    end
    

    View Slide

24. Step 1: receive new connections
    Step 2: accept and send messages
    

    View Slide

25. View Slide

26. def recv_loop(pid, socket) do
    receive do
    {:tcp, _port, msg} ->
    # process an incoming message
    {:tcp_closed, port} ->
    # close the sockets
    {:send, msg} ->
    # send an outgoing message
    end
    end
    end
    

    View Slide

27. Step 1: receive new connections
    Step 2: accept and send messages
    Step 3: do a bunch of Steps 1 and 2
    

    View Slide

28. Raspberry Pi #1 Raspberry Pi #2
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. “Does it scale?”
    

    View Slide

33. View Slide

34. View Slide

35. g
    

    View Slide

36. Gnutella
    

    View Slide

37. Gnutella
    

    View Slide

38. Gnutella
    

    View Slide

39. Gnutella
    

    View Slide

40. Gnutella
    

    View Slide

41. g
    

    View Slide

42. g
    (gnutella2)
    

    View Slide

43. Gnutella
    

    View Slide

44. G2/Gnutella2
    

    View Slide

45. G2/Gnutella2
    

    View Slide

46. G2/Gnutella2
    

    View Slide

47. G2/Gnutella2
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. View Slide

52. View Slide

53. HyParView
    

    View Slide

54. View Slide

55. View Slide

56. View Slide

57. View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. Plumtrees
    

    View Slide

63. Optimal number of messages
    

    View Slide

64. But you can’t afford to lose nodes
    

    View Slide

65. View Slide

66. View Slide

67. View Slide

68. View Slide

69. View Slide

70. View Slide

71. View Slide

72. View Slide

73. View Slide

74. View Slide

75. View Slide

76. “Aha! It works on my computer!”
    

    View Slide

77. “Aha! It works on my computer!”
    

    View Slide

78. “Great but we need
    something to show”
    

    View Slide

79. “Great but we need
    something to show”
    (aka Raspberry Pi time)
    

    View Slide

80. “Guys… Is this
    a bomb? Are we
    going to die?”
    — @naps62
    

    View Slide

81. “Hey, I can borrow™ someone else’s code”
    

    View Slide

82. View Slide

83. View Slide

84. View Slide

85. you shall not pass!
    

    View Slide

86. Stick everything on Raspberry Pi’s
    

    View Slide

87. Things running on one Raspberry Pi
    

    View Slide

88. Things running on one Raspberry Pi
    ✓BEAM
    

    View Slide

89. Things running on one Raspberry Pi
    ✓BEAM
    ✓thebox (sensors)
    

    View Slide

90. Things running on one Raspberry Pi
    ✓BEAM
    ✓thebox (sensors)
    ✓Phoenix app
    

    View Slide

91. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    

    View Slide

92. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres
    

    View Slide

93. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres
    ✓Cassandra
    

    View Slide

94. Things running on one Raspberry Pi
    ✓BEAM (x2)
    ✓thebox (sensors)
    ✓Phoenix app
    ✓Postgres
    ✓Cassandra
    it works!
    

    View Slide

95. View Slide

96. View Slide

97. View Slide

98. “Looking good! Everything’s working!”
    

    View Slide

99. lol, nope
    

    View Slide

100. State of each node:
     

     View Slide

101. State of each node:
     • Last sensor readings
     

     View Slide

102. State of each node:
     • Last sensor readings
     • Network map (MAC-IP)
     

     View Slide

103. State of each node:
     • Last sensor readings
     • Network map (MAC-IP)
     • Target values
     

     View Slide

104. State of each node:
     • Last sensor readings
     • Network map (MAC-IP)
     • Target values
     

     View Slide

105. View Slide

106. How do we handle concurrency?
     

     View Slide

107. View Slide

108. View Slide

109. No database locks.
     No transactions.
     You’re on your own, kiddo.
     

     View Slide

110. Vector Clocks
     

     View Slide

111. View Slide

112. View Slide

113. View Slide

114. View Slide

115. View Slide

116. View Slide

117. View Slide

118. View Slide

119. Vector = (1, 0)
     Vector = (0, 1)
     

     View Slide

120. CAP Theorem
     

     View Slide

121. CAP Theorem
     “you’re a programmer.
     you can’t have nice things.”
     

     View Slide

122. consistency
     availability partitioning
     

     View Slide

123. consistency
     availability partitioning
     

     View Slide

124. View Slide

125. Eventual Consistency
     

     View Slide

126. CRDTs
     

     View Slide

127. View Slide

128. Operation-Based CRDT
     

     View Slide

129. Operation-Based CRDT
     commutative but not idempotent
     update exactly once
     

     View Slide

130. no CRDTs
     

     View Slide

131. no CRDTs
     

     View Slide

132. no CRDTs
     

     View Slide

133. no CRDTs
     

     View Slide

134. Op-based CRDTs
     

     View Slide

135. Op-based CRDTs
     

     View Slide

136. Op-based CRDTs
     

     View Slide

137. Op-based CRDTs
     

     View Slide

138. State-Based CRDT
     

     View Slide

139. State-Based CRDT
     commutative and idempotent
     heavier on the network
     

     View Slide

140. State-based CRDTs
     

     View Slide

141. State-based CRDTs
     

     View Slide

142. State-based CRDTs
     

     View Slide

143. State-based CRDTs
     

     View Slide

144. View Slide

145. View Slide

146. View Slide

147. View Slide

148. View Slide

149. View Slide

150. View Slide

151. View Slide

152. Wrapping up
     

     View Slide

153. System resources matter
     

     View Slide

154. System resources matter
     your algorithms should
     account for them
     

     View Slide

155. There are models.
     Use them.
     

     View Slide

156. Distributed System Checklist
     

     View Slide

157. Distributed System Checklist
     •Is the number of processes known or finite?
     

     View Slide

158. Distributed System Checklist
     •Is the number of processes known or finite?
     •Is there a global notion of time?
     

     View Slide

159. Distributed System Checklist
     •Is the number of processes known or finite?
     •Is there a global notion of time?
     •Is the network reliable?
     

     View Slide

160. Distributed System Checklist
     •Is the number of processes known or finite?
     •Is there a global notion of time?
     •Is the network reliable?
     •Is there full connectivity?
     

     View Slide

161. Distributed System Checklist
     •Is the number of processes known or finite?
     •Is there a global notion of time?
     •Is the network reliable?
     •Is there full connectivity?
     •What happens when a process crashes?
     

     View Slide

162. It really doesn’t change
     that much
     

     View Slide

163. CRDTs aren’t a golden hammer
     

     View Slide

164. Reinventing the wheel is stupid
     

     View Slide

165. View Slide

166. Knee-Deep Into P2P
     A Tale of Fail
     @fribmendes
     

     View Slide