Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Knee-Deep Into P2P: A Tale of Fail (non-Elixir)
Search
Fernando Mendes
March 07, 2018
Programming
0
130
Knee-Deep Into P2P: A Tale of Fail (non-Elixir)
The slides for my talk at Lambda Days 2018 (
www.lambdadays.org/lambdadays2018
)
Fernando Mendes
March 07, 2018
Tweet
Share
More Decks by Fernando Mendes
See All by Fernando Mendes
you. and the morals of technology
fribmendes
1
100
Knee-Deep Into P2P: A Tale of Fail (PWL Porto)
fribmendes
0
44
Knee-Deep Into P2P: A Tale of Fail (ElixirConf EU 2018 version)
fribmendes
0
89
A Look Into Bloom Filters
fribmendes
0
210
Bloom Filters: A Look Into Ruby
fribmendes
0
98
Programming WTF: HTML & CSS
fribmendes
4
150
Ruby: A (pointless) Workshop
fribmendes
1
150
Elixir: A Talk For College Students
fribmendes
0
140
Riding Rails
fribmendes
0
94
Other Decks in Programming
See All in Programming
try! Swift Tokyo 初参加報告LT
hinakko2
0
220
Netty Chicago Java User Group 2024-04-17
sullis
0
170
Apache Hive 4 on Treasure Data
ryukobayashi
0
320
新宿ダンジョンを可視化してみた
satoshi7190
2
260
Git Lint
bkuhlmann
4
750
効率化に挑戦してみたらモバイル開発が少し快適になった話
ryunakayama
0
130
[技育CAMPアカデミア]アイディアを形に!【超入門】スマホアプリ開発〜リリースまでの流れをご紹介
teamlab
PRO
0
370
R言語の環境構築と基礎 Tokyo.R 112
bob3bob3
0
270
DMMプラットフォームがTiDB Cloudを採用した背景
pospome
8
4.1k
educure_カリキュラム生操作マニュアル.pdf
linew_official
0
790
try!Swift Tokyo 2024 参加報告 LT
akidon0000
1
220
Goのエラースタックトレースの歴史と今後
sonatard
9
1.4k
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
289
19k
Fantastic passwords and where to find them - at NoRuKo
philnash
37
2.5k
Adopting Sorbet at Scale
ufuk
68
8.6k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
Statistics for Hackers
jakevdp
789
220k
The Power of CSS Pseudo Elements
geoffreycrofte
60
5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
7
1k
Producing Creativity
orderedlist
PRO
337
39k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
2
3.4k
What the flash - Photography Introduction
edds
64
11k
Code Review Best Practice
trishagee
55
15k
Transcript
Knee-Deep Into P2P A Tale of Fail @fribmendes
Knee-Deep Into P2P A Tale of Fail @fribmendes
“This is SOOO boring…” - me, circa last year or
smth
None
None
None
“This is SOOO boring…” - me, 10 months ago
None
None
None
None
None
“ ” - me, whenever I get to play with
distributed systems
None
I don’t know how to smart office
I know how to web development
I know how to web development … what now?
None
Step 1: receive new connections
Step 1: receive new connections Step 2: accept and send
messages
Step 1: receive new connections Step 2: accept and send
messages Step 3: do a bunch of Steps 1 and 2
Step 1: receive new connections
None
defmodule Gossip.Server do def listen(pid, port) do {:ok, server_socket} =
:gen_tcp.listen(port, @socket_opts) accept_loop(pid, server_socket) end defp accept_loop(pid, server_socket) do {:ok, client} = :gen_tcp.accept(server_socket) :inet.setopts(client, [active: true]) :gen_tcp.controlling_process(client, pid) Gossip.accept(pid, client) accept_loop(pid, server_socket) end end
defmodule Gossip.Server do def listen(pid, port) do {:ok, server_socket} =
:gen_tcp.listen(port, @socket_opts) accept_loop(pid, server_socket) end defp accept_loop(pid, server_socket) do {:ok, client} = :gen_tcp.accept(server_socket) :inet.setopts(client, [active: true]) :gen_tcp.controlling_process(client, pid) Gossip.accept(pid, client) accept_loop(pid, server_socket) end end
Step 1: receive new connections Step 2: accept and send
messages
None
defmodule Gossip.Worker do def recv_loop(pid, socket) do continue = receive
do {:tcp, _port, msg} -> Gossip.recv(pid, msg) true {:tcp_closed, port} -> :gen_tcp.close(port) Gossip.disconnect(pid, self()) false {:send, msg} -> :gen_tcp.send(socket, msg) true end continue and recv_loop(pid, socket) end end
defmodule Gossip.Worker do def recv_loop(pid, socket) do continue = receive
do {:tcp, _port, msg} -> Gossip.recv(pid, msg) true {:tcp_closed, port} -> :gen_tcp.close(port) Gossip.disconnect(pid, self()) false {:send, msg} -> :gen_tcp.send(socket, msg) true end continue and recv_loop(pid, socket) end end
Step 1: receive new connections Step 2: accept and send
messages Step 3: do a bunch of Steps 1 and 2
Raspberry Pi #1 Raspberry Pi #2
None
None
“Does it scale?”
None
Time to copy off the internet find a creative solution
None
g
Gnutella
Gnutella
Gnutella
Gnutella
Gnutella
g (gnutella2)
Gnutella
G2/Gnutella2
G2/Gnutella2
G2/Gnutella2
G2/Gnutella2
None
None
None
None
None
HyParView
None
None
None
None
None
None
Plumtrees
Optimal number of messages
But you can’t afford to lose nodes
None
None
None
None
None
None
None
None
None
None
“Aha! It works on my computer!”
“Great but we need something to show”
“Great but we need something to show” (aka Raspberry Pi
time)
“Sure. Seems legit…” — @iampfac
“Wait. He works here!?” — @naps62
“Hey, I can borrow™ someone else’s code”
None
None
None
you shall not pass!
Solution: stick everything on Raspberry Pi’s
Things running on one Raspberry Pi
Things running on one Raspberry Pi ✓BEAM
Things running on one Raspberry Pi ✓BEAM ✓thebox (sensors)
Things running on one Raspberry Pi ✓BEAM ✓thebox (sensors) ✓Phoenix
app
Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)
✓Phoenix app
Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)
✓Phoenix app ✓Postgres
Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)
✓Phoenix app ✓Postgres ✓Cassandra
Things running on one Raspberry Pi ✓BEAM (x2) ✓thebox (sensors)
✓Phoenix app ✓Postgres ✓Cassandra it works!
None
None
@antenna = Satellite::Antenna.new(host, port) @antenna.on(“data") do |data| if data.avg_temp >
25 slack.send_msg(“people, it's too hot") end end @antenna.on("lights") do |payload, data| payload == "on" ? turn_lights_on : turn_lights_off end @antenna.watch
“Looking good! Everything’s working!”
None
State of each node:
State of each node: • Last sensor readings
State of each node: • Last sensor readings • Network
map (MAC-IP)
State of each node: • Last sensor readings • Network
map (MAC-IP) • Target values
State of each node: • Last sensor readings • Network
map (MAC-IP) • Target values
None
How do we handle concurrency?
None
None
No database locks. No transactions. You’re on your own, kiddo.
Vector Clocks
None
None
None
None
None
None
None
Shared state and why it sucks
Vector = (1, 0) Vector = (0, 1)
CAP Theorem
CAP Theorem “you’re a programmer. you can’t have nice things.”
consistency availability partitioning
consistency availability partitioning
Eventual Consistency
CRDTs
Operation-Based CRDT
Operation-Based CRDT commutative but not idempotent update exactly once
no CRDTs
no CRDTs
no CRDTs
no CRDTs
Op-based CRDTs
Op-based CRDTs
Op-based CRDTs
Op-based CRDTs
State-Based CRDT
State-Based CRDT commutative and idempotent heavier on the network
State-based CRDTs
State-based CRDTs
State-based CRDTs
State-based CRDTs
None
None
None
None
None
None
None
Wrapping up
System resources matter
System resources matter your algorithms should account for them
There are models. Use them.
Distributed System Checklist
Distributed System Checklist •Is the number of processes known or
finite?
Distributed System Checklist •Is the number of processes known or
finite? •Is there a global notion of time?
Distributed System Checklist •Is the number of processes known or
finite? •Is there a global notion of time? •Is the network reliable?
Distributed System Checklist •Is the number of processes known or
finite? •Is there a global notion of time? •Is the network reliable? •Is there full connectivity?
Distributed System Checklist •Is the number of processes known or
finite? •Is there a global notion of time? •Is the network reliable? •Is there full connectivity? •What happens when a process crashes?
It really doesn’t change that much
CRDTs aren’t a golden hammer
Reinventing the wheel is stupid
None
Knee-Deep Into P2P A Tale of Fail @fribmendes