Scaling connections in P2P apps

This presentation discusses various protocols that can be used to establish and propagate messages in huge clusters of nodes (>1000), focused on context of Peer to Peer systems. It covers the differences between P2P and client-server model, challenges related to local and global networks operating in P2P fashion.

Transcript

1. SCALING CONNECTIONS IN P2P APPS
2. INTRODUCTION Bartosz Sypytkowski @Horusiath b.sypytkowski@gmail.com bartoszsypytkowski.com
3. — P2P vs. Client/Server — Networking 101 — Membership & peer discovery — Gossiping data efficiently AGENDA
4. CLIENT / SERVER PEER TO PEER Local network Local network “The cloud” Local network Local network
5. CLIENT / SERVER PEER TO PEER • Roles: initiator or acceptor • Cluster of predictable size • DC network: solid, fast • Servers: powerful hardware, always on • Dedicated roles (DB, app server, cache) • Roles: both initiator and acceptor • Unbounded number of members • Variadic, unpredictable network • Peers: various hardware, may require power management • No preconfigured roles
6. ▪ Data ownership ▪ Always available ▪ Cost reduction ▪ “Simpler” programming model LOCAL-FIRST SOFTWARE
7. 1. P2P systems 2. NOT IoT 3. Hundreds/thousands users 4. Focused on message exchange OUR USE CASES
8. CLIENT / SERVER PEER TO PEER Local network Local network “The cloud” Local network Local neatwork How two devices can discover each other?
9. NETWORK ADDRESS TRANSLATION Internet Router Server 172.23.208.1 200.100.10.1 Client
10. NETWORK ADDRESS TRANSLATION Internet Router Server 172.23.208.1 200.100.10.1 Source: 172.23.208.1 Destination: 200.100.10.1 Client
11. NETWORK ADDRESS TRANSLATION Internet Router Server 172.23.208.1 200.100.10.1 Source: 172.23.208.1 Destination: 200.100.10.1 Client
12. NETWORK ADDRESS TRANSLATION Internet Router Server 172.23.208.1 200.100.10.1 Source: 84.11.65.1 Destination: 200.100.10.1 NAT table 172.23.208.1 84.11.65.1 Client
13. NETWORK ADDRESS TRANSLATION Internet Router Server 172.23.208.1 200.100.10.1 Source: 84.11.65.1 Destination: 200.100.10.1 NAT table 172.23.208.1 84.11.65.1 Possible further source IP changes Client
14. HOW DO WE KNOW THE IP ADDRESS OF ACCEPTOR?
15. NETWORK 101 DNS 172.23.208.1 Server 200.100.10.1 DNS Server DNS records example.com 200.100.10.1 Client W hat’s the IP of exam ple.com ? 200.100.10.1
16. NETWORK 101 DNS 172.23.208.1 Server 200.100.10.1 DNS Server DNS records example.com 200.100.10.1 Client
17. WHY CAN’T WE SETUP A DNS RECORD FOR EVERY PEER?
18. STUN Router STUN Server 172.23.208.1 stun.l.google.com Client DISCOVERING PUBLIC IP
19. STUN Router STUN Server 172.23.208.1 stun.l.google.com Client DISCOVERING PUBLIC IP What is my public IP? Source: 172.23.208.1 Destination: 200.100.10.1
20. STUN Router STUN Server 172.23.208.1 stun.l.google.com Client DISCOVERING PUBLIC IP What is my public IP? Source: 84.11.65.1 Destination: 200.100.10.1 NAT table 172.23.208.1 84.11.65.1
21. STUN Router STUN Server 172.23.208.1 stun.l.google.com Client DISCOVERING PUBLIC IP Your public IP is: 84.11.65.1 NAT table 172.23.208.1 84.11.65.1
22. STUN Router STUN Server 172.23.208.1 stun.l.google.com Client DISCOVERING PUBLIC IP Your public IP is: 84.11.65.1 NAT table 172.23.208.1 84.11.65.1 Periodically send ping to keep the NAT table mapping unchanged.
23. NOT ALL FIREWALLS ENABLE DIRECT DEVICE-DEVICE CONNECTION
24. TURN Firewall 84.16.55.1 Client RELAYING MESSAGES OVER NATS/FIREWALLS TURN Server 54.23.201.1 Client
25. ICE NEGOTIATE THE CONNECTION CAPABILITIES
26. ICE NEGOTIATE WHICH CONNECTION TO USE const conn = new RTCPeerConnection({ iceServers: [ { urls: "stun:stunserver.example.com:3478", }, { urls: 'turn:turnserver.com:3478', credential: 'password', username: 'username' } ] })
27. CLIENT / SERVER PEER TO PEER Local network Local network “The cloud” Local network Local neatwork What about that part?
28. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4
29. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4 Where is svc-4.local?
30. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4 Where is svc-4.local? Where is svc-4.local? Where is svc-4.local? Where is svc-4.local? multicast
31. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4 svc-4.local = 192.168.0.4
32. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4 svc-4.local = 192.168.04 svc-4.local = 192.168.04 svc-4.local = 192.168.04 svc-4.local = 192.168.04 multicast
33. mDNS svc-1.local 192.168.0.1 svc-4.local 192.168.0.4 svc-3.local 192.168.0.3 svc-4.local 192.168.0.4 svc-5.local 192.168.0.5 svc-4.local 192.168.0.4 svc-2.local 192.168.0.2 svc-4.local 192.168.0.4 svc-4.local 192.168.0.4 svc-4.local = 192.168.0.4
34. mDNS const mdns = require('mdns') // advertise service svc-1 at port 9999 via TCP const service = mdns.createAdvertisement(mdns.tcp(), 9999, { name: 'svc-1' }) service.start() // discover services const browser = mdns.createBrowser(mdns.tcp()) browser.on('ready', () => browser.discover()) browser.on('update', (data) => { console.log(data); // { // interfaceIndex: 4, // name: svc-1', // networkInterface: 'en0', // type: {name: '', protocol: 'tcp', subtypes: []}, // replyDomain: 'local.', // fullname: 'svc-1._tcp.local.', // host: 'svc-1.local.', // port: 9999, // addresses: [ '10.1.1.50', 'fe80::21f:5bff:fecd:ce64' ] // } })
35. CLUSTERING
36. HOW TO BUILD A CLUSTER THAT COULD SPAN OVER >1K NODES USING DIFFERENT NETWORKS?
37. HYPARVIEW HYBRID PARTIAL VIEW FOR CLUSTER MEMBERSHIP
38. CONNECTING EVERYONE TO EACH OTHER DOESN’T SCALE
39. CONNECTING EVERYONE TO EACH OTHER DOESN’T SCALE BUT…
40. NAÏVE CONNECTIVITY ISSUES F A E B C D Connections limit: 4
41. NAÏVE CONNECTIVITY ISSUES F A E B C D Connections limit: 4 Can I join?
42. NAÏVE CONNECTIVITY ISSUES F A E B C D Connections limit: 4 Sorry, I’m at my limit.
43. SOLUTION: INTRODUCE PRIORITY CONNECTIONS
44. PRIORITY CONNECTIONS F A E B C D Connections limit: 4
45. PRIORITY CONNECTIONS F A E B C D Connections limit: 4 SUDO: let me join
46. PRIORITY CONNECTIONS F A E B C D Connections limit: 4 drop existing connection at random to free the pool
47. PRIORITY CONNECTIONS F A E B C D Connections limit: 4 establish new connection
48. F A E B C D Connections limit: 4 FWD(2) FWD(2) FWD(2) FORWARD NEW PEER INFOR TO OTHERS
49. F A E B C D Connections limit: 4 FWD(1) FWD(1) FWD(1) FWD(1) FWD(1) FWD(1) FORWARD NEW PEER INFOR TO OTHERS
50. NON PRIORITY CONNECTIONS F A E B C D Connections limit: 4 Can I join?
51. NON PRIORITY CONNECTIONS F A E B C D Connections limit: 4 Sorry, I’m at my limit.
52. NON PRIORITY CONNECTIONS F A E B C D Connections limit: 4
53. HYPARVIEW 101 A C B D passive view active view K J I H G F E L M N Active peers Passive peers
54. HOW TO GOSSIP MESSAGES EFFICIENTLY?
55. PLUMTREE EPIDEMIC BROADCAST TREES
56. PLUMTREE 101 A C B D K J I H G F E L M N
57. BUILDING A TREE A C B D K J I H G F E L M N
58. BUILDING A TREE A C B D K J I H G F E L M N
59. BUILDING A TREE A C B D K J I H G F E L M N
60. BUILDING A TREE A C B D K J I H G F E L M N NACK NACK
61. BUILDING A TREE A C B D K J I H G F E L M N
62. BUILDING A TREE A C B D K J I H G F E L M N
63. BUILDING A TREE A C B D K J I H G F E L M N
64. BUILDING A TREE A C B D K J I H G F E L M N NACK NACK
65. BUILDING A TREE A C B D K J I H G F E L M N
66. BUILDING A TREE A C B D K J I H G F E L M N
67. BUILDING A TREE A C B D K J I H G F E L M N NACK
68. BUILDING A TREE A C B D K J I H G F E L M N Eager peers Lazy peers
69. WHAT IF CONNECTION FAILS?
70. TREE REPAIR A C B D K J I H G F E L M N connection failure
71. TREE REPAIR A C B D K J I H G F E L M N Part of the gossip tree is disconnected from the rest
72. TREE REPAIR A C B D K J I H G F E L M N Once in a while send message to lazy peers about latest gossips ids (m1 ) IHAVE(m1 )
73. TREE REPAIR A C B D K J I H G F E L M N Message receiver awaits for m1
74. TREE REPAIR A C B D K J I H G F E L M N If m1 didn’t arrive before timeout, send graft back GRAFT(m1 )
75. TREE REPAIR A C B D K J I H G F E L M N Promote lazy peer to eager one
76. TREE REPAIR A C B D K J I H G F E L M N
77. TREE VS. MESH BROADCAST
78. SUMMARY
79. — Rainbow connections: https://www.youtube.com/watch?v=8_A1CkYfzoM — Hyparview: https://bartoszsypytkowski.com/hyparview/ — Conflict-Free Replicated Data Types: https://bartoszsypytkowski.com/tag/crdt/ — Partisan: https://github.com/lasp-lang/partisan — DotNext (Hyparview): https://github.com/dotnet/dotNext/ LINKS
80. THANK YOU