Intro to Building Decentralized (P2P) Applications - SFNode

Most people think peer-to-peer (P2P) networks are just for file sharing, but it turns out you can also build other types of P2P applications with advantages like enhanced privacy and security. We’ll walk through the process of building an increasingly complex P2P cloud file storage system (think Dropbox), and touch on the challenges you’d run into and some of their possible solutions. Topics include efficiently locating data within a large network and building a system where we can trust random people on the internet with our personal files.

Transcript

1. An Introduction to Building Decentralized (P2P) Applications
2. @dkbarns Cloud File Storage
3. Topics • What/Why - Decentralized systems • How - Decentralization problems + solutions • Project
4. @dkbarns CLIENT CENTRAL SERVER RESPONSE REQUEST Client/server model
5. @dkbarns Cloud storage providers (CSPs) CLIENT CENTRAL SERVER CLIENT CLIENT CLIENT
6. @dkbarns Centralization CLIENT CENTRAL SERVER CLIENT CLIENT CLIENT Central coordination
7. @dkbarns Centralization - Pros & Cons + Familiar development model
8. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. Centralization - Pros & Cons + Familiar development model
9. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. - Data security Centralization - Pros & Cons + Familiar development model
10. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. - Data security - Anonymity Centralization - Pros & Cons + Familiar development model
11. @dkbarns
12. @dkbarns What would a trust-less system look like?
13. @dkbarns Decentralized system! CENTRALIZED DECENTRALIZED
14. @dkbarns Centralized vs. decentralized systems No central authority
15. @dkbarns Centralized vs. decentralized systems No central authority client/server model peer-to-peer (P2P)
16. @dkbarns Centralized vs. decentralized systems No central authority client/server model peer-to-peer (P2P) clients & server(s) client-server nodes
17. @dkbarns Centralized vs. decentralized systems No central authority client/server model peer-to-peer (P2P) clients & servers client-server nodes Terminology - node, peer, device - personal device (laptop, etc.) - remote server (DO Droplet)
18. @dkbarns Basic cloud file storage system 1. Uploadfile 2. Delete local file 3. Download file
19. @dkbarns Basic cloud file storage system 1. Uploadfile 2. Delete local file 3. Download file 4. Peers-only decentralized
20. @dkbarns 2. Connect to HOST server 3. Send JSON STORE message + file Basic upload process OWNER HOST 1. Create TCP server … 4. Write local file copy FILE FILE COPY
21. @dkbarns Equal, autonomous node 2. Connect to HOST server 3. Send JSON STORE message + file Basic upload process OWNER HOST 1. Create TCP server … 4. Write local file copy FILE FILE COPY Equal, autonomous node
22. @dkbarns 2. Connect to HOST server 3. Send JSON STORE message + file Basic upload process OWNER HOST 1. Create TCP server … 4. Write local file copy FILE FILE COPY NODE1 NODE2
23. @dkbarns Basic upload process HOST OWNER FILE FILE COPY NODE2 NODE1 1. Create TCP server … 4. Write local file copy 2. Connect to HOST server 3. Send JSON STORE message + file
24. @dkbarns Basic upload process HOST OWNER FILE FILE COPY NODE2 NODE1 1. Create TCP server … 4. Write local file copy 2. Connect to HOST server 3. Send JSON STORE message + file CLIENT/SERVER == HOST/OWNER
25. 1. Create TCP server HOST
26. OWNER FILE 2. Create TCP server 3. Connect to HOST server 4. Send STORE message + data
27. OWNER FILE 2. Create TCP server 3. Connect to HOST server 4. Send STORE message + data
28. OWNER FILE 2. Create TCP server 3. Connect to HOST server 4. Send STORE message + data
29. 1. Create TCP server … 5. Stores file HOST
30. 1. Create TCP server … 5. Stores file HOST
31. @dkbarns for worst cloud storage system!
32. @dkbarns Highly likely to fail OWNER FILE HOST X - Device shutdowns - File modification, etc.
33. @dkbarns Highly likely to fail OWNER FILE HOST X - Device shutdowns - File modification, etc. - Expect failure - Self-healing
34. Credit - Tristan Slominski, Distribute All The Things
35. Credit - Tristan Slominski, Distribute All The Things
36. @dkbarns A better cloud storage system Ownership: How to keep files private On-demand: How to deal with uncertain uptime Data loss: How to ensure data integrity Incentives: How to encourage participation
37. @dkbarns A better cloud storage system Ownership: How to keep files private On-demand: How to deal with uncertain uptime Data loss: How to ensure data integrity Incentives: How to encourage participation
38. @dkbarns File privacy
39. @dkbarns Thinking about file privacy/ownership CLIENT CENTRAL SERVER CLIENT Authentication
40. @dkbarns Thinking about file privacy/ownership CLIENT CENTRAL SERVER NODE NODE NODE CLIENT Authentication Physical access => useless
41. @dkbarns Steps to secure file data 1. Encryption 2. File splitting aka “Sharding”
42. @dkbarns Encryption PRIVATE KEY Generate key Step 1:
43. @dkbarns Encryption PRIVATE KEY Generate key Step 1: Encryption Algorithm “Hello world!" “7sr2ygq7sa” Encrypt Step 2:
44. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt
45. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt - symmetric - client side
46. @dkbarns “7sr2ygq7sa” “Hello world!" Decryption FILE FILE
47. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt
48. @dkbarns FILE Split file Step 1: SHARDS File Splitting (“Sharding”)
49. @dkbarns Generate shard file name “ello w" FILE a15cv Split file Step 1: Step 2: Hashing Algorithm SHARDS File Splitting (“Sharding”)
50. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a a15cv 75cav 03p2i ii24a HOST 3. Send Upload with sharding
51. @dkbarns Manifest files FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a MANIFEST WRITE SHARD IDS TO MANIFEST
52. @dkbarns Manifests: Associating shards with a file shard id
53. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a a15cv 75cav 03p2i ii24a HOST 3. Send Upload with sharding MANIFEST WRITE SHARD IDS TO MANIFEST
54. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a 3. Send How to distribute shards? MANIFEST WRITE SHARD IDS TO MANIFEST ?
55. @dkbarns Use a master list? OWNER NODE LIST HOST a15cv 03p2i 75cav ii24a 75cav a15cv ii24a 03p2i
56. @dkbarns Use master lists? OWNER NODE LIST HOST NODE LIST a15cv 03p2i 75cav ii24a 75cav a15cv ii24a 03p2i
57. @dkbarns Where to store a given shard? OWNER SHARD HOST ?
58. @dkbarns How to retrieve given shard? OWNER SHARD HOST ?
59. @dkbarns Routing schemes www.google.com
60. @dkbarns Routing schemes www.google.com
61. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
62. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
63. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
64. @dkbarns Where to store a given shard? OWNER SHARD HOST ?
65. @dkbarns Where to store a given shard? OWNER SHARD HOST ? 75cav IDs
66. @dkbarns Where to store a given shard? OWNER SHARD HOST ? 75cav IDs i80l9 1plm1 Distribute data?
67. @dkbarns Creating a routing scheme 37.7N, 122.4W 40.7N,70.4W
68. @dkbarns Creating a routing scheme 40.7N,70.4W 37.7N, 122.4W
69. @dkbarns Routing scheme - geographic closeness 40.7N,70.4W 37.7N, 122.4W
70. @dkbarns Routing scheme - geographic closeness 40.7N,70.4W 37.7N, 122.4W
71. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1
72. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 Logical closeness - shard id’s “closeness” to a node’s id
73. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 XOR distance Logical closeness - shard id’s “closeness” to a node’s id
74. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 XOR distance 0 ^ 0 => 0 1 ^ 0 => 1 0 ^ 1 => 1 1 ^ 1 => 0 Logical closeness - shard id’s “closeness” to a node’s id 1) Are the bits different? 2) Is there one 1?
75. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 0 ^ 0 => 0 1 ^ 0 => 1 0 ^ 1 => 1 1 ^ 1 => 0 1) Are the bits different? 2) Is there one 1? 1001 1000 1 1110 1010 100 (4) XOR distance Logical closeness - shard id’s “closeness” to a node’s id
76. @dkbarns Kademlia Distributed Hash Table (DHT) • Node-to-node communication layer
77. @dkbarns Kademlia Distributed Hash Table (DHT) • Node-to-node communication layer Kadence https://github.com/kadence/kadence
78. @dkbarns Organizing “contacts” (other nodes) Node Routing Table Bucket Contact Contact Contact Contact Bucket Bucket Contact Contact ID k = contacts / bucket # of buckets = log(n/k) Credit - Xoro project
79. @dkbarns How does Kademlia work? - How are routing tables populated/updated? - How are values stored and located? - What happens if a node doesn’t respond?
80. @dkbarns How does Kademlia work? - How are routing tables populated/updated? - How are values stored and located? - What happens if a node doesn’t respond? THIS IS A BLACK BOX BLACK BOX COVERS RABBIT HOLE
81. @dkbarns Contact lists HOST CONTACT LIST BUCKET/ CONTACT LIST OWNER • Smaller lists: N => log(N) space
82. @dkbarns Secure, private P2P cloud file storage MANIFEST WRITE SHARD IDS TO MANIFEST 2. Encrypt 3. Shard 4. Distribute HOST CONTACT LIST a15cv 75cav 03p2i ii24a a15cv 75cav CONTACT LIST ii24a 03p2i 1. Upload FILE OWNER • Smaller lists: N => log(N) space • Flexible, distributed storage
83. @dkbarns Secure, private P2P cloud file storage MANIFEST WRITE SHARD IDS TO MANIFEST 2. Encrypt 3. Shard 4. Distribute HOST CONTACT LIST • Smaller lists: N => log(N) space • Flexible, distributed storage • Large files a15cv 75cav 03p2i ii24a a15cv 75cav CONTACT LIST ii24a 03p2i 1. Upload FILE OWNER
84. @dkbarns 1. File ownership 2. On-demand upload/download 3. No data loss 4. Incentives - Proofs of Retrievability - Mirroring / Patching - Crypto test net transactions
85. @dkbarns Team Dylan Flo Tannr
86. Demo: File upload and retrieval
87. Demo: File patching
88. @dkbarns Learn more layr-team.github.io/layr-project/
89. @dkbarns Kademlia links 1. Kademlia: A Peer-to-peer Information System Based on the XOR Metric (Slides) - Maymounkov, Mazi`eres - https://tams.informatik.uni-hamburg.de/lehre/2004ss/vorlesung/medientec hnik/material/kpres.pdf 2. Xoro P2P - Chen, Kurutz, Lee, - https://xorro-p2p.github.io/resources/ 3. Tristan Slominski, Distribute All The Things - https://docs.google.com/presentation/d/11qGZlPWu6vEAhA7p3qsQaQtW H7KofEC9dMeBFZ1gYeA/edit#slide=id.g1718cc2bc_03157
90. @dkbarns Looking for my next team! On the at ... - dylankb.github.io - @dkbarns - github.com/dylankb