Intro to Building Decentralized Applications - Fog City Ruby

Most people think peer-to-peer (P2P) networks are just for file sharing, but it turns out you can also build other types of P2P applications with advantages like enhanced privacy and security. We’ll walk through the process of building an increasingly complex P2P cloud file storage system (think Dropbox), and touch on the challenges you’d run into and some of their possible solutions. Topics include efficiently locating data within a large network and building a system where we can trust random people on the internet with our personal files.

Transcript

1. An Introduction to Building Decentralized (P2P) Applications
2. @dkbarns Cloud File Storage
3. @dkbarns CLIENT CENTRAL SERVER Client/server model RESPONSE REQUEST
4. @dkbarns Cloud storage providers (CSPs) CLIENT CENTRAL SERVER CLIENT CLIENT CLIENT
5. @dkbarns Centralization CLIENT CENTRAL SERVER CLIENT CLIENT CLIENT Central coordination
6. @dkbarns Pros & Cons + Familiar development model
7. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. Pros & Cons + Familiar development model
8. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. - Data security Pros & Cons + Familiar development model
9. @dkbarns - Data privacy: “Technically someone could do that ...” - Markie Z. - Data security - Anonymity Pros & Cons + Familiar development model
10. @dkbarns
11. @dkbarns Centralized vs. decentralized systems No central authority client/server model
12. @dkbarns Centralized vs. decentralized systems No central authority client/server model client-server nodes
13. @dkbarns Centralized vs. decentralized systems Terminology - node, peer, device (laptop, Digital Ocean Droplet) No central authority client/server model client-server nodes
14. @dkbarns Basic cloud file storage 1. Uploadfile 2. Delete local file 3. Download file
15. @dkbarns 2. Connect to HOST server 3. Send JSON STORE message + file Basic upload process OWNER HOST 1. Create TCP server … 4. Write local file copy FILE FILE COPY
16. @dkbarns 1. Create TCP server HOST
17. @dkbarns OWNER FILE 2. Connect to HOST server 3. Send STORE message + data
18. @dkbarns OWNER FILE 2. Connect to HOST server 3. Send STORE message + data
19. @dkbarns 1. Create TCP server … 4. Write local file copy HOST
20. @dkbarns for worst cloud storage system!
21. @dkbarns A better cloud storage system Ownership: How to keep files private On-demand: How to deal with uncertain uptime Data loss: How to ensure data integrity Incentives: How to encourage participation
22. @dkbarns A better cloud storage system Ownership: How to keep files private On-demand: How to deal with uncertain uptime Data loss: How to ensure data integrity Incentives: How to encourage participation
23. @dkbarns File privacy
24. @dkbarns Steps to secure file data 1. Encryption 2. File splitting aka “Sharding”
25. @dkbarns Encryption PRIVATE KEY Generate key Step 1:
26. @dkbarns Encryption PRIVATE KEY Generate key Step 1: Encryption Algorithm “Hello world!" “7sr2ygq7sa” Encrypt Step 2:
27. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt
28. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt - symmetric - client side
29. @dkbarns “7sr2ygq7sa” “Hello world!" Decryption FILE FILE
30. @dkbarns Secure upload process FILE OWNER 2. Send HOST 1. Encrypt
31. @dkbarns FILE Split file Step 1: SHARDS File Splitting (“Sharding”)
32. @dkbarns Generate shard file name “ello w" FILE a15cv Split file Step 1: Step 2: Hashing Algorithm SHARDS File Splitting (“Sharding”)
33. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a a15cv 75cav 03p2i ii24a HOST 3. Send Upload with sharding
34. @dkbarns Manifest files FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a MANIFEST WRITE SHARD IDS TO MANIFEST
35. @dkbarns Manifests: Associating shards with a file shard id
36. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a a15cv 75cav 03p2i ii24a HOST 3. Send Upload with sharding MANIFEST WRITE SHARD IDS TO MANIFEST
37. @dkbarns FILE OWNER 1. Encrypt 2. Shard a15cv 75cav 03p2i ii24a 3. Send How to distribute shards? MANIFEST WRITE SHARD IDS TO MANIFEST ?
38. @dkbarns Use a master list? OWNER NODE LIST HOST a15cv 03p2i 75cav ii24a 75cav a15cv ii24a 03p2i
39. @dkbarns Use master lists? OWNER NODE LIST HOST NODE LIST a15cv 03p2i 75cav ii24a 75cav a15cv ii24a 03p2i
40. @dkbarns Where to store a given shard? OWNER SHARD HOST ?
41. @dkbarns How to retrieve given shard? OWNER SHARD HOST ?
42. @dkbarns Routing schemes www.google.com
43. @dkbarns Routing schemes www.google.com
44. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
45. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
46. @dkbarns Routing schemes www.google.com DNS! 172.217.164.110
47. @dkbarns Where to store a given shard? OWNER SHARD HOST ?
48. @dkbarns Where to store a given shard? OWNER SHARD HOST ? 75cav IDs
49. @dkbarns Where to store a given shard? OWNER SHARD HOST ? 75cav IDs i80l9 1plm1 Distribute data?
50. @dkbarns Creating a routing scheme
51. @dkbarns Routing scheme - geographic closeness
52. @dkbarns Routing scheme - geographic closeness
53. @dkbarns Routing scheme - geographic closeness 37.7N, 122.4W 40.7N,70.4W
54. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1
55. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 Based on the shard id Logical closeness - shard id’s “closeness” to other node’s id
56. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 Based on the shard id Logical closeness - shard id’s “closeness” to other node’s id THIS IS A BLACK BOX
57. @dkbarns What is our closeness metric? OWNER SHARD HOST ? 75cav i80l9 1plm1 Based on the shard id Logical closeness - shard id’s “closeness” to other node’s id THIS IS A BLACK BOX BLACK BOX COVERS RABBIT HOLE
58. @dkbarns Kademlia Distributed Hash Table (DHT) • Node-to-node communication layer
59. @dkbarns Kademlia Distributed Hash Table (DHT) • Node-to-node communication layer Kadence https://github.com/kadence/kadence
60. @dkbarns Contact lists HOST CONTACT LIST • Smaller routing lists CONTACT LIST OWNER
61. @dkbarns Secure, private P2P cloud file storage MANIFEST WRITE SHARD IDS TO MANIFEST 2. Encrypt 3. Shard 4. Distribute HOST CONTACT LIST • Smaller routing lists • Smaller storage a15cv 75cav 03p2i ii24a a15cv 75cav CONTACT LIST ii24a 03p2i 1. Upload FILE OWNER
62. @dkbarns Secure, private P2P cloud file storage MANIFEST WRITE SHARD IDS TO MANIFEST 2. Encrypt 3. Shard 4. Distribute HOST CONTACT LIST • Smaller routing lists • Smaller storage • Large files a15cv 75cav 03p2i ii24a a15cv 75cav CONTACT LIST ii24a 03p2i 1. Upload FILE OWNER
63. Demo: File upload and retrieval
64. Demo: File patching
65. @dkbarns Learn more layr-team.github.io/layr-project/
66. @dkbarns Kademlia links 1. Kademlia: A Peer-to-peer Information System Based on the XOR Metric (Slides) - Maymounkov, Mazi`eres - https://tams.informatik.uni-hamburg.de/lehre/2004ss/vorlesung/medientec hnik/material/kpres.pdf 2. Xoro P2P - Chen, Kurutz, Lee, - https://xorro-p2p.github.io/resources/ 3. Tristan Slominski, Distribute All The Things - https://docs.google.com/presentation/d/11qGZlPWu6vEAhA7p3qsQaQtW H7KofEC9dMeBFZ1gYeA/edit#slide=id.g1718cc2bc_03157
67. @dkbarns Looking for my next team! On the at ... - dylankb.github.io - @dkbarns - github.com/dylankb