Intro to BitTorrent & WebTorrent

Learn about how BitTorrent and WebTorrent work!

WebTorrent website: http://webtorrent.io

Transcript

1. Intro to BitTorrent & WebTorrent
   by Feross Aboukhadijeh
3. BitTorrent
4. client HTTP server
5. GET /file.txt HTTP/1.1 HTTP request HTTP response HTTP/1.1 200 OK URL: http://example.com/file.txt
6. client HTTP server
7. client client HTTP server
8. client client client HTTP server
9. client client client client HTTP server
10. client client client client client HTTP server
11. client client client client client client HTTP server
12. client client client client client client client HTTP server
13. client client client client client client client client HTTP server
14. client client client client client client client client client HTTP server
15. client client client client client client client client client client HTTP server
16. client client client client client client client client client client client HTTP server
17. client client client client client client client client client client client client HTTP server
18. client client client client client client client client client client client client client HTTP server
19. client
20. client client
21. REQUEST 0 0 16384 BitTorrent request BitTorrent response PIECE 0 0 <data> method piece index byte offset length method piece index byte offset piece data
22. client client
23. client client client
24. client client client client
25. client client client client client
26. client client client client client client
27. client client client client client client client
28. client client client client client client client client
29. client client client client client client client client client
30. No central point of failure Fast downloads Efficient protocol Every downloader also uploads Decentralized
31. Peer 1 Peer 2 Peer 3 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4
32. Peer 1 Peer 2 Peer 3 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4
33. Peer 1 Peer 2 Peer 3 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 1 3
34. Peer 1 Peer 2 Peer 3 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 1 3 2 4
35. Peer 1 Peer 2 Peer 3 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 2 3 4 0 1 2
36. 0 1 2 3 4 Client reassembles pieces
37. 0 1 2 3 4 Client reassembles pieces
38. Request same piece from multiple peers Tit-for-tat Help peers that help us Preempt slow peers Download rarest piece first Preserve torrent health Incentivize peers to trade with us
39. Torrent file { name: 'My awesome file', pieces:[ '1f9c3f59beec079715ec53324bde8569e4a0b4eb', 'ec42307d4ce5557b5d3964c5ef55d354cf4a6ecc', '7bf1bcaf79d11fa5e0be06593c8faafc0c2ba2cf', '76d71c5b01526b23007f9e9929beafc5151e6511' ], announce: 'http://tracker.example.com' }
40. Torrent file { name: 'My awesome file', pieces:[ '1f9c3f59beec079715ec53324bde8569e4a0b4eb', 'ec42307d4ce5557b5d3964c5ef55d354cf4a6ecc', '7bf1bcaf79d11fa5e0be06593c8faafc0c2ba2cf', '76d71c5b01526b23007f9e9929beafc5151e6511' ], announce: 'http://tracker.example.com' } Info Hash: d2474e86c95b19b8bcfdb92bc12c9d44667cfa36
41. Cryptographic hashes • “One-way function” • Same input always produces same output • Given output, very very hard to find input
42. Cryptographic hashes e88e8338c572a06e3c801b29f519df532b3e76f6 0a60bcba24797692efa8770d23df0a830d91cb35
43. Tracker server { 'd2474e86c95b19b8bcfdb92bc12c9d44667cfa36': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ], '52164f03629fd1322636babb2c014b7dae582da4': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ] }
44. Decentralization HTTP Torrent files Magnet links Find URIs Search engines, friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
45. Decentralization HTTP Torrent files Magnet links Find URIs Search engines, friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
46. Decentralization HTTP Torrent files Magnet links Find URIs Search engines, friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
47. Distributed Hash Table • Distributed key-value lookup • Efficient (Kademlia) • Every peer is a tracker server
48. Distributed Hash Table 0 2160 230 Bucket size: K=4 (usually K=8 or K=16)
49. • if new peer would make bucket K+1 • if our ID is in the bucket • split the bucket, & add the new peer • else • ping all peers in bucket • if some peer is dead • replace dead peer with new peer • else • throw away new peer Bucket size cannot surpass K
50. Distributed Hash Table 0 2160 230 keep peer? YES
51. Distributed Hash Table 0 2160 230
52. Distributed Hash Table 0 2160 230
53. Distributed Hash Table 0 2160 230 keep peer? NO
54. Distributed Hash Table 0 2160 230
55. Distributed Hash Table 0 2160 a node's table is densest around it's own id
56. put(key, value) • if key is “near” our node ID • store the { key, value } • else • throw it away
57. put(key, value) • if key is “near” our node ID • store the { key, value } • else • throw it away put( 'e88e8338c572a06e3c801b29f519df532b3e76f6', '12.34.56.78:3000' )
58. DHT storage { 'd2474e86c95b19b8bcfdb92bc12c9d44667cfa36': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ], 'd2474e86629fd1322636babb2c014b7dae582da4': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ] }
59. lookup(key) • pick K closest nodes, send them lookup(key) message • if they return new DHT nodes (that are closer) • send them lookup(key) message & recurse • else if they return peers (they are a tracker!) • we're done – connect to the peers!
60. Distributed Hash Table 0 2160 a node's table is densest around it's own id
61. Distributed Hash Table • Fast O(log n) lookup, where n is number of nodes • Over time, table fills with stable, long-running nodes
62. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36
63. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36 &dn=Leaves+of+Grass+by+Walt+Whitman.epub
64. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36 &dn=Leaves+of+Grass+by+Walt+Whitman.epub &tr=udp%3A%2F%2Ftracker.example.com%3A80 &tr=udp%3A%2F%2Ftracker.example2.com%3A80
65. Decentralization HTTP Torrent files Magnet links Find URIs Search engines, friends Search engines, friends Search engines, friends Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Table) File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
68. WebRTC
    Web Real Time Communications
69. “A project to bring realtime communication (RTC) to the open web platform” - Justin Uberti @ Google I/O 2012
70. INSTALL
74. bittorrent bittorrent bittorrent bittorrent
75. bittorrent bittorrent bittorrent bittorrent bittorrent
76. bittorrent bittorrent bittorrent bittorrent bittorrent
77. bittorrent bittorrent bittorrent bittorrent hybrid hybrid bittorrent
78. bittorrent bittorrent bittorrent bittorrent hybrid hybrid bittorrent
79. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent bittorrent webtorrent
80. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc) bittorrent webtorrent
81. bittorrent webtorrent NOT POSSIBLE TCP/UDP WebRTC
82. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc) bittorrent webtorrent IMPORTANT!
83. let's build an app with webtorrent
84. instant.io
85. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc) bittorrent webtorrent IMPORTANT!
86. Usage: webtorrent <options> <torrent> Download the torrent, given as: * magnet uri * http/https url to .torrent file * filesystem path to .torrent file * info hash (as hex string) Options: --airplay stream to Apple TV (AirPlay) --chromecast stream to Chromecast --vlc stream in VLC --mplayer stream in MPlayer --mpv stream in MPV --omx [jack] stream in omx (jack=local|hdmi) -p, --port [number] change the http port [default: 9000] -b, --blocklist [path] use the specified blocklist -t, --subtitles [file] load subtitles file -l, --list list available files in torrent -n, --no-quit do not quit webtorrent on vlc exit -r, --remove remove downloaded files on exit -q, --quiet silence stdout
87. webtorrent "magnet uri" --vlc
88. same code in node and the browser?
89. webtorrent bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension torrent-discovery create-torrent ut_pex parse-torrent-file magnet-uri
90. webtorrent bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension torrent-discovery create-torrent ut_pex parse-torrent-file magnet-uri
91. "browser" field "browser": { "bittorrent-dht": false, "bittorrent-tracker": "webtorrent-tracker", "bittorrent-swarm": "webtorrent-swarm" }
92. tcp/udp is simple 12.34.56.78:1234
93. getUserMedia()
94. getUserMedia() <stream>
95. new RTCPeerConnection(constraints, config) getUserMedia() <stream>
96. new RTCPeerConnection(constraints, config) pc getUserMedia() <stream>
97. new RTCPeerConnection(constraints, config) pc getUserMedia() addStream() <stream>
98. new RTCPeerConnection(constraints, config) pc getUserMedia() addStream() <stream> “negotiationneeded”
99. new RTCPeerConnection(constraints, config) pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded”
100. new RTCPeerConnection(constraints, config) <offer> pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded”
101. new RTCPeerConnection(constraints, config) <offer> pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
102. new RTCPeerConnection(constraints, config) <offer> pc Server createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
103. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
104. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
105. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
106. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
107. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() “negotiationneeded” setLocalDescription()
108. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() <stream> “negotiationneeded” setLocalDescription()
109. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
110. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() createAnswer() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
111. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
112. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
113. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
114. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
115. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
116. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
117. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc Server ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
118. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc Server <candidate> ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
119. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc Server <candidate> addIceCandidate() ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
120. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
121. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
122. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
123. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
124. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream”
125. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream”
126. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream” “addstream”
127. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate> “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream” “addstream”
128. pc pc createDataChannel() <channel>
129. pc pc createDataChannel() <channel> “datachannel” <channel>
130. pc pc createDataChannel() <channel> “datachannel” <channel> send() <secret message>
131. pc pc createDataChannel() <channel> “datachannel” <channel> send() <secret message> “message” <secret message>
132. TUTORIALS, HELP, MORE… FALLBACK TO HTTP DOCUMENTATION BUILD WEBTORRENT APPS CALLING MAD SCIENTISTS! WEB SEEDS
133. THIS IS STILL UNIMPLEMENTED JS EVERYWHERE WEBRTC DHT NATIVE APP NODE-WEBKIT CHROME/FIREFOX APPS
134. webtorrent http://webtorrent.io
135. Peer-to-peer Wikipedia torrent-mount Mount torrents as a filesystem peerwiki http://github.com/mafintosh
136. feross aboukhadijeh webtorrent.io @feross happy hacking!