Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to BitTorrent & WebTorrent

Intro to BitTorrent & WebTorrent

Learn about how BitTorrent and WebTorrent work!

WebTorrent website: http://webtorrent.io

B498d33041627b07726dc29c28f02df7?s=128

Feross Aboukhadijeh

November 21, 2014
Tweet

Transcript

  1. Intro to BitTorrent & WebTorrent by Feross Aboukhadijeh

  2. None
  3. BitTorrent

  4. client HTTP server

  5. GET /file.txt HTTP/1.1 HTTP request HTTP response HTTP/1.1 200 OK

    URL: http://example.com/file.txt
  6. client HTTP server

  7. client client HTTP server

  8. client client client HTTP server

  9. client client client client HTTP server

  10. client client client client client HTTP server

  11. client client client client client client HTTP server

  12. client client client client client client client HTTP server

  13. client client client client client client client client HTTP server

  14. client client client client client client client client client HTTP

    server
  15. client client client client client client client client client client

    HTTP server
  16. client client client client client client client client client client

    client HTTP server
  17. client client client client client client client client client client

    client client HTTP server
  18. client client client client client client client client client client

    client client client HTTP server
  19. client

  20. client client

  21. REQUEST 0 0 16384 BitTorrent request BitTorrent response PIECE 0

    0 <data> method piece index byte offset length method piece index byte offset piece data
  22. client client

  23. client client client

  24. client client client client

  25. client client client client client

  26. client client client client client client

  27. client client client client client client client

  28. client client client client client client client client

  29. client client client client client client client client client

  30. No central point of failure Fast downloads Efficient protocol Every

    downloader also uploads Decentralized
  31. Peer 1 Peer 2 Peer 3 0 1 2 3

    4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4
  32. Peer 1 Peer 2 Peer 3 0 1 2 3

    4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4
  33. Peer 1 Peer 2 Peer 3 0 1 2 3

    4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 1 3
  34. Peer 1 Peer 2 Peer 3 0 1 2 3

    4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 1 3 2 4
  35. Peer 1 Peer 2 Peer 3 0 1 2 3

    4 0 1 2 3 4 0 1 2 3 4 You 0 1 2 3 4 0 2 3 4 0 1 2
  36. 0 1 2 3 4 Client reassembles pieces

  37. 0 1 2 3 4 Client reassembles pieces

  38. Request same piece from multiple peers Tit-for-tat Help peers that

    help us Preempt slow peers Download rarest piece first Preserve torrent health Incentivize peers to trade with us
  39. Torrent file { name: 'My awesome file', pieces:[ '1f9c3f59beec079715ec53324bde8569e4a0b4eb', 'ec42307d4ce5557b5d3964c5ef55d354cf4a6ecc',

    '7bf1bcaf79d11fa5e0be06593c8faafc0c2ba2cf', '76d71c5b01526b23007f9e9929beafc5151e6511' ], announce: 'http://tracker.example.com' }
  40. Torrent file { name: 'My awesome file', pieces:[ '1f9c3f59beec079715ec53324bde8569e4a0b4eb', 'ec42307d4ce5557b5d3964c5ef55d354cf4a6ecc',

    '7bf1bcaf79d11fa5e0be06593c8faafc0c2ba2cf', '76d71c5b01526b23007f9e9929beafc5151e6511' ], announce: 'http://tracker.example.com' } Info Hash: d2474e86c95b19b8bcfdb92bc12c9d44667cfa36
  41. Cryptographic hashes • “One-way function” • Same input always produces

    same output • Given output, very very hard to find input
  42. Cryptographic hashes e88e8338c572a06e3c801b29f519df532b3e76f6 0a60bcba24797692efa8770d23df0a830d91cb35

  43. Tracker server { 'd2474e86c95b19b8bcfdb92bc12c9d44667cfa36': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ], '52164f03629fd1322636babb2c014b7dae582da4':

    [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ] }
  44. Decentralization HTTP Torrent files Magnet links Find URIs Search engines,

    friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
  45. Decentralization HTTP Torrent files Magnet links Find URIs Search engines,

    friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
  46. Decentralization HTTP Torrent files Magnet links Find URIs Search engines,

    friends Search engines, friends Search engines, friend Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Tabl File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
  47. Distributed Hash Table • Distributed key-value lookup • Efficient (Kademlia)

    • Every peer is a tracker server
  48. Distributed Hash Table 0 2160 230 Bucket size: K=4 (usually

    K=8 or K=16)
  49. • if new peer would make bucket K+1 • if

    our ID is in the bucket • split the bucket, & add the new peer • else • ping all peers in bucket • if some peer is dead • replace dead peer with new peer • else • throw away new peer Bucket size cannot surpass K
  50. Distributed Hash Table 0 2160 230 keep peer? YES

  51. Distributed Hash Table 0 2160 230

  52. Distributed Hash Table 0 2160 230

  53. Distributed Hash Table 0 2160 230 keep peer? NO

  54. Distributed Hash Table 0 2160 230

  55. Distributed Hash Table 0 2160 a node's table is densest

    around it's own id
  56. put(key, value) • if key is “near” our node ID

    • store the { key, value } • else • throw it away
  57. put(key, value) • if key is “near” our node ID

    • store the { key, value } • else • throw it away put( 'e88e8338c572a06e3c801b29f519df532b3e76f6', '12.34.56.78:3000' )
  58. DHT storage { 'd2474e86c95b19b8bcfdb92bc12c9d44667cfa36': [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ], 'd2474e86629fd1322636babb2c014b7dae582da4':

    [ '12.34.56.78:3000', '34.56.78.12:4000', '56.78.12.34:5000' ] }
  59. lookup(key) • pick K closest nodes, send them lookup(key) message

    • if they return new DHT nodes (that are closer) • send them lookup(key) message & recurse • else if they return peers (they are a tracker!) • we're done – connect to the peers!
  60. Distributed Hash Table 0 2160 a node's table is densest

    around it's own id
  61. Distributed Hash Table • Fast O(log n) lookup, where n

    is number of nodes • Over time, table fills with stable, long-running nodes
  62. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36

  63. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36 &dn=Leaves+of+Grass+by+Walt+Whitman.epub

  64. Magnet URI magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36 &dn=Leaves+of+Grass+by+Walt+Whitman.epub &tr=udp%3A%2F%2Ftracker.example.com%3A80 &tr=udp%3A%2F%2Ftracker.example2.com%3A80

  65. Decentralization HTTP Torrent files Magnet links Find URIs Search engines,

    friends Search engines, friends Search engines, friends Find peers DNS (Domain Name System) Tracker server DHT (Distributed Hash Table) File transfer Single HTTP server Many BitTorrent peers Many BitTorrent peers
  66. None
  67. None
  68. WebRTC Web Real Time Communications

  69. “A project to bring realtime communication (RTC) to the open

    web platform” - Justin Uberti @ Google I/O 2012
  70. INSTALL

  71. None
  72. None
  73. None
  74. bittorrent bittorrent bittorrent bittorrent

  75. bittorrent bittorrent bittorrent bittorrent bittorrent

  76. bittorrent bittorrent bittorrent bittorrent bittorrent

  77. bittorrent bittorrent bittorrent bittorrent hybrid hybrid bittorrent

  78. bittorrent bittorrent bittorrent bittorrent hybrid hybrid bittorrent

  79. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent bittorrent webtorrent

  80. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc)

    bittorrent webtorrent
  81. bittorrent webtorrent NOT POSSIBLE TCP/UDP WebRTC

  82. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc)

    bittorrent webtorrent IMPORTANT!
  83. let's build an app with webtorrent

  84. instant.io

  85. bittorrent bittorrent bittorrent bittorrent hybrid hybrid webtorrent webtorrent webtorrent (webrtc)

    bittorrent webtorrent IMPORTANT!
  86. Usage: webtorrent <options> <torrent> Download the torrent, given as: *

    magnet uri * http/https url to .torrent file * filesystem path to .torrent file * info hash (as hex string) Options: --airplay stream to Apple TV (AirPlay) --chromecast stream to Chromecast --vlc stream in VLC --mplayer stream in MPlayer --mpv stream in MPV --omx [jack] stream in omx (jack=local|hdmi) -p, --port [number] change the http port [default: 9000] -b, --blocklist [path] use the specified blocklist -t, --subtitles [file] load subtitles file -l, --list list available files in torrent -n, --no-quit do not quit webtorrent on vlc exit -r, --remove remove downloaded files on exit -q, --quiet silence stdout
  87. webtorrent "magnet uri" --vlc

  88. same code in node and the browser?

  89. webtorrent bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension torrent-discovery

    create-torrent ut_pex parse-torrent-file magnet-uri
  90. webtorrent bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension torrent-discovery

    create-torrent ut_pex parse-torrent-file magnet-uri
  91. "browser" field "browser": { "bittorrent-dht": false, "bittorrent-tracker": "webtorrent-tracker", "bittorrent-swarm": "webtorrent-swarm"

    }
  92. tcp/udp is simple 12.34.56.78:1234

  93. getUserMedia()

  94. getUserMedia() <stream>

  95. new RTCPeerConnection(constraints, config) getUserMedia() <stream>

  96. new RTCPeerConnection(constraints, config) pc getUserMedia() <stream>

  97. new RTCPeerConnection(constraints, config) pc getUserMedia() addStream() <stream>

  98. new RTCPeerConnection(constraints, config) pc getUserMedia() addStream() <stream> “negotiationneeded”

  99. new RTCPeerConnection(constraints, config) pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded”

  100. new RTCPeerConnection(constraints, config) <offer> pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded”

  101. new RTCPeerConnection(constraints, config) <offer> pc createOffer() getUserMedia() addStream() <stream> “negotiationneeded”

    setLocalDescription()
  102. new RTCPeerConnection(constraints, config) <offer> pc Server createOffer() getUserMedia() addStream() <stream>

    “negotiationneeded” setLocalDescription()
  103. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server createOffer() getUserMedia() addStream()

    <stream> “negotiationneeded” setLocalDescription()
  104. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server new RTCPeerConnection(…) createOffer()

    getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
  105. new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new RTCPeerConnection(…)

    createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
  106. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new

    RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> “negotiationneeded” setLocalDescription()
  107. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new

    RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() “negotiationneeded” setLocalDescription()
  108. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> pc Server pc new

    RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() <stream> “negotiationneeded” setLocalDescription()
  109. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  110. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() createAnswer() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  111. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  112. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  113. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  114. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  115. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() pc Server pc

    new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  116. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc

    Server pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  117. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc

    Server ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  118. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc

    Server <candidate> ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  119. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> pc

    Server <candidate> addIceCandidate() ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  120. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  121. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  122. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  123. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded”
  124. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream”
  125. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream”
  126. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream” “addstream”
  127. setRemoteDescription() new RTCPeerConnection(constraints, config) <offer> <offer> setLocalDescription() “icecandidate” <candidate> <candidate>

    “icecandidate” pc Server <candidate> addIceCandidate() <candidate> ⋮ ⋮ pc new RTCPeerConnection(…) createOffer() setLocalDescription() createAnswer() <answer> <answer> setRemoteDescription() addIceCandidate() getUserMedia() addStream() <stream> getUserMedia() addStream() <stream> “negotiationneeded” “addstream” “addstream”
  128. pc pc createDataChannel() <channel>

  129. pc pc createDataChannel() <channel> “datachannel” <channel>

  130. pc pc createDataChannel() <channel> “datachannel” <channel> send() <secret message>

  131. pc pc createDataChannel() <channel> “datachannel” <channel> send() <secret message> “message”

    <secret message>
  132. TUTORIALS, HELP, MORE… FALLBACK TO HTTP DOCUMENTATION BUILD WEBTORRENT APPS

    CALLING MAD SCIENTISTS! WEB SEEDS
  133. THIS IS STILL UNIMPLEMENTED JS EVERYWHERE WEBRTC DHT NATIVE APP

    NODE-WEBKIT CHROME/FIREFOX APPS
  134. webtorrent http://webtorrent.io

  135. Peer-to-peer Wikipedia torrent-mount Mount torrents as a filesystem peerwiki http://github.com/mafintosh

  136. feross aboukhadijeh webtorrent.io @feross happy hacking!