WebTorrent: Bringing BitTorrent to the Web (with WebRTC and Mad Science)

Feross Aboukhadijeh talking at CraftConf in Budapest, Hungary on April 25, 2013.

About the talk
--------------------

WebRTC changes everything! For the first time, we can do peer-to-peer, or client-to-client, communication in the browser without plugins, extensions, or a software installation. This enables an entirely new class of applications to be built for the web! This talk will look at WebTorrent, a BitTorrent client for the browser that fully-interoperates with the regular BitTorrent network. WebTorrent uses WebRTC Data Channels and special "hybrid clients" to connect to the wider BitTorrent network. By making BitTorrent easier, we're making it accessible to new swathes of users who were previously intimidated, confused, or unwilling to install a program on their computer to participate.

About Feross
-------------------

Feross Aboukhadijeh is a programmer, designer, teacher, and mad scientist. He is currently building WebTorrent, a streaming BitTorrent client for the browser, powered by WebRTC. Before that, he built PeerCDN, a peer-to-peer content delivery network that dramatically reduces bandwidth costs. Feross is a graduate of Stanford University and has worked at Quora, Facebook, and Intel. In the past, he did research in the Stanford human-computer interaction and computer security labs. Feross enjoys working on "mad science" -- projects that make people say, "Whoa! I didn't know that was possible!". Feross frequently wins hackathons by crafting code that gives the kids with spectacles spectacular fits, like his recent virtual reality piloting hack that lets you control a quadcopter with an Oculus Rift.

Links
--------

Feross Aboukhadijeh (http://feross.org)
CraftConf (http://craft-conf.com/2014/)

Transcript

1. WebTorrent
   Bringing BitTorrent to the Web
3. Topics •How does WebTorrent work? •How to craft awesome software using: 1. npm 2. Browserify 3. WebRTC
13. Virus Scan Results •MacOS:GoPhoto-A [Adw] •Trojan.Crossrider.9294 •RTF:Malware.OddRTF/Heur!1.9E6F
18. http://mg8.org/processing/bt.html
20. Improve Bittorrent UX •Streaming video playback •Works in the browser •Easy to share links •No installation required •Elegant search experience
21. bittorrent bittorrent bittorrent bittorrent
22. uTorrent Transmission Azureus etc.
23. bittorrent bittorrent bittorrent bittorrent
24. bittorrent bittorrent bittorrent bittorrent bittorrent
25. bittorrent bittorrent bittorrent bittorrent webtorrent webtorrent bittorrent
26. bittorrent bittorrent bittorrent bittorrent webtorrent webtorrent bittorrent
27. bittorrent bittorrent bittorrent bittorrent webtorrent webtorrent webtorrent.js webtorrent.js bittorrent webtorrent.js
28. bittorrent bittorrent bittorrent bittorrent webtorrent webtorrent webtorrent.js webtorrent.js webtorrent (webrtc) bittorrent webtorrent.js
29. bittorrent webtorrent.js NOT POSSIBLE
30. bittorrent bittorrent bittorrent bittorrent webtorrent webtorrent webtorrent.js webtorrent.js webtorrent (webrtc) bittorrent webtorrent.js IMPORTANT!
31. webtorrent v1 (chrome app)
32. webtorrent v2
33. The plan 1. Build awesome JavaScript bittorrent client for OS X, Windows, Linux 2. Make node.js code run in the browser! 3. Use WebRTC to create peer-to-peer connections.
34. 1 npm
35. webtorrent.app! (os x, windows, linux) webtorrent.js! (browser js library) webtorrent-chrome! (chrome app)
36. webtorrent.app! (os x, windows, linux) webtorrent.js! (browser js library) webtorrent-chrome! (chrome app)
37. bittorrent-client! (stream from bittorrent) webtorrent! (http range server, cmd line) webtorrent-client! (stream from webtorrent) webtorrent.app! (os x, windows, linux) webtorrent.js! (browser js library) webtorrent-chrome! (chrome app) bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension
38. the way™
39. Do one thing well.
40. Compose larger units out of smaller, independent units of functionality.
41. "When applications are done well, they are just the really application-specific, brackish residue that can't be so easily abstracted away. ! All the nice, reusable components sublimate away onto github and npm where everybody can collaborate to advance the commons.” ! — substack
42. airport amplitude-viewer ansi-keycode ap archy array-map array-reduce astw asynth attr-bind attr-chooser attr-ev attr-range attr-scope attr-submit attractor autorev bashful battleship-search baudio baudio-party ben bigint binary bouncy brake brfs browser-badge browser-launcher browser-pack browser-resolve browser-terminal browser-unix browser-unpack browserify buffer-equal bufferlist buffers bulk-require bulkify bunker burrito c-tokenizer camelize caps-lock catch-links catw cert- unbundle chainsaw char-size charm chdir chi-squared choose chunky cicada cipherhub cities1000 clocker code-art code-music-studio codebux coffeeify comandante commondir concat-map confuse console-stream continued-fraction cookie-cutter coords core-http-benchmark coverify covert crytter css-prefix cursory cypher-feed de- bruijn deck decode-prompt deep-equal deep-freeze defined deps-sort deputy destroyer detective difflet disorder dnode dnode-protocol dnode-stack dotc duplex-pipe earl-grey editor editor-timeline emit-stream endian-toggle energy-relay ent ever exportify extents exterminate factor-bundle falafel faucet figc fileify findit fleet foreign-key fort freestyle frequency-viewer fritter funstance gamma garbage gcd git-emit git-file git-history git-http-backend github-avatar github-from-package github-push-receive glog graph-stream grave gray-code gtest gutter hacker-deps hash-join hashish hat heatmap http-alt http-browserify http-duplex http-noupgrade http-raw https-browserify https-detect hugh-detector humbug hyperglue hyperkey hyperquest hyperspace hyperstream identifier insert-css insert-module-globals internet-timestamp intestine intreq invoicer is-triangle isaacs jadeify join-stream jquery-mousewheel json-literal-parse json-scrape json-stable-stringify jsonify jsonparse jsup key-range keysym keyx level-assoc level-join level-party level-proxy level-query level-track lexical-scope lexicographic-integer liver logdir lump marak markdown-directory markov marx meteor mine minimist mkdirp mmmify module-deps mountie mrcolor multilevel-feed multimeter n-pair normalize-browser-names npm-package-search npm-package-sync npmdep npmtop nub number-grouper number-script object-inspect oppressor optimist ordered-emitter parcel-detector parcel- finder parcel-map parcel-processor parcelify parents parse-color parsley password-reset path-browserify pathway permafrost persona-id picture-tube pier pipestop pkginit placename ploy plucky point-grouper point-in-polygon polygon-hash pony prehost pricing-widget progressify prompter propagit provinces pushover put pw py quack-array quote-stream quotemeta rap-battle recon relaychum render-assoc replicant resolve response-stream resumer resware rfb rhyme rolling-hash rolling-reduce rref rsa-json rsa-stream rsa-unpack running-mean safe-regex schoolbus scoper scrux seaport secure-peer seq seqsolve sesame shadow-npm shallow-copy shell-quote shimify shoe shortcode shp2json shux sigsolve sillyscope single-page slice-file slideways slidey-termy smtp-protocol sockjs-windows song sorta sorted source split-by split-css ssh stackedy stagecoach static-eval static-module still-alive straggler stream-adventure stream-browserify string_decoder subarg subdeps subdir surrender svg-morph svg-pencil svg-select switch-stream swoop syntax-error tabby tap-finished tap-parser tape terminal-menu testling testling-jasmine testling-server text- table throughout tilemap time-window-stream toss tracer-bullet transfuse traverse travisify trumpet tty-browserify tuner tuple-stream typedarray undead undirender upnode uppercase.c utf8-bytes utf8-length utf8-stream vectorwave vm-browserify voicebox-karaoke voxel-chunks voxel-creature voxel-debris voxel- forest voxel-json voxel-player voxel-portal voxel-spider waiting-around waitlist watchify webworkify whatever wit wordwrap wreq x256 xhr-write-stream xit xrandr- parse yarnify zygote ! — substack
43. add-deps adiff all amd another-progress-bar ansi-highlight ansi-recover assertions async-chain async-iterator asynct autonode badass bash-vars battery bed between bikeshed binomial-hash-list blake2s bracket-matcher browser-stream bundle-metadata bv canvas-browserify canvas-progress-bar carpenter center client-reloader config-chain connect-restreamer console-log content-addressable-store continuable-para continuable-series convoy-stream couch-stream couch-sync couchfeed couchlegs count-tabs cradle-init crdt create-systemd-service crypto-browserify cryptomx ctrlflow ctrlflow_tests curry cyphernet d-utils d64 dart-stream dat-table default-pager delay-stream demonstrate deploy deterministic-tar dom-vector duplex duplexer easyfs el-streamo element-splice eohbjcobpchokp event-stream event- tree f-core feedopensource from fs-reverse fsm gen get-deps git-deploy github-url goatdb graphlib-adjacency graphlib-git h hash-change hdiff header-stream himark hipster how-big hud hyperscript idle indexes-of indexhtmlify insert-queue invert-stream it-is iterate jrep js-tokenizer json-buffer json-logdb json-rest json-select json-sst JSON.sh JSONStream keep kiddb kv lamport level-binomial-replication level-content-addressable-store level-content-cache level-couch-sync level-fix-range level-hooks level-index level-inverted-index level-live-stream level-manifest level-map level-map-merge level-map-tile level-map-tiles level-master level-merkle level- peek level-post level-queue level-reduce level-replicate level-repred level-scuttlebutt level-search level-static level-sublevel level-test level-trigger level-update level- update-stream level-view-stream level-window line-graph lm localstorage-scuttlebutt lock looper ls-modules ls-r lu macgyver map-reduce map-stream map-tile-url markdown-sections math-buffer matrix merkle-stream message-stream meta-test mkdir-and-mv-stream modal-widget modem-stream mokuai-map monotonic- timestamp mpg123 msgpack-stream mux-demux mw-pipes my-local-ip nih-op ninja-tools nodejitsu-client nonsense npm-atomic-publish npm-github-resolve-url npm- remapper npm-search npmd npmd-bin npmd-cache npmd-config npmd-git-resolve npmd-install npmd-leaves npmd-link npmd-pack npmd-rebuild npmd-resolve npmd- tree npmd-unpack ntk observable padded-semver parentchild parse-regexp parse-table pause-stream peers per-second poset probe-stream proxy-by-url proxy80 ps- tree pull-2step-replicate pull-cat pull-core pull-crypto pull-csv pull-flow pull-fs pull-glob pull-handshake pull-level pull-markable pull-merge pull-merkle pull-pair pull- paramap pull-pushable pull-server pull-sorted-merge pull-split pull-stream pull-stream-merge pull-stream-range pull-stream-to-stream pull-stringify pull-switch pull-tee pull-through pull-traverse pull-utf8-decoder pull-window pull-zip query-stream quickansi r-array r-edit r-value random-name range-bucket rc readme rec2 reconnect reconnect-widget reconnector redis-protocol-stream redis-raw regular-stream relational-join-stream remap remember remote-events render repl-stream replace-stream replicated-reduce repred retro rpc-stream rpc-with-streams rumours runloop scuttlebucket scuttlebutt scuttlebutt-remote scuttlebutt-schema search- context securify seneca-level-store seneca-pay seneca-rumours-store seneca-stripe-pay sha.js sha1sum sha256d shadow-npm shasum signed-http skates smoothie smoothie-stream snob sort-stream sortable split spork ssh ssh-key-to-pem stability statistics stream-combiner stream-serializer stream-spec stream-tester stream- to-pull-stream string-range strm style synct tab-stream tacodb test-cmd test-helper test-report test-report-view testbed through tiles timestamp tiny-route tmpdir to-base traverser tree-query trees tslide typewiselite ubelt udid unordered-map-stream useragent-wtf vec2-animate vec2-dom vec2-easing vec2-layout vec2-tile- layer vec2-touch vec2-view vector-touch voice vu watch-stream wikiwiki wildfile x11 xcp xdiff xml-the-good-parts xmodel youtube-player ! — dominictarr
44. beepbeep bittorrent-client bittorrent-dht bittorrent-protocol bittorrent-swarm bittorrent-tracker buffer call-log chrome- dgram chrome-http chrome-net chrome-portfinder connectivity cyberhobo drag-drop futils hostile ieee754 jquery-placeholder magnet-uri nagger native-buffer- browserify parse-torrent purge-netdna run-auto run-parallel run-series rusha-browserify simple-build spoof string2compact studynotes timers-ref typedarray-to-buffer ut_metadata webtorrent zelda zero-fill ! — feross
45. moduletopia!
46. ~70,000 modules
47. gzip wc -c uglifyjs -m -c cat script.js
48. gzip wc -c uglifyjs -m -c cat script.js | | |
49. gzip wc -c uglifyjs -m -c cat script.js | | |
50. the npm way •have a good module system •pull code out into modules early, often •modules should do one thing, do it well •easy to test (npm test) •Badges, docs
51. http://www.yasiv.com/npm#view/webtorrent
52. bittorrent-client! (stream from bittorrent) webtorrent! (http range server, cmd line) webtorrent-client! (stream from webtorrent) webtorrent.app! (os x, windows, linux) webtorrent.js! (browser js library) webtorrent-chrome! (chrome app) bittorrent-dht bittorrent-protocol bittorrent-tracker bittorrent-swarm ut_metadata magnet-uri parse-torrent extension
53. streams
54. bittorrent-protocol
55. bittorrent-protocol in out
56. bittorrent-protocol in out onei1e4:woahd3:arr d3:bla4:blup3:foo3:bar3
57. bittorrent-protocol “handshake” in out d3:bla4:blup3:foo3:bar3:onei1e4:woa :blup3:foo3:bar3:onei1e4:woahd3:arr
58. bittorrent-protocol “handshake” “request” in out d3:bla4:blup3:foo3:bar3:onei1e4:woa :blup3:foo3:bar3:onei1e4:woahd3:arr
59. bittorrent-protocol “handshake” “request” in out d3:bla4:blup3:foo3:bar3:onei1e4:woa :blup3:foo3:bar3:onei1e4:woahd3:arr handshake()
60. bittorrent-protocol “handshake” “request” in out d3:bla4:blup3:foo3:bar3:onei1e4:woa :blup3:foo3:bar3:onei1e4:woahd3:arr handshake() data()
61. tcp socket bittorrent-protocol in out in out
62. bittorrent-protocol in out tcp socket in out
63. bittorrent-protocol in out webrtc in out
64. bittorrent-protocol in out ANYTHING! in out
65. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! ! ! ! ! ! ! ! !
66. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! net.createServer(function (socket) { ! ! ! ! ! ! ! })
67. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! net.createServer(function (socket) { var wire = new Protocol() ! ! ! ! ! ! })
68. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! net.createServer(function (socket) { var wire = new Protocol() ! socket.pipe(wire).pipe(socket) ! ! ! ! })
69. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! net.createServer(function (socket) { var wire = new Protocol() ! socket.pipe(wire).pipe(socket) ! wire.on(‘handshake’, function (infoHash, peerId) { }) })
70. var Protocol = require(‘bittorrent-protocol’) var net = require('net') ! net.createServer(function (socket) { var wire = new Protocol() ! socket.pipe(wire).pipe(socket) ! wire.on(‘handshake’, function (infoHash, peerId) { wire.handshake(infoHash, myPeerId) }) })
71. 2 Browserify
72. // foo.js ! module.exports = function (num) { return num + 2 } ! ! ! ! !
73. // foo.js ! module.exports = function (num) { return num + 2 } ! // bar.js ! module.exports = function (num) { return num * 10 }
74. // main.js ! var foo = require(‘./foo’) var bar = require(‘./bar’) ! console.log(foo(bar(4)))
75. // main.js ! var foo = require(‘./foo’) var bar = require(‘./bar’) ! console.log(foo(bar(4))) // 42
76. browserify main.js > bundle.js
77. <script src=“bundle.js”></script>
78. require(‘buffer’) require(‘stream’)
79. require(‘net’) require(‘dgram’)
80. // package.json ! { browser: { “net”: “chrome-net” “dgram”: “chrome-dgram” } }
81. require(‘net’) require(‘dgram’)
82. require(‘chrome-net’) require(‘chrome-dgram’)
83. Browserify lets you require('modules') in the browser by bundling up all of your dependencies.
84. 3 WebRTC
85. <script src=“webtorrent.js”></script> ! ! ! !
86. <script src=“webtorrent.js”></script> ! <script> var pdf = ‘d2474e86c95b19b8bcfdb92bc12c9d44667cfa36’ webtorrent(pdf, ‘#viewer’) </script>
87. <script src=“webtorrent.js”></script> ! <script> var movie = ‘d2474e86c95b19b8bcfdb92bc12c9d44667cfa36’ webtorrent(movie, function (err, stream) { if (!err) video.src = stream }) </script>
88. WebSocket WebSocket Server Browser Browser
89. WebRTC Browser Browser
90. Is peer-to-peer possible in the browser? •What about security? •What about the same origin policy? •What about asking the user for permission? •ISN’T THIS CRAZY?
91. WebRTC •Protocol and API to establish P2P connections •W3C/IETF web standard •Special support for video & audio •But, can send arbitrary data too •Simple JavaScript API
92. This is huge!
94. http://talky.io
95. INSTALL
98. 1,000,000,000 WebRTC devices
100. WebRTC video data
101. Data Channel •Send arbitrary data between browsers •Text and binary data •Reliable or unreliable transport •Encryption out-of-the-box •Simple WebSocket-like API
102. var pc = new RTCPeerConnection({ iceServers: [] }) pc.createOffer(function (offer) { pc.setLocalDescription(offer, function () { // send offer to peer }, function (err) { // could not set local description }) , function (err) { // could not create offer })
103. // peer 1 var channel = pc.createDataChannel('reliable', {}) ! channel.send(‘hi’) ! // peer 2 channel.onmessage = function (e) { console.log(e.data) // ‘hi’ }
104. // peer 1 var channel = pc.createDataChannel('unreliable', { ordered: false, maxRetransmits: 0 }) ! channel.send(‘hi’) ! // peer 2 channel.onmessage = function (e) { console.log(e.data) // ‘hi’ }
105. What can you do with Data Channel? •Games (especially first-person shooters) •CDNs (Content Delivery Networks) •File sharing •Replicated/distributed DBs, DHTs, etc. •Dark Web, Tor, Freenet …FREEDOM TECHNOLOGY??? •MAD SCIENCE!
107. Why is UDP useful? •Fire and forget •Realtime apps (games, stock quotes, etc.) •Live video streaming •Use UDP to create your own protocol!
108. npm install wrtc
113. quick tangent
114. How do you access the browser cache? •You can’t. No programmatic access. •Alternatives: •Filesystem API •IndexedDB •LocalStorage ! (10% of free space for temp., prompt for perm., Chrome only) (50 MB, prompt for more) (5-10 MB maximum)
115. Local Storage “User agents should limit the total amount of space allowed for storage areas. User agents should guard against sites storing data under the origins other affiliated sites, e.g. storing up to the limit in a1.example.com, a2.example.com, a3.example.com, etc, circumventing the main example.com storage limit.”
116. Local Storage •Chrome, IE, and Safari ignore recommendation •Every subdomain gets its own 5-10 MB quota •Subdomains are free •Thus, unlimited disk space with no prompt!
118. filldisk.com
120. Back to WebRTC
121. How do peers find each other? •Up to you! •Here’s an example: •Each user reports to server that they’re online •Make information available to all peers •When a peer wants to connect to another peer, it asks the server for an introduction to that peer
122. Signaling
123. LAN LAN WebSocket WebSocket Signaling server
124. LAN LAN WebSocket WebSocket Signaling server
125. LAN LAN WebSocket WebSocket Signaling server new RTCPeerConnection()
126. LAN LAN WebSocket WebSocket Offer
 SDP Signaling server
127. LAN LAN WebSocket WebSocket Offer
 SDP Signaling server
128. LAN LAN WebSocket WebSocket Offer
 SDP Signaling server
129. LAN LAN WebSocket WebSocket Offer
 SDP Signaling server
130. LAN LAN WebSocket WebSocket Offer
 SDP Signaling server
131. LAN LAN WebSocket WebSocket Answer
 SDP Signaling server
132. LAN LAN WebSocket WebSocket Answer
 SDP Signaling server
133. LAN LAN WebSocket WebSocket Answer
 SDP Signaling server
134. LAN LAN WebSocket WebSocket Answer
 SDP Signaling server
135. LAN LAN WebSocket WebSocket Answer
 SDP Signaling server
137. NAT (Network Address Translation)
138. LAN LAN ? The NAT problem Router Router User User
139. The NAT problem 192.168.1.100 192.168.1.101 44.44.44.44
140. STUN (Session Traversal Utilities for NAT)
141. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
142. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
143. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
144. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
145. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
146. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
147. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
148. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
149. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
150. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
151. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
152. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
153. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
154. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
155. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
156. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
157. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
158. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
159. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
160. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
161. STUN server LAN LAN WebSocket WebSocket Ice Candidate Ice Candidate What’s my ip? What’s my ip? Signaling server
162. Peer-to-peer engaged!
163. wrapping up
164. 1. npm 2. browserify 3. WebRTC
165. SimpleWebRTC (super easy, for video)
 PeerJS (super easy, for data)
166. webtorrent.io
167. feross aboukhadijeh webtorrent.io @feross happy hacking!