Serving Data From Browsers

In this talk, we'll cover how service workers can be used to gather, process & generate content in the browser. I'll show a practical use case of using this to drive web-based visualisation of geo-location data.

Transcript

1. SERVING DATA FROM BROWSERS
2. I’M BEN (@BENJAMINBENBEN)
5. THE INTERNET BUTTON
8. SERVING DATA FROM BROWSERS
9. SPOILER ALERT: SERVICE WORKERS
10. ! THANKS, PHIL
11. WHAT WE’RE TALKING ABOUT: MY DATA ➡ ‘ AS A USER, I WANT TO HAVE MY DATA IN A CSV DOCUMENT
12. LASTFM RUNKEEPER
13. BENFOXALL/LAST-FM-TO-CSV BENFOXALL/RUNKEEPER-TO-CSV
14. HOW DO WE GET OUR DATA? (RUNKEEPER)
15. ATTEMPT 1
16. DOWNLOAD IT
19. ! > Job Done > Works Offline
20. ! > No format control > Functionality might change/disappear
21. ATTEMPT 2
22. WRITE A SCRIPT
23. export BEARER=MY_TOKEN_FROM_THE_CONSOLE curl https://api.runkeeper.com/fitnessActivities?pageSize=100 -H "Authorization: Bearer $BEARER" > page1.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=2 -H "Authorization: Bearer $BEARER" > page2.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=3 -H "Authorization: Bearer $BEARER" > page3.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=4 -H "Authorization: Bearer $BEARER" > page4.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=5 -H "Authorization: Bearer $BEARER" > page5.json jq '.items' page*.json | grep uri | sed 's/^.*"\///g' | sed 's/"//g' > urls.txt mkdir fitnessActivities while read p; do curl https://api.runkeeper.com/$p -H "Authorization: Bearer $BEARER" > $p.json done < urls.txt
25. ! > format choices > sharable > offline
26. ! > Inaccessible
27. ATTEMPT 3
28. MAKE A WEB SERVICE
31. ! > Accessible to more people > Works offline*
32. ! > Non-trivial backend > Handling sensitive data > Waiting for processing
33. ATTEMPT 4
34. GO CLIENT-SIDE
38. ! > Data stored locally > Available instantly
39. ! IT MIGHT NOT TOTALLY WORK ALL THE TIME
40. COOL
41. TODO (IN BROWSER): 1. Request from data source 2. Process data (➡ csv) 3. Serve to user
42. 1. REQUEST > 2. PROCESS > 3. SERVE (LASTFM)
43. 1. REQUEST
44. A ONE FUNCTION JAVASCRIPT API const lastFM = params => fetch("http://ws.audioscrobbler.com/2.0/?"+ "method=user.getrecenttracks&format=json"+ "&api_key=974a5ebc077564f72bd639d122479d4b&"+ params) .then(r => r.json())
45. WHAT REQUESTS DO WE NEED FOR A USER? const requestsForUser = username => lastFM(`user=${username}`) .then(_find_number_of_pages_) .then(number => Array.from({length: number - 1}, (_,i) => `user=${username}&page=${i+1}` ) )
46. WHAT REQUESTS DO WE NEED FOR A USER? requestsForUser("benjaminf") // resolves to: [ "user=benjaminf&page=1", "user=benjaminf&page=2", "user=benjaminf&page=3", "user=benjaminf&page=4", … ]
47. MAKE REQUESTS const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map(lastFM) ) )
48. 2. PROCESS
49. JSON ➡ CSV
50. THE WAY YOU SHOULD PROBABLY DO THIS… papaparse.com/docs#json-to-csv
51. …ANOTHER WAY // pull out a row of keys const row = (keys, obj) => keys.map(k => obj[k]) // create a csv row from an array const csv = array => array.map(item => typeof(item) === 'string' ? item.replace(/[\",]/g,'') : item ).join(',') + "\n" /* csv(row(["a","b","n"],{b: "\"bar\"", c: "yeah", a: "foo", n: 42}) > "foo,bar,42" */
52. const process = response => response.tracks.map(track => csv(track, ["artist", "name"]) )
53. Promise.all( requests.map( r => lastFM(r).then(process) ) ) .then(parts => parts.join(""))
54. 3. SERVE
55. DATA URIS
56. data:[<mediatype>][;base64],<data>
57. data:text/html,<h1>Hello World
58. data:text/csv,a,b,c%0A1,2,3
59. ALL TOGETHER const process = response => response.tracks.map(track => csv(track, ["artist", "name"]) ) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map(lastFM) ) .then(parts => parts.join("")) )
60. ALL TOGETHER dataForUser("benjaminf") .then( csv => { const link = document.createElement("a") link.href = _to_data_uri(csv, "text/csv") link.innerText = "Download csv" document.body.appendChild(link) })
61. URGH .then(csv => { _to_data_uri(csv, "text/csv")
62. BLOBS
63. new Blob(parts [,options])
64. const blob = new Blob(["a,b,c\n", "d,e,f"],{type : "text/csv"}) const url = window.URL.createObjectURL(blob) // > "blob:…/41f8da4d-…-dbdc62106843" const a = document.createElement("a") a.textContent = a.href = url document.body.appendChild(a)
65. const process = response => response.tracks.map(track => csv(track, ["artist", "name"]) ) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map( req => lastFM(req).then(process) ) ) .then(parts => parts.join("")) )
66. const process = response => new Blob(response.tracks.map(track => csv(track, ["artist", "name"]) )) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map( req => lastFM(req).then(process) ) ) .then(parts => new Blob(parts, {type : "text/csv"})) )
67. dataForUser("benjaminf") .then( csv => { const link = document.createElement("a") link.href = _to_data_uri(csv, "text/csv") link.innerText = "Download csv" document.body.appendChild(link) })
68. dataForUser("benjaminf") .then( csv => { const link = document.createElement("a") link.href = URL.createObjectURL(csv) link.innerText = "Download csv" document.body.appendChild(link) })
69. 1. REQUEST > 2. PROCESS > 3. SERVE
70. benjaminbenben.com/lastfm-to-csv/
71. COOL, BUT… 1. DATA LOSS ON EXIT 2. NO URLS 3. DOESN’T WORK OFFLINE
72. ! SERVICE WORKERS ! (& INDEXEDDB)
73. 1. REQUEST > 2 SAVE > 3. PROCESS > 4. CACHE > 5. SERVE* (RUNKEEPER)
74. 1. REQUEST
75. PRETTY MUCH THE SAME FRONTEND
77. 2. SAVE !
78. IndexedDB (via Dexie) var db = new Dexie("Runkeeper") db.version(1).stores({ activities: "&uri" }) db.open(); // on request fetch('data' + uri, { credentials: 'include' }) .then(res => res.json() ) .then(data => db.activities.put(data))
80. 3. PROCESS
81. SUMMARY RESPONSE function summaryResponse() { console.time('build summary') var keys = ['uri', 'duration', 'type', 'start_time', 'total_distance', 'climb', 'total_calories']; var rows = [keys]; return db .activities .reverse() .each(function(activity){ rows.push( row(keys, activity) ) }) .then(function(){ console.timeEnd('build summary'); var data = new Blob(rows.map(function(row){ return csv(row) + '\n'; })); return new Response( data, { headers: { 'Content-Type': 'text/csv' } }); }) }
82. 4. CACHE ! "
83. function respond(event, generator){ event.respondWith( caches.match(event.request) .then(cached => { if(cached) return cached else return respondAndCache() }) ) function respondAndCache(){ return generator() .then((response) => { var responseToCache = response.clone(); caches.open(CACHE_NAME) .then((cache) => cache.put(event.request, responseToCache); ) return response; }) } }
84. 5. SERVE ! " !
85. SERVE SW/SUMMARY.CSV self.addEventListener('fetch', event => if(event.request.url.match(/sw\/summary\.csv$/)) respond(event, summaryResponse) )
86. WIN: MULTIPLE VIEWS self.addEventListener('fetch', event => { if(event.request.url.match(/sw\/summary\.csv$/)) respond(event, summaryResponse) if(event.request.url.match(/sw\/distances\.csv$/)) respond(event, distancesResponse) if(event.request.url.match(/sw\/paths\.csv$/)) respond(event, pathsResponse) if(event.request.url.match(/sw\/geo\.json$/)) respond(event, geoJSONResponse) if(event.request.url.match(/sw\/geo\.simple\.json$/)) respond(event, geoJSONResponseSimple) if(event.request.url.match(/sw\/binary\.path\.b$/)) respond(event, binaryPathResponse) })
87. ! > Offline > Super Fast > Super great
88. DEMO
89. GITHUB: BENFOXALL/LAST-FM-TO-CSV BENFOXALL/RUNKEEPER-TO-CSV
90. ONLINE HTTP://BENJAMINBENBEN.COM/ LASTFM-TO-CSV/ HTTPS://RUNKEEPER-TO- CSV.HEROKUAPP.COM/
91. $ $ PROFIT! $ $
92. THANKS! @BENJAMINBENBEN