Save 37% off PRO during our Black Friday Sale! »

Serving Data From Browsers

Serving Data From Browsers

In this talk, we'll cover how service workers can be used to gather, process & generate content in the browser. I'll show a practical use case of using this to drive web-based visualisation of geo-location data.

D5f36cb3a2b97a68cc812d985726b5dd?s=128

Ben Foxall

April 27, 2016
Tweet

Transcript

  1. SERVING DATA FROM BROWSERS

  2. I’M BEN (@BENJAMINBENBEN)

  3. None
  4. None
  5. THE INTERNET BUTTON

  6. None
  7. None
  8. SERVING DATA FROM BROWSERS

  9. SPOILER ALERT: SERVICE WORKERS

  10. ! THANKS, PHIL

  11. WHAT WE’RE TALKING ABOUT: MY DATA ➡ ‘ AS A

    USER, I WANT TO HAVE MY DATA IN A CSV DOCUMENT
  12. LASTFM RUNKEEPER

  13. BENFOXALL/LAST-FM-TO-CSV BENFOXALL/RUNKEEPER-TO-CSV

  14. HOW DO WE GET OUR DATA? (RUNKEEPER)

  15. ATTEMPT 1

  16. DOWNLOAD IT

  17. None
  18. None
  19. ! > Job Done > Works Offline

  20. ! > No format control > Functionality might change/disappear

  21. ATTEMPT 2

  22. WRITE A SCRIPT

  23. export BEARER=MY_TOKEN_FROM_THE_CONSOLE curl https://api.runkeeper.com/fitnessActivities?pageSize=100 -H "Authorization: Bearer $BEARER" > page1.json

    curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=2 -H "Authorization: Bearer $BEARER" > page2.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=3 -H "Authorization: Bearer $BEARER" > page3.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=4 -H "Authorization: Bearer $BEARER" > page4.json curl https://api.runkeeper.com/fitnessActivities?pageSize=100&page=5 -H "Authorization: Bearer $BEARER" > page5.json jq '.items' page*.json | grep uri | sed 's/^.*"\///g' | sed 's/"//g' > urls.txt mkdir fitnessActivities while read p; do curl https://api.runkeeper.com/$p -H "Authorization: Bearer $BEARER" > $p.json done < urls.txt
  24. None
  25. ! > format choices > sharable > offline

  26. ! > Inaccessible

  27. ATTEMPT 3

  28. MAKE A WEB SERVICE

  29. None
  30. None
  31. ! > Accessible to more people > Works offline*

  32. ! > Non-trivial backend > Handling sensitive data > Waiting

    for processing
  33. ATTEMPT 4

  34. GO CLIENT-SIDE

  35. None
  36. None
  37. None
  38. ! > Data stored locally > Available instantly

  39. ! IT MIGHT NOT TOTALLY WORK ALL THE TIME

  40. COOL

  41. TODO (IN BROWSER): 1. Request from data source 2. Process

    data (➡ csv) 3. Serve to user
  42. 1. REQUEST > 2. PROCESS > 3. SERVE (LASTFM)

  43. 1. REQUEST

  44. A ONE FUNCTION JAVASCRIPT API const lastFM = params =>

    fetch("http://ws.audioscrobbler.com/2.0/?"+ "method=user.getrecenttracks&format=json"+ "&api_key=974a5ebc077564f72bd639d122479d4b&"+ params) .then(r => r.json())
  45. WHAT REQUESTS DO WE NEED FOR A USER? const requestsForUser

    = username => lastFM(`user=${username}`) .then(_find_number_of_pages_) .then(number => Array.from({length: number - 1}, (_,i) => `user=${username}&page=${i+1}` ) )
  46. WHAT REQUESTS DO WE NEED FOR A USER? requestsForUser("benjaminf") //

    resolves to: [ "user=benjaminf&page=1", "user=benjaminf&page=2", "user=benjaminf&page=3", "user=benjaminf&page=4", … ]
  47. MAKE REQUESTS const dataForUser = username => requestsForUser(username) .then(requests =>

    Promise.all( requests.map(lastFM) ) )
  48. 2. PROCESS

  49. JSON ➡ CSV

  50. THE WAY YOU SHOULD PROBABLY DO THIS… papaparse.com/docs#json-to-csv

  51. …ANOTHER WAY // pull out a row of keys const

    row = (keys, obj) => keys.map(k => obj[k]) // create a csv row from an array const csv = array => array.map(item => typeof(item) === 'string' ? item.replace(/[\",]/g,'') : item ).join(',') + "\n" /* csv(row(["a","b","n"],{b: "\"bar\"", c: "yeah", a: "foo", n: 42}) > "foo,bar,42" */
  52. const process = response => response.tracks.map(track => csv(track, ["artist", "name"])

    )
  53. Promise.all( requests.map( r => lastFM(r).then(process) ) ) .then(parts => parts.join(""))

  54. 3. SERVE

  55. DATA URIS

  56. data:[<mediatype>][;base64],<data>

  57. data:text/html,<h1>Hello World

  58. data:text/csv,a,b,c%0A1,2,3

  59. ALL TOGETHER const process = response => response.tracks.map(track => csv(track,

    ["artist", "name"]) ) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map(lastFM) ) .then(parts => parts.join("")) )
  60. ALL TOGETHER dataForUser("benjaminf") .then( csv => { const link =

    document.createElement("a") link.href = _to_data_uri(csv, "text/csv") link.innerText = "Download csv" document.body.appendChild(link) })
  61. URGH .then(csv => { _to_data_uri(csv, "text/csv")

  62. BLOBS

  63. new Blob(parts [,options])

  64. const blob = new Blob(["a,b,c\n", "d,e,f"],{type : "text/csv"}) const url

    = window.URL.createObjectURL(blob) // > "blob:…/41f8da4d-…-dbdc62106843" const a = document.createElement("a") a.textContent = a.href = url document.body.appendChild(a)
  65. const process = response => response.tracks.map(track => csv(track, ["artist", "name"])

    ) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map( req => lastFM(req).then(process) ) ) .then(parts => parts.join("")) )
  66. const process = response => new Blob(response.tracks.map(track => csv(track, ["artist",

    "name"]) )) const dataForUser = username => requestsForUser(username) .then(requests => Promise.all( requests.map( req => lastFM(req).then(process) ) ) .then(parts => new Blob(parts, {type : "text/csv"})) )
  67. dataForUser("benjaminf") .then( csv => { const link = document.createElement("a") link.href

    = _to_data_uri(csv, "text/csv") link.innerText = "Download csv" document.body.appendChild(link) })
  68. dataForUser("benjaminf") .then( csv => { const link = document.createElement("a") link.href

    = URL.createObjectURL(csv) link.innerText = "Download csv" document.body.appendChild(link) })
  69. 1. REQUEST > 2. PROCESS > 3. SERVE

  70. benjaminbenben.com/lastfm-to-csv/

  71. COOL, BUT… 1. DATA LOSS ON EXIT 2. NO URLS

    3. DOESN’T WORK OFFLINE
  72. ! SERVICE WORKERS ! (& INDEXEDDB)

  73. 1. REQUEST > 2 SAVE > 3. PROCESS > 4.

    CACHE > 5. SERVE* (RUNKEEPER)
  74. 1. REQUEST

  75. PRETTY MUCH THE SAME FRONTEND

  76. None
  77. 2. SAVE !

  78. IndexedDB (via Dexie) var db = new Dexie("Runkeeper") db.version(1).stores({ activities:

    "&uri" }) db.open(); // on request fetch('data' + uri, { credentials: 'include' }) .then(res => res.json() ) .then(data => db.activities.put(data))
  79. None
  80. 3. PROCESS

  81. SUMMARY RESPONSE function summaryResponse() { console.time('build summary') var keys =

    ['uri', 'duration', 'type', 'start_time', 'total_distance', 'climb', 'total_calories']; var rows = [keys]; return db .activities .reverse() .each(function(activity){ rows.push( row(keys, activity) ) }) .then(function(){ console.timeEnd('build summary'); var data = new Blob(rows.map(function(row){ return csv(row) + '\n'; })); return new Response( data, { headers: { 'Content-Type': 'text/csv' } }); }) }
  82. 4. CACHE ! "

  83. function respond(event, generator){ event.respondWith( caches.match(event.request) .then(cached => { if(cached) return

    cached else return respondAndCache() }) ) function respondAndCache(){ return generator() .then((response) => { var responseToCache = response.clone(); caches.open(CACHE_NAME) .then((cache) => cache.put(event.request, responseToCache); ) return response; }) } }
  84. 5. SERVE ! " !

  85. SERVE SW/SUMMARY.CSV self.addEventListener('fetch', event => if(event.request.url.match(/sw\/summary\.csv$/)) respond(event, summaryResponse) )

  86. WIN: MULTIPLE VIEWS self.addEventListener('fetch', event => { if(event.request.url.match(/sw\/summary\.csv$/)) respond(event, summaryResponse)

    if(event.request.url.match(/sw\/distances\.csv$/)) respond(event, distancesResponse) if(event.request.url.match(/sw\/paths\.csv$/)) respond(event, pathsResponse) if(event.request.url.match(/sw\/geo\.json$/)) respond(event, geoJSONResponse) if(event.request.url.match(/sw\/geo\.simple\.json$/)) respond(event, geoJSONResponseSimple) if(event.request.url.match(/sw\/binary\.path\.b$/)) respond(event, binaryPathResponse) })
  87. ! > Offline > Super Fast > Super great

  88. DEMO

  89. GITHUB: BENFOXALL/LAST-FM-TO-CSV BENFOXALL/RUNKEEPER-TO-CSV

  90. ONLINE HTTP://BENJAMINBENBEN.COM/ LASTFM-TO-CSV/ HTTPS://RUNKEEPER-TO- CSV.HEROKUAPP.COM/

  91. $ $ PROFIT! $ $

  92. THANKS! @BENJAMINBENBEN