[Kenneth Christiansen] Bridging the Web and the Physical world

Presentation from GDG DevFest Ukraine 2017 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

Transcript

1. Kenneth Rohde Christiansen Web Platform Architect Поєднуємо веб і реальний світ Bridging the Web and the Physical world
2. Hi! I am Kenneth
3. Danish - happiest people in the world second most
7. Smart devices
8. What makes a thing "smart"?
9. Devices which are complicated or slow to use, are they really smart?
10. ➫ Solve a real issue ➫ Be there when you need them ➫ Ease to use, with low friction
12. I don't want to download your app, I just want to do stuff
13. Now! Now! Nowww!
14. With many devices around us apps don't scale well
15. Too many icons! Lack of overview Lots of updates pending for stuff I never use!
16. Too many icons! ➫ Lack of overview ➫ Updates pending for stuff I never use!
17. Let's look at usages
18. There are different common usages - Everyday use turn on the light! - Occasionally change temperature of my fridge - Once paying for parking when travelling
19. Better model?
20. We want low friction and ease of use
23. Focus on speed HTTP 2.0 push, Service Workers, PRPL pattern, streams, scrolling, CSS Houdini, Compositing …
24. Fast with Service Worker caching
26. The web is up to task
28. https://www.youtube.com/watch?v=Dsv5NT4PYG0
29. But that can't be web?! No address bar - super fast - icon on homescreen
30. We call that experience for Progressive Web Apps https://developers.google.com/web/progressive-web-apps/
32. Great for one-time use The low friction of the web
33. Always handy if I want it to be Add to homescreen
34. No clutter Looks and acts like an app
35. And safe
36. But... Copyright Club Skirts Dinah Shore Weekend https://commons.wikimedia.org/wiki/File:White_Diamonds_Party.jpg
37. Devices live outside the sandbox
38. And yes, the web can do that too!
39. Keep in mind Users and companies don't care about how stuff works, it should just work
40. Two great technologies
41. Advantages Widely used, Low energy, Wireless
42. Advantages Cheap, Easy to use, Beacon functionality
43. Disadvantages Security (just radio waves), Not powered, Stability, Speed
44. https://www.youtube.com/watch?v=fBCPA9gIxlU
46. ➫ Based on Bluetooth 4+ (Low Energy) GATT protocol ➫ Easy to use, modern web API ➫ Uses promises and typed arrays (ie. Uint8Array etc) ➫ No Classic mode support ➫ Only Central, not Peripheral Chrome only for now
48. An example HTTPS only User Gesture required to request device
49. https://www.youtube.com/watch?v=Fb_Pgf3rCdA
50. Requesting device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); navigator.bluetooth.requestDevice({ filters: [{ services: [0x1234, 0x12345678, '99999999-0000-1000-8000-00805f9b34fb'] }] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You may provide either the full Bluetooth UUID or a short 16- or 32-bit form, if not a standardized service Standardized services are easy to connect to
51. Requesting device navigator.bluetooth.requestDevice({ filters: [{ name: "Kenneth's robot" }], optionalServices: ['battery_service'] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You can also search by advertized name. Services can be optional, but needs to be listed if you want to access them when available
52. BLE Services and Characteristics GATT (Generic Attribute profile) defines how two BLE devices transfer data back and forth using concepts known as Services and Characteristics ➫ Services are logical entities, and contain specific chunks of data, known as Characteristics ➫ Characteristics encapsulate a single data point Services and Characterics are identified by 16- or 128 bit UUID's Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
53. BLE Services and Characteristics In JavaScript you can think of a Service being something like an object - ie. a collection of properties Characteristics are similar to properties on an object Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
54. Connect to a device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] }) .then(device => { // Human-readable name of the device. console.log(device.name); // Attempts to connect to remote GATT Server. return device.gatt.connect(); }) .then(server => { /* ... */ }) .catch(error => { console.log(error); }); When you have a device, you can connect to the GATT (Generic Attributes) server from where you can access Characteristics
55. Deal with Characteristics .then(service => { // Getting Battery Level Characteristic... return service.getCharacteristic('battery_level'); }) .then(characteristic => { // Reading Battery Level... return characteristic.readValue(); }) .then(value => { console.log('Battery percentage is ' + value.getUint8(0)); }) readValue() returns a DataView (part of the Typed Array family), so it is easy to extract values from.
56. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_control_point')) .then(characteristic => { // Writing 1 is the signal to reset energy expended. const resetEnergyExpended = Uint8Array.of(1); return characteristic.writeValue(resetEnergyExpended); }) .then(_ => { console.log('Energy expended has been reset.'); }) Writing is easy too!
57. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_measurement')) .then(characteristic => characteristic.startNotifications()) .then(characteristic => { characteristic.addEventListener('characteristicvaluechanged', handleCharacteristicValueChanged); console.log('Notifications have been started.'); }) .catch(error => { console.log(error); }); function handleCharacteristicValueChanged(event) { var value = event.target.value; console.log('Received ' + value); // TODO: Parse Heart Rate Measurement value. // See https://github.com/WebBluetoothCG/demos/blob/gh-pages/heart-rate-sensor/heartRateSensor.js } GATT notifications - subscribe to Characteristics changes
58. Bootstrapping - aka. how do people find my site from my device
59. Bluetooth devices can advertize web sites around them! That is Eddystone
60. Shows as notifications Click, and they open
61. Shows as notifications Click, and they open Can be regular sites No additional Bluetooth functionality needed
62. In near future Clicking on a Physical Web notification*, will allow the opened page to auto connect to device *Eddystone URL
63. Get started today! (or tomorrow) https://developers.google.com/web/updates/2015/07/ interact-with-ble-devices-on-the-web
64. Advantages Widely in use, Cheap, Cabled, More Secure, Fast, Stable, Can power devices
65. Disvantages Cabled
66. Safe and secure USB has no radio interference USB connector can be locked behind a hatch
67. Recall Users and companies don't care about how stuff works - it should just work Bluetooth and USB coexists just fine!
68. https://medium.com/@kennethrohde/program-a-smart-device-directly-no-install-needed-cd8b29320d76#.nla4jzls9 https://www.youtube.com/watch?v=o7wGt9RfHVA
69. Nice example showing ➫ A device capable of running JS! ➫ Bootstrapping and auto connect ➫ Device being powered over USB
70. Three different transfer types Bulk, Interrupt, Isochronous
71. Bulk Bulk transfers are the fastest* but have no guaranteed timing Printers and most CDC** (e.g serial) use bulk transfers. * Bulk transfers will use spare un-allocated bandwidth on the bus after all other transactions have been allocated. If the bus is busy with isochronous and/or interrupt then bulk data may slowly trickle over the bus. ** Communications Device Class
72. Interrupt Interrupt transfers have guaranteed maximum latency, or time between transaction attempts Mice and keyboards use interrupt transfers.
73. Isochronous Isochronous transfers have guaranteed timing but no error correcting Streaming audio and video use isochronous transfers
74. Web USB specific headers Creates further security and restrictions
75. Web USB specific headers ➫ Optional ➫ Required for bootstrapping popup ➫ Restrict which sites can access the peripheral
77. Issues The Linux modemmanager (enabled by default on most distros) hijacks (claims for 16sec) CDC devices unless their VID/PID are blacklisted (like the Arduino Leonardo) Note: Only affects CDC devices
78. Issues Windows autoloads drivers for a range of classes, e.g. HID, MSD and CDC (Win 10) Other Web USB devices additionally requires MS OS descriptors in order to autoload (or INF file, see below) Earlier versions (< 10) of Windows, requires a signed "driver", which is basically a signed INF text file, binding the VIP/PID to usbser.sys (the default USB serial driver) More info: https://medium.com/@larsgk/web-enabling-legacy-devices-dc3ecb9400ed#.fj7bd1ba3
79. API example let device; navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] }) .then(selectedDevice => { device = selectedDevice; return device.open(); // Begin a session. }) .then(() => device.selectConfiguration(1)) // Select configuration #1 for the device. .then(() => device.claimInterface(2)) // Request exclusive control over interface #2. .then(() => device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 })) // Ready to receive data .then(() => device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. .then(result => { let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); }) .catch(error => { console.log(error); }); In many ways similar in spirit to Web Bluetooth, but of course USB specific try { let device = await navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] }) await device.open(); // Begin a session. await device.selectConfiguration(1)) // Select configuration #1 for the device. await device.claimInterface(2)) // Request exclusive control over interface #2. await device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 }); // Ready to receive data let result = device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); } catch(error) { console.log(error); }
80. Availability Works in Chrome today on Linux, Mac and Windows
81. Get started today! https://developers.google.com/web/updates/2016/03/a ccess-usb-devices-on-the-web
82. Wow! But there is more! Photo by Andrés Þór
83. Many companies often need to track stuff
84. Bar codes and similar tech ➫ Quickly read and track things and products ➫ Get information about things around you ➫ Quick actions
85. Bluetooth Beacons might be the best solution
86. Or even NFC tags!
87. Or Quick Response codes and barcodes + they are cheap and printable
88. Barcode reader The Shape Detection API covering barcodes, faces and more: https://wicg.github.io/shape-detection-api/ try { let barcodeDetector = new BarcodeDetector(); // Assuming |theImage| is e.g. a <img> content, or a Blob. let detectedCodes = await barcodeDetector.detect(theImage); for (const barcode of detectedCodes) { const bx = barcode.boundingBox; console.log(`Barcode ${barcode.rawValue} @(${bx.x}, ${ox.y}) with size ${bx.width}x${bx.height}`); } } catch(err) { console.error(`Barcode Detection failed, boo: ${err}`); } Behind a flag In Chrome
89. NFC reader/writer The Web NFC API covers reading from and writing to NFC tags: https://w3c.github.io/web-nfc/ try { await navigator.nfc.push({ data: [{ recordType: "url", data: "https://w3c.github.io/web-nfc/" }] }); console.log("Message pushed."); } catch(err) { console.log("Push failed :-( try again."); } Behind a flag In Chrome
90. NFC reader/writer navigator.nfc.watch(message => { if (message.data[0].recordType == 'empty') { navigator.nfc.push({ data: [{ recordType: "json", mediaType: "application/json", data: { firstName: 'Kenneth' } }] }); } else { console.log( `Read msg written by ${message.url}` ); processMessage(message); } }); Behind a flag In Chrome function processMessage(message) { for (let d of message.data) { if (d.recordType == "json") { console.log(`Hello ${record.data.firstName}!`); } if (d.recordType == "opaque" && d.mediaType == 'image/png') { const blob = new Blob(d.data, d.mediaType); const img = document.createElement("img"); img.src = URL.createObjectURL(blob); img.onload = _ => URL.revokeObjectURL(img.src); $('images').appendChild(img); } } } }
91. Pretty cool, even Sir Tim Berners-Lee is excited
92. https://my-grocery-plan.glitch.me/ https://www.youtube.com/watch?v=rkjr6tveEuw
93. But what about Connecting to other devices to control or receive data
94. Like Sensorial input
95. But my already has sensors! Gimme gimme access pretty pretty please with sugar on top!
96. Could an API work for Local and Remote sensors? Web Platform, Node, IoT?
98. Find sensors hard? Read my motion sensors explainer https://w3c.github.io/motion-sensors/ Gravity Sensor Absolute Orientation Sensor Linear Acceleration Sensor Geomagnetic Orientation Sensor Low pass filter Sensor fusion
99. Very easy to use API Low-level, so you can do your own filters and sensor fusion const accel = new Accelerometer({ frequency: 50 }); const gyro = new Gyroscope({ frequency: 50 }); gyro.onreading = _ => { … } accel.start(); gyro.start(); Behind a flag In Chrome
100. let timestamp = null; gyro.onreading = _ => { let norm = Math.sqrt(accel.x ** 2 + accel.y ** 2) + accel.z ** 2); let xAccel = (accel.x / norm) * 90; let yAccel = (accel.y / norm) * -90; let zAccel = (accel.z / norm); let xGyro = gyro.x * 180 / Math.PI; let yGyro = gyro.y * 180 / Math.PI; let zGyro = gyro.z * 180 / Math.PI; let dt = timestamp ? ((gyro.timestamp - this.timestamp) / 1000) : 0; timestamp = gyro.timestamp; // Complementary filter // E.g.: new angle = 98% * (current angle + gyro rotation rate) + (2% * accel angle) this.alpha = weight * (this.alpha + zGyro * dt) + (1.0 - weight) * zAccel; this.beta = weight * (this.beta + xGyro * dt) + (1.0 - weight) * xAccel; this.gamma = weight * (this.gamma + yGyro * dt) + (1.0 - weight) * yAccel; console.log(`(${this.alpha}, ${this.beta}, ${this.gamma})`); };
101. Let's look at a video! ➫ Daydream controller exposed via Web Bluetooth ➫ Exposed as Generic Sensors ➫ Try it https://sensor-compass.appspot.com/
102. This is the Daydream controller
103. https://www.youtube.com/watch?v=qkvHJf8N7W0
104. What about Smart devices?
105. That is quite tiny
106. https://www.youtube.com/watch?v=K0efnqWDrkY
109. Summary The web is ready for nice app like experiences ⬤ ⬤ ⬤ ⬤
110. Summary You can effortless connect to devices over Bluetooth and USB Chrome for now, but expect other browser vendors to follow suit ⬤ ⬤ ⬤ ⬤
111. Summary NFC support and barcode readers are coming too! ⬤ ⬤ ⬤ ⬤
112. Summary Generic Sensors exposes sensors to the web Plus the API is easy to replace with custom implementation, say, via Web Bluetooth ⬤ ⬤ ⬤ ⬤
113. The web is ready, let's build the future
114. Follow me on Twitter, where I will post these slides today Or get them here now https://goo.gl/5zErV8 @kennethrohde