Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Kenneth Christiansen] Bridging the Web and the Physical world

[Kenneth Christiansen] Bridging the Web and the Physical world

Presentation from GDG DevFest Ukraine 2017 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

Google Developers Group Lviv

October 14, 2017
Tweet

More Decks by Google Developers Group Lviv

Other Decks in Technology

Transcript

  1. ➫ Solve a real issue ➫ Be there when you

    need them ➫ Ease to use, with low friction
  2. There are different common usages - Everyday use turn on

    the light! - Occasionally change temperature of my fridge - Once paying for parking when travelling
  3. Focus on speed HTTP 2.0 push, Service Workers, PRPL pattern,

    streams, scrolling, CSS Houdini, Compositing …
  4. ➫ Based on Bluetooth 4+ (Low Energy) GATT protocol ➫

    Easy to use, modern web API ➫ Uses promises and typed arrays (ie. Uint8Array etc) ➫ No Classic mode support ➫ Only Central, not Peripheral Chrome only for now
  5. Requesting device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] }) .then(device

    => { /* ... */ }) .catch(error => { console.log(error); }); navigator.bluetooth.requestDevice({ filters: [{ services: [0x1234, 0x12345678, '99999999-0000-1000-8000-00805f9b34fb'] }] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You may provide either the full Bluetooth UUID or a short 16- or 32-bit form, if not a standardized service Standardized services are easy to connect to
  6. Requesting device navigator.bluetooth.requestDevice({ filters: [{ name: "Kenneth's robot" }], optionalServices:

    ['battery_service'] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You can also search by advertized name. Services can be optional, but needs to be listed if you want to access them when available
  7. BLE Services and Characteristics GATT (Generic Attribute profile) defines how

    two BLE devices transfer data back and forth using concepts known as Services and Characteristics ➫ Services are logical entities, and contain specific chunks of data, known as Characteristics ➫ Characteristics encapsulate a single data point Services and Characterics are identified by 16- or 128 bit UUID's Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
  8. BLE Services and Characteristics In JavaScript you can think of

    a Service being something like an object - ie. a collection of properties Characteristics are similar to properties on an object Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
  9. Connect to a device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }]

    }) .then(device => { // Human-readable name of the device. console.log(device.name); // Attempts to connect to remote GATT Server. return device.gatt.connect(); }) .then(server => { /* ... */ }) .catch(error => { console.log(error); }); When you have a device, you can connect to the GATT (Generic Attributes) server from where you can access Characteristics
  10. Deal with Characteristics .then(service => { // Getting Battery Level

    Characteristic... return service.getCharacteristic('battery_level'); }) .then(characteristic => { // Reading Battery Level... return characteristic.readValue(); }) .then(value => { console.log('Battery percentage is ' + value.getUint8(0)); }) readValue() returns a DataView (part of the Typed Array family), so it is easy to extract values from.
  11. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_control_point')) .then(characteristic => { //

    Writing 1 is the signal to reset energy expended. const resetEnergyExpended = Uint8Array.of(1); return characteristic.writeValue(resetEnergyExpended); }) .then(_ => { console.log('Energy expended has been reset.'); }) Writing is easy too!
  12. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_measurement')) .then(characteristic => characteristic.startNotifications()) .then(characteristic

    => { characteristic.addEventListener('characteristicvaluechanged', handleCharacteristicValueChanged); console.log('Notifications have been started.'); }) .catch(error => { console.log(error); }); function handleCharacteristicValueChanged(event) { var value = event.target.value; console.log('Received ' + value); // TODO: Parse Heart Rate Measurement value. // See https://github.com/WebBluetoothCG/demos/blob/gh-pages/heart-rate-sensor/heartRateSensor.js } GATT notifications - subscribe to Characteristics changes
  13. Shows as notifications Click, and they open Can be regular

    sites No additional Bluetooth functionality needed
  14. In near future Clicking on a Physical Web notification*, will

    allow the opened page to auto connect to device *Eddystone URL
  15. Recall Users and companies don't care about how stuff works

    - it should just work Bluetooth and USB coexists just fine!
  16. Nice example showing ➫ A device capable of running JS!

    ➫ Bootstrapping and auto connect ➫ Device being powered over USB
  17. Bulk Bulk transfers are the fastest* but have no guaranteed

    timing Printers and most CDC** (e.g serial) use bulk transfers. * Bulk transfers will use spare un-allocated bandwidth on the bus after all other transactions have been allocated. If the bus is busy with isochronous and/or interrupt then bulk data may slowly trickle over the bus. ** Communications Device Class
  18. Interrupt Interrupt transfers have guaranteed maximum latency, or time between

    transaction attempts Mice and keyboards use interrupt transfers.
  19. Isochronous Isochronous transfers have guaranteed timing but no error correcting

    Streaming audio and video use isochronous transfers
  20. Web USB specific headers ➫ Optional ➫ Required for bootstrapping

    popup ➫ Restrict which sites can access the peripheral
  21. Issues The Linux modemmanager (enabled by default on most distros)

    hijacks (claims for 16sec) CDC devices unless their VID/PID are blacklisted (like the Arduino Leonardo) Note: Only affects CDC devices
  22. Issues Windows autoloads drivers for a range of classes, e.g.

    HID, MSD and CDC (Win 10) Other Web USB devices additionally requires MS OS descriptors in order to autoload (or INF file, see below) Earlier versions (< 10) of Windows, requires a signed "driver", which is basically a signed INF text file, binding the VIP/PID to usbser.sys (the default USB serial driver) More info: https://medium.com/@larsgk/web-enabling-legacy-devices-dc3ecb9400ed#.fj7bd1ba3
  23. API example let device; navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }]

    }) .then(selectedDevice => { device = selectedDevice; return device.open(); // Begin a session. }) .then(() => device.selectConfiguration(1)) // Select configuration #1 for the device. .then(() => device.claimInterface(2)) // Request exclusive control over interface #2. .then(() => device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 })) // Ready to receive data .then(() => device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. .then(result => { let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); }) .catch(error => { console.log(error); }); In many ways similar in spirit to Web Bluetooth, but of course USB specific try { let device = await navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] }) await device.open(); // Begin a session. await device.selectConfiguration(1)) // Select configuration #1 for the device. await device.claimInterface(2)) // Request exclusive control over interface #2. await device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 }); // Ready to receive data let result = device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); } catch(error) { console.log(error); }
  24. Bar codes and similar tech ➫ Quickly read and track

    things and products ➫ Get information about things around you ➫ Quick actions
  25. Barcode reader The Shape Detection API covering barcodes, faces and

    more: https://wicg.github.io/shape-detection-api/ try { let barcodeDetector = new BarcodeDetector(); // Assuming |theImage| is e.g. a <img> content, or a Blob. let detectedCodes = await barcodeDetector.detect(theImage); for (const barcode of detectedCodes) { const bx = barcode.boundingBox; console.log(`Barcode ${barcode.rawValue} @(${bx.x}, ${ox.y}) with size ${bx.width}x${bx.height}`); } } catch(err) { console.error(`Barcode Detection failed, boo: ${err}`); } Behind a flag In Chrome
  26. NFC reader/writer The Web NFC API covers reading from and

    writing to NFC tags: https://w3c.github.io/web-nfc/ try { await navigator.nfc.push({ data: [{ recordType: "url", data: "https://w3c.github.io/web-nfc/" }] }); console.log("Message pushed."); } catch(err) { console.log("Push failed :-( try again."); } Behind a flag In Chrome
  27. NFC reader/writer navigator.nfc.watch(message => { if (message.data[0].recordType == 'empty') {

    navigator.nfc.push({ data: [{ recordType: "json", mediaType: "application/json", data: { firstName: 'Kenneth' } }] }); } else { console.log( `Read msg written by ${message.url}` ); processMessage(message); } }); Behind a flag In Chrome function processMessage(message) { for (let d of message.data) { if (d.recordType == "json") { console.log(`Hello ${record.data.firstName}!`); } if (d.recordType == "opaque" && d.mediaType == 'image/png') { const blob = new Blob(d.data, d.mediaType); const img = document.createElement("img"); img.src = URL.createObjectURL(blob); img.onload = _ => URL.revokeObjectURL(img.src); $('images').appendChild(img); } } } }
  28. Find sensors hard? Read my motion sensors explainer https://w3c.github.io/motion-sensors/ Gravity

    Sensor Absolute Orientation Sensor Linear Acceleration Sensor Geomagnetic Orientation Sensor Low pass filter Sensor fusion
  29. Very easy to use API Low-level, so you can do

    your own filters and sensor fusion const accel = new Accelerometer({ frequency: 50 }); const gyro = new Gyroscope({ frequency: 50 }); gyro.onreading = _ => { … } accel.start(); gyro.start(); Behind a flag In Chrome
  30. let timestamp = null; gyro.onreading = _ => { let

    norm = Math.sqrt(accel.x ** 2 + accel.y ** 2) + accel.z ** 2); let xAccel = (accel.x / norm) * 90; let yAccel = (accel.y / norm) * -90; let zAccel = (accel.z / norm); let xGyro = gyro.x * 180 / Math.PI; let yGyro = gyro.y * 180 / Math.PI; let zGyro = gyro.z * 180 / Math.PI; let dt = timestamp ? ((gyro.timestamp - this.timestamp) / 1000) : 0; timestamp = gyro.timestamp; // Complementary filter // E.g.: new angle = 98% * (current angle + gyro rotation rate) + (2% * accel angle) this.alpha = weight * (this.alpha + zGyro * dt) + (1.0 - weight) * zAccel; this.beta = weight * (this.beta + xGyro * dt) + (1.0 - weight) * xAccel; this.gamma = weight * (this.gamma + yGyro * dt) + (1.0 - weight) * yAccel; console.log(`(${this.alpha}, ${this.beta}, ${this.gamma})`); };
  31. Let's look at a video! ➫ Daydream controller exposed via

    Web Bluetooth ➫ Exposed as Generic Sensors ➫ Try it https://sensor-compass.appspot.com/
  32. Summary You can effortless connect to devices over Bluetooth and

    USB Chrome for now, but expect other browser vendors to follow suit ⬤ ⬤ ⬤ ⬤
  33. Summary Generic Sensors exposes sensors to the web Plus the

    API is easy to replace with custom implementation, say, via Web Bluetooth ⬤ ⬤ ⬤ ⬤
  34. Follow me on Twitter, where I will post these slides

    today Or get them here now https://goo.gl/5zErV8 @kennethrohde