Slide 1

Slide 1 text

Web Ecosystem for Embedded Linux @ Japan Technical Jamboree 71 WebDINO Japan - Tomoya ASAI (dynamis)

Slide 2

Slide 2 text

about:me Tomoya ASAI - dynamis webdino.org (ex-Mozilla Japan) CTO Lecturer at Keio University Lecturer at Chuo University dynamis @ Community Twitter: @dynamitter facebook.com/dynamis speakerdeck.com/dynamis note: my avatar is red panda (fire fox, lesser panda) 2 @

Slide 3

Slide 3 text

about: WebDINO Japan Break the border of the Web! Non pro t org for Web, Browser, Openness Research & Consulting Research using Browsers (network, user behavior etc.) Building custom Browsers (Chromium, Firefox) Porting Browsers (for Embedded Devices etc) Human resource development & Event produce Web & Open education for students and youth Educating Engineers of Web, IoT and relating techs note: in 2017 "Mozilla Japan" renamed to "WebDINO Japan" https://www.webdino.org/ 3

Slide 4

Slide 4 text

Topics: Overview of Web platform update Web Ecosystem Web for rich HMI Accessing Native from Web Porting Browser Engines appendix: License checker for JS Modules 4

Slide 5

Slide 5 text

Web Ecosystem

Slide 6

Slide 6 text

"HTML5" as "Buzzword" in late 2000s HTML was buzzword in late 2000s starting from Gmail (2004), Google Maps (2005), many desktop apps switch to Web apps from 2008, Web is treated as OS independent platform to support new platforms such as iOS and Android. "Plateau of Productivity" now after too much expectations and disappointment, now widely used properly in many industries buzzword: a word or phrase that becomes very popular for a period of time 6

Slide 7

Slide 7 text

HTML5 in Hype Cycle by Gartner (2012) https://www.gartner.co.jp/press/html/pr20120906-01.html 7 HTML5 expectations for
 HTML5 became
 max around 2012 Big Data or IoT etc.
 follow HTML5 in 2012

Slide 8

Slide 8 text

Web (HTML5) ecosystem is... Most Popular Technologies all 1st ~ 3rd popular techs are web (JavaScript, HTML, CSS) Most Popular Frameworks all 1st ~ 3rd popular frameworks are web (Node.js, Angular, React) Wide Platform Support OS independent implementation with Web technologies Easy to work with Web services Free to Use No license fees depends on the device sales nor engineers Patent free (except some video codecs if you use it) Vendor independent No worry to stop support by the vendor (good for long-term) Rapid improvement by Google, Mozilla, MS, Apple etc Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 8

Slide 9

Slide 9 text

Most Popular Tech = Web Technologies TypeScript is meta lang of JavaScript
 Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 9

Slide 10

Slide 10 text

Most Popular Tech = Web Technologies TypeScript is meta lang (type-ed version) of JavaScript
 Stack Over Flow Developer Survey Results 2019 https://insights.stackoverflow.com/survey/2019 10

Slide 11

Slide 11 text

Most Popular for OSS, Most In-Demand for job https://stackify.com/popular-programming-languages-2018/ 11

Slide 12

Slide 12 text

Web Standard Support (as of 2018.04) API numbers counted with Web IDL https://github.com/MicrosoftEdge/APICatalogData 12 Chrome and Firefox (Gecko) lead web standard implementations 14,194 10,225 7,537 8,706 11,764

Slide 13

Slide 13 text

Past Web Standardization and evolution (~2015) ~2015, whole basic techs are updated HTML5, CSS 2015, ECMAScript 2015, HTTP/2 ... Incrementally continue to improve them... HTML5.1, 5.2, 5.3 ..., CSS 2017, 2018 ..., HTTP/3 ...
 ECMAScript 2016, 2017, 2018 ... Incremental/rapid update of Platforms Windows 10, Unity, Browsers... HTML/CSS/JS... 13

Slide 14

Slide 14 text

Existing Web
 APIs Trend: Web API evolution 2013~ Extensible Web Manifest Focus on low level, primitive APIs fetch, stream, custom element (component) ... 2015~ Progressive Web Apps (PWA) No more Gap between Native (mobile) Apps install, background, o ine, cache management, app/content packaging, worklets, webauthn ... 14 Low Level APIs
 (Extensible Web) Padding missing features (PWA etc.) Higher Lower

Slide 15

Slide 15 text

Trend: Web Browser evolution Utilize more GPU and multi cores Servo/Quantum (rewrite engines with Rust-lang) WebGPU (WIP: Vulkan, Metal etc version of WebGL)... Flow Browser (WIP: new layout/rendering engine + SpiderMonkey) targeted for STB or similar platform Smaller footprint, Faster page load 2017: Ignition+Turbofan in V8 2019: Baseline Interpreter in SpiderMonkey 2019: v8 Lite for low memory / embedder 15 memory saving by v8 Lite

Slide 16

Slide 16 text

NEW: Web as System Runtime WebAssembly (WASM): "compile to the Web" binary format compiled from C/C++/Rust etc same code run on JS engines & standalone runtime similar perf as C/C++/Rust (faster than Java etc) WASI: POSIX like system interface for WASM building "secure by default" ecosystem Developed/hosted by ByteCode Alliance
 (Mozilla, Fastly, Intel, Red Hat and more?) 16

Slide 17

Slide 17 text

generic/speci c WASI Runtime Wasmtime (generic) standalone WebAssembly runtime Lucet (by Fastly, CDNs and edge compute) focused on low latency, high concurrency ~50μs with few KB overhead to instantiate
 (5ms with tens of MB memory required for V8) AOT compilation from Wasm to Native code WebAssembly Micro Runtime (WAMR) focused on limited resources (embedded devices) require 2MB by default, can run on 256KB etc 17

Slide 18

Slide 18 text

Web for rich HMI

Slide 19

Slide 19 text

Where Browser Engine can be used? Just example device types... 19 Windows 10 Windows 10 IoT Consumer Prototype MCU MPU SoC Android & AOSP Realtime OS No OS Linux Qt Browser (main target) Arduino, RasPi etc. Industry iOS CPU+GPU Makers mainly Devices with rich HMI Win10 mobile macOS automotive ATM medical kiosk FA HEMS PC intercom signage sensor copy mobile Robot projector home appliance Nest AR echo TV STB PLC

Slide 20

Slide 20 text

Web use cases for embedded devices Printers (Richo etc) touch panels use HTML5 remotely manage custom display for each divisions of the company hardware control part of the main body also implemented with Web tech (Node.js) and work with browser of touch panel HMI Door Phone (Aiphone etc) Including Web Browser Support 3rd party app with Web Digital Signage Show HTML contents to make content management easy TVs (Almost smart TVs) Menu implemented with HTML support 3rd party apps with Web Apps (Japan or Euro region speci c apps such as Hybridcast and Maker speci c apps for Panasonic (Ajax App)) 20

Slide 21

Slide 21 text

HTML5 as GUI Framework develop Apps with HTML5 use as GUI Framework run on Gecko Engine use with Middle ware when needed to communication native component, use web socket (IP) connection to localhost or API enhancement of Gecko engine no app management (install etc) feature (yet) https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 21

Slide 22

Slide 22 text

HTML5 for Rich User Interface Rich design capabilities including i18n support or publishing level design Can be fast (but also can be slow) GPU accelerated animations/rendering use GPUs with WebGL (Web version of OpenGL) use WebAssembly for faster processing Many OSS framework/theme for UI design 22

Slide 23

Slide 23 text

Web support complex layouts (publishing) 2017 IDPF join W3C Digital Publishing format (ePub) now standardized in W3C 2019 writing mode level 3 become REC vertical text for Japanese, Mongolian etc RTL (right to left) for Arabic Hebrew, Persian etc. Japanese books can now made with CSS CSS Secrets by O'Reilly Japan & Vivliostyle http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja 23

Slide 24

Slide 24 text

CSS Font Features (2011~) https://helpx.adobe.com/jp/typekit/using/open-type-syntax.html 24 Various font features are supported by browser (especially by Gecko)

Slide 25

Slide 25 text

Vertical Text (Writing Mode) http://tategaki.github.io/awards/ https://developer.mozilla.org/docs/Web/CSS/writing-mode 25

Slide 26

Slide 26 text

Accessing Native
 from Web

Slide 27

Slide 27 text

Accessing Native libraries/devices (Gecko) 27 Browser Engine JavaScript Engine Graphics Backend Necko / Network Cache NSS OpenGL, D2D ... Skia GFX / Layout Thebe Azure Web Standard APIs Browser User Interface NPAPI
 Plug-ins Plug-in container NPAPI interpreter XPCOM Native Libraries js-ctypes Graphics Driver Device Drivers JIT Compilers Node.js Web Applications Hardwares Server & Apps Node
 modules HTML CSS JavaScript Custom Web APIs 3. support web protocol by Native libraries 5. add custom Web APIs 4. use Plug-in 1. via local server Cairo Web Extension APIs 2. native messaging API

Slide 28

Slide 28 text

Accessing Native libraries/devices from Web 1. via local (web socket or http) server Pros: easiest to implement Cons: network layer and local server overhead 2. Native Messaging API Pros: using standard extension API implemented in Browsers Cons: need communicate via extension' background js context 3. support Web protocol by Native side Pros: no local server overhead Cons: network layer overhead 4. Plug-ins Pros: better perf Cons: NPAPI is now deprecated 5. custom Web APIs Pros: better perf, exibility Cons: need to hack browser engine 28

Slide 29

Slide 29 text

Example: WebGPIO/I2C by community Easy-to-Prototype environment Single JavaScript program can control both graphical interface and hardwares released for Raspberry Pi and Jetson Nano so far access GPIO/I2C devices from Web apps poly ll.js provides WebGPIO/I2C APIs proposed in W3C browser and Robotics community group Web apps will easily access hardware devices community provides 30+ I2C device drivers (js library to init and control devices) implemented with WebI2C API: https://github.com/chirimen-oh/chirimen-drivers 29 Node Server Web Applications using WebGPIO/I2C apis GPIO/I2C Hardwares poly ll.js GPIO/I2C backend srv web socket

Slide 30

Slide 30 text

WebI2C API usage (via browser) (async function() { var i2cAccess = await navigator.requestI2CAccess(); var port = i2cAccess.ports.get(1); var sht = new SHT30(port); await sht.init(); for (;;) { var data = await sht.readData(); document.getElementById("temp").innerHTML = data.temperature.toFixed(2) + "degree"; document.getElementById("humidity").innerHTML = data.humidity.toFixed(2) + "%"; await sleep(200); } })(); 30

Slide 31

Slide 31 text

WebI2C API usage (via node) const { requestI2CAccess } = require("node-web-i2c"); const VL53L0X = require("@chirimen/vl53l0x"); (async function main() { const i2cAccess = await requestI2CAccess(); const port = i2cAccess.ports.get(1); const vl = new VL53L0X(port, 0x29); await vl.init(); // for Long Range Mode (<2m) : await vl.init(true); for (;;) { const distance = await vl.getRange(); console.log(`${distance} [mm]`); await sleep(500); } })() 31

Slide 32

Slide 32 text

Porting browser Engines (meta-browser)

Slide 33

Slide 33 text

Building Browser engines on Yocto OSSystems/meta-browser includes recipes of Chromium/Firefox browser! Pain Points: not enough maintained Many troubles to build for your Yocto... GPU acceleration or JIT compiler etc will be disabled for many embedded SoCs (by default) Firefox depends on newer meta-rust/meta-clang and cannot build with old Yocto ... etc... Gecko Embedded project is working for them https://github.com/webdino/gecko-embedded 33

Slide 34

Slide 34 text

Gecko on RZ/G RZ/G Linux Platform support HTML5 2 HMI platform = Qt / HTML5 HMI development with most popular Web tech HMI platform without license fees Gecko as VLP (Veri ed Linux Package) port Gecko, browser engine of Firefox which lead web tech improvement with Chrome Gecko support enable GPU accelerated Web-based HMI, App development https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 34

Slide 35

Slide 35 text

HTML5 Acceleration and SoC speci c porting Browser do JavaScript (main thread), Style and Layout calculation, Painting and Composting between each frame rendering If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow) SoC speci c porting needed to accelerate each steps for example, JIT engine(s) which use assembly code are needed to accelerate JavaScript (or about 10 times slower with interpreter execution) RZ/G use GPU for Paint (Canvas, Video) and Composite steps too! RZ/G support new Style engine "Stylo" and Layout engine "WebRender" details: https://developers.google.com/web/fundamentals/performance/rendering/ 35

Slide 36

Slide 36 text

Animation perf with/without GPU GPU Accelerated Compositor disabled vs enabled performance test with Gecko 52 on RZ/G1M 36 Fish Bowl
 (fps, 50 sh) Speed Reading
 (average draw) Fish IE Tank
 (fps, 250 sh) Speedometer
 (runs/min) 0 2.5 5 7.5 10 without GPU Compositor with GPU Compositor Bench mainly for Composite
 about 10 times faster 3d benchmark with fish tank about 2.6 times faster repeatedly render 96 elements about 3.2 times faster similar to real (various) app UI about 1.7 times faster

Slide 37

Slide 37 text

Do you need Browser? or Browser Engine? meta-browser will build "Browser" Full browser client app (same as desktop PCs) No o cial/standard way to use Gecko for embedding yet (Mozilla is developing GeckoView for Android but not for Linux yet) Embedded Linux need simple HTML viewer, not full-featured Browser! 37

Slide 38

Slide 38 text

Firefox and Gecko Firefox: browser as a whole Firefox is Branding name for users we port Firefox (as non brand browser) to RZ/G but UI part is not veri ed nor supported Gecko: Browser Engine part core program to load web apps We sometimes call Layout Engine part as Gecko 38 Devtools Server Browser Engine,
 Layout Engine JavaScript Engine Network, Security Rich & Heavy Full Browser UI Graphics APIs Devtools Client Linux BSP
 (HAL, Graphics backend etc...) Gecko Firefox telemetry, sync client ... etc.

Slide 39

Slide 39 text

WebViewer (simple HTML loader recipe) WebViewer slim wrapper to load HTML remove unnecessary modules no more access to Mozilla/ Google servers automatically smaller footprint (90~120MB smaller than Firefox on RZ/G) lower CPU usage (fewer background tasks in browser) fork of meta-browser includes WebViewer recipes etc https://github.com/webdino/meta-browser/ 39 Devtools Server Browser Engine,
 Layout Engine JavaScript Engine Network, Security Slim wrapper for
 Web content Graphics APIs Linux BSP
 (HAL, Graphics backend etc...) Gecko WebViewer

Slide 40

Slide 40 text

Appendix: lisence notice for JS modules

Slide 41

Slide 41 text

License of React (most famous framework) MIT License May 2013: React gets open sourced during the JSConfUS 2013 (with BSD + Patents license) Jul 2017: AFS moved Facebook BSD + Patents license to Category X (can not include) Aug 2017: Facebook explained about the license Sep 2017: React 16 (and 15.6.2) or later fainally relicensed under the MIT License 41

Slide 42

Slide 42 text

MIT License (for react) MIT License Copyright (c) Facebook, Inc. and its affiliates. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 42

Slide 43

Slide 43 text

license violation by o cial production React artifact react.development.js (OK) includes license comment of object-assign react.production.min.js (NG) doesn't include license comment of object-assign
 (code of object-assign is included) If you use production artifact of React (hosted by unpkg), you're violating the MIT license (unless you include separate license le mentioning object-assign, not only React). issue filed: https://github.com/facebook/react/issues/17492 43

Slide 44

Slide 44 text

react.development.js /** @license React v16.12.0 * react.development.js * * Copyright (c) Facebook, Inc. and its affiliates. * * This source code is licensed under the MIT license found in the * LICENSE file in the root directory of this source tree. */ 'use strict'; (function (global, factory) { typeof exports === 'object' && typeof module !== 'undefi ...snip... https://unpkg.com/browse/[email protected]/umd/react.development.js 44

Slide 45

Slide 45 text

react.development.js /** @license React v16.12.0 * react.development.js ...snip... /* object-assign (c) Sindre Sorhus @license MIT */ ...snip... https://unpkg.com/browse/[email protected]/umd/react.development.js 45

Slide 46

Slide 46 text

react.production.min.js /** @license React v16.12.0 * react.production.min.js * * Copyright (c) Facebook, Inc. and its affiliates. * * This source code is licensed under the MIT license found in the * LICENSE file in the root directory of this source tree. */ 'use strict';(function(v,m){"object"===typeof exports&&"undef ined"!==typeof module?module.exports=m():"function"===typeof define&&define.amd?define(m):v.React=m()})(this,function(){fu nction v(a){for(var b="https://reactjs.org/docs/error-decoder .html?invariant="+a,c=1;c

Slide 47

Slide 47 text

tool: NPM License Checker license of the modules and dependencies All the npm modules or production modules installed in current dir will be listed. You can show list, count by license or output as csv/json le. note: this tool cannot automatically lter modules included in the build results (modules like web pack will be includes and you need to excludes them manually) https://github.com/davglass/license-checker 47

Slide 48

Slide 48 text

license-checker - basic usage > npx license-checker --production --unknown ├─ @babel/code-frame │ ├─ licenses: MIT │ ├─ repository: https://github.com/babel/babel/tree/master/ packages/babel-code-frame │ ├─ publisher: Sebastian McKenzie │ ├─ email: [email protected] │ ├─ path: /.../node_modules/@babel/code-frame │ └─ licenseFile: /.../node_modules/@babel/code-frame/LICENSE ...snip... > npx license-checker --production --unknown --csv "module name","license","repository" "@babel/[email protected]","MIT","https://github.com/babel/babel/ tree/master/packages/babel-code-frame" "@babel/[email protected]","MIT","https://github.com/babel/babel/tree/ master/packages/babel-core" ...snip... 48

Slide 49

Slide 49 text

Thank You! Any Questions? Feel free to contact me: [email protected]
 @dynamitter, facebook.com/dynamis Lesser Panda illustration by @4687koSourijun