Web Ecosystem for Embedded Linux.pdf

Japan Technical Jamboree 71 で使用したスライド
https://elinux.org/Japan_Technical_Jamboree_71

Transcript

1. Web Ecosystem for Embedded Linux @ Japan Technical Jamboree 71 WebDINO Japan - Tomoya ASAI (dynamis)
2. about:me Tomoya ASAI - dynamis webdino.org (ex-Mozilla Japan) CTO Lecturer at Keio University Lecturer at Chuo University dynamis @ Community Twitter: @dynamitter facebook.com/dynamis speakerdeck.com/dynamis note: my avatar is red panda (fire fox, lesser panda) 2 @
3. about: WebDINO Japan Break the border of the Web! Non pro t org for Web, Browser, Openness Research & Consulting Research using Browsers (network, user behavior etc.) Building custom Browsers (Chromium, Firefox) Porting Browsers (for Embedded Devices etc) Human resource development & Event produce Web & Open education for students and youth Educating Engineers of Web, IoT and relating techs note: in 2017 "Mozilla Japan" renamed to "WebDINO Japan" https://www.webdino.org/ 3
4. Topics: Overview of Web platform update Web Ecosystem Web for rich HMI Accessing Native from Web Porting Browser Engines appendix: License checker for JS Modules 4
5. Web Ecosystem
6. "HTML5" as "Buzzword" in late 2000s HTML was buzzword in late 2000s starting from Gmail (2004), Google Maps (2005), many desktop apps switch to Web apps from 2008, Web is treated as OS independent platform to support new platforms such as iOS and Android. "Plateau of Productivity" now after too much expectations and disappointment, now widely used properly in many industries buzzword: a word or phrase that becomes very popular for a period of time 6
7. HTML5 in Hype Cycle by Gartner (2012) https://www.gartner.co.jp/press/html/pr20120906-01.html 7 HTML5 expectations for
 HTML5 became
 max around 2012 Big Data or IoT etc.
 follow HTML5 in 2012
8. Web (HTML5) ecosystem is... Most Popular Technologies all 1st ~ 3rd popular techs are web (JavaScript, HTML, CSS) Most Popular Frameworks all 1st ~ 3rd popular frameworks are web (Node.js, Angular, React) Wide Platform Support OS independent implementation with Web technologies Easy to work with Web services Free to Use No license fees depends on the device sales nor engineers Patent free (except some video codecs if you use it) Vendor independent No worry to stop support by the vendor (good for long-term) Rapid improvement by Google, Mozilla, MS, Apple etc Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 8
9. Most Popular Tech = Web Technologies TypeScript is meta lang of JavaScript
 Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 9
10. Most Popular Tech = Web Technologies TypeScript is meta lang (type-ed version) of JavaScript
 Stack Over Flow Developer Survey Results 2019 https://insights.stackoverflow.com/survey/2019 10
11. Most Popular for OSS, Most In-Demand for job https://stackify.com/popular-programming-languages-2018/ 11
12. Web Standard Support (as of 2018.04) API numbers counted with Web IDL https://github.com/MicrosoftEdge/APICatalogData 12 Chrome and Firefox (Gecko) lead web standard implementations 14,194 10,225 7,537 8,706 11,764
13. Past Web Standardization and evolution (~2015) ~2015, whole basic techs are updated HTML5, CSS 2015, ECMAScript 2015, HTTP/2 ... Incrementally continue to improve them... HTML5.1, 5.2, 5.3 ..., CSS 2017, 2018 ..., HTTP/3 ...
 ECMAScript 2016, 2017, 2018 ... Incremental/rapid update of Platforms Windows 10, Unity, Browsers... HTML/CSS/JS... 13
14. Existing Web
 APIs Trend: Web API evolution 2013~ Extensible Web Manifest Focus on low level, primitive APIs fetch, stream, custom element (component) ... 2015~ Progressive Web Apps (PWA) No more Gap between Native (mobile) Apps install, background, o ine, cache management, app/content packaging, worklets, webauthn ... 14 Low Level APIs
 (Extensible Web) Padding missing features (PWA etc.) Higher Lower
15. Trend: Web Browser evolution Utilize more GPU and multi cores Servo/Quantum (rewrite engines with Rust-lang) WebGPU (WIP: Vulkan, Metal etc version of WebGL)... Flow Browser (WIP: new layout/rendering engine + SpiderMonkey) targeted for STB or similar platform Smaller footprint, Faster page load 2017: Ignition+Turbofan in V8 2019: Baseline Interpreter in SpiderMonkey 2019: v8 Lite for low memory / embedder 15 memory saving by v8 Lite
16. NEW: Web as System Runtime WebAssembly (WASM): "compile to the Web" binary format compiled from C/C++/Rust etc same code run on JS engines & standalone runtime similar perf as C/C++/Rust (faster than Java etc) WASI: POSIX like system interface for WASM building "secure by default" ecosystem Developed/hosted by ByteCode Alliance
 (Mozilla, Fastly, Intel, Red Hat and more?) 16
17. generic/speci c WASI Runtime Wasmtime (generic) standalone WebAssembly runtime Lucet (by Fastly, CDNs and edge compute) focused on low latency, high concurrency ~50μs with few KB overhead to instantiate
 (5ms with tens of MB memory required for V8) AOT compilation from Wasm to Native code WebAssembly Micro Runtime (WAMR) focused on limited resources (embedded devices) require 2MB by default, can run on 256KB etc 17
18. Web for rich HMI
19. Where Browser Engine can be used? Just example device types... 19 Windows 10 Windows 10 IoT Consumer Prototype MCU MPU SoC Android & AOSP Realtime OS No OS Linux Qt Browser (main target) Arduino, RasPi etc. Industry iOS CPU+GPU Makers mainly Devices with rich HMI Win10 mobile macOS automotive ATM medical kiosk FA HEMS PC intercom signage sensor copy mobile Robot projector home appliance Nest AR echo TV STB PLC
20. Web use cases for embedded devices Printers (Richo etc) touch panels use HTML5 remotely manage custom display for each divisions of the company hardware control part of the main body also implemented with Web tech (Node.js) and work with browser of touch panel HMI Door Phone (Aiphone etc) Including Web Browser Support 3rd party app with Web Digital Signage Show HTML contents to make content management easy TVs (Almost smart TVs) Menu implemented with HTML support 3rd party apps with Web Apps (Japan or Euro region speci c apps such as Hybridcast and Maker speci c apps for Panasonic (Ajax App)) 20
21. HTML5 as GUI Framework develop Apps with HTML5 use as GUI Framework run on Gecko Engine use with Middle ware when needed to communication native component, use web socket (IP) connection to localhost or API enhancement of Gecko engine no app management (install etc) feature (yet) https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 21
22. HTML5 for Rich User Interface Rich design capabilities including i18n support or publishing level design Can be fast (but also can be slow) GPU accelerated animations/rendering use GPUs with WebGL (Web version of OpenGL) use WebAssembly for faster processing Many OSS framework/theme for UI design 22
23. Web support complex layouts (publishing) 2017 IDPF join W3C Digital Publishing format (ePub) now standardized in W3C 2019 writing mode level 3 become REC vertical text for Japanese, Mongolian etc RTL (right to left) for Arabic Hebrew, Persian etc. Japanese books can now made with CSS CSS Secrets by O'Reilly Japan & Vivliostyle http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja 23
24. CSS Font Features (2011~) https://helpx.adobe.com/jp/typekit/using/open-type-syntax.html 24 Various font features are supported by browser (especially by Gecko)
25. Vertical Text (Writing Mode) http://tategaki.github.io/awards/ https://developer.mozilla.org/docs/Web/CSS/writing-mode 25
26. Accessing Native
 from Web
27. Accessing Native libraries/devices (Gecko) 27 Browser Engine JavaScript Engine Graphics Backend Necko / Network Cache NSS OpenGL, D2D ... Skia GFX / Layout Thebe Azure Web Standard APIs Browser User Interface NPAPI
 Plug-ins Plug-in container NPAPI interpreter XPCOM Native Libraries js-ctypes Graphics Driver Device Drivers JIT Compilers Node.js Web Applications Hardwares Server & Apps Node
 modules HTML CSS JavaScript Custom Web APIs 3. support web protocol by Native libraries 5. add custom Web APIs 4. use Plug-in 1. via local server Cairo Web Extension APIs 2. native messaging API
28. Accessing Native libraries/devices from Web 1. via local (web socket or http) server Pros: easiest to implement Cons: network layer and local server overhead 2. Native Messaging API Pros: using standard extension API implemented in Browsers Cons: need communicate via extension' background js context 3. support Web protocol by Native side Pros: no local server overhead Cons: network layer overhead 4. Plug-ins Pros: better perf Cons: NPAPI is now deprecated 5. custom Web APIs Pros: better perf, exibility Cons: need to hack browser engine 28
29. Example: WebGPIO/I2C by community Easy-to-Prototype environment Single JavaScript program can control both graphical interface and hardwares released for Raspberry Pi and Jetson Nano so far access GPIO/I2C devices from Web apps poly ll.js provides WebGPIO/I2C APIs proposed in W3C browser and Robotics community group Web apps will easily access hardware devices community provides 30+ I2C device drivers (js library to init and control devices) implemented with WebI2C API: https://github.com/chirimen-oh/chirimen-drivers 29 Node Server Web Applications using WebGPIO/I2C apis GPIO/I2C Hardwares poly ll.js GPIO/I2C backend srv web socket
30. WebI2C API usage (via browser) (async function() { var i2cAccess = await navigator.requestI2CAccess(); var port = i2cAccess.ports.get(1); var sht = new SHT30(port); await sht.init(); for (;;) { var data = await sht.readData(); document.getElementById("temp").innerHTML = data.temperature.toFixed(2) + "degree"; document.getElementById("humidity").innerHTML = data.humidity.toFixed(2) + "%"; await sleep(200); } })(); 30
31. WebI2C API usage (via node) const { requestI2CAccess } = require("node-web-i2c"); const VL53L0X = require("@chirimen/vl53l0x"); (async function main() { const i2cAccess = await requestI2CAccess(); const port = i2cAccess.ports.get(1); const vl = new VL53L0X(port, 0x29); await vl.init(); // for Long Range Mode (<2m) : await vl.init(true); for (;;) { const distance = await vl.getRange(); console.log(`${distance} [mm]`); await sleep(500); } })() 31
32. Porting browser Engines (meta-browser)
33. Building Browser engines on Yocto OSSystems/meta-browser includes recipes of Chromium/Firefox browser! Pain Points: not enough maintained Many troubles to build for your Yocto... GPU acceleration or JIT compiler etc will be disabled for many embedded SoCs (by default) Firefox depends on newer meta-rust/meta-clang and cannot build with old Yocto ... etc... Gecko Embedded project is working for them https://github.com/webdino/gecko-embedded 33
34. Gecko on RZ/G RZ/G Linux Platform support HTML5 2 HMI platform = Qt / HTML5 HMI development with most popular Web tech HMI platform without license fees Gecko as VLP (Veri ed Linux Package) port Gecko, browser engine of Firefox which lead web tech improvement with Chrome Gecko support enable GPU accelerated Web-based HMI, App development https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 34
35. HTML5 Acceleration and SoC speci c porting Browser do JavaScript (main thread), Style and Layout calculation, Painting and Composting between each frame rendering If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow) SoC speci c porting needed to accelerate each steps for example, JIT engine(s) which use assembly code are needed to accelerate JavaScript (or about 10 times slower with interpreter execution) RZ/G use GPU for Paint (Canvas, Video) and Composite steps too! RZ/G support new Style engine "Stylo" and Layout engine "WebRender" details: https://developers.google.com/web/fundamentals/performance/rendering/ 35
36. Animation perf with/without GPU GPU Accelerated Compositor disabled vs enabled performance test with Gecko 52 on RZ/G1M 36 Fish Bowl
 (fps, 50 sh) Speed Reading
 (average draw) Fish IE Tank
 (fps, 250 sh) Speedometer
 (runs/min) 0 2.5 5 7.5 10 without GPU Compositor with GPU Compositor Bench mainly for Composite
 about 10 times faster 3d benchmark with fish tank about 2.6 times faster repeatedly render 96 elements about 3.2 times faster similar to real (various) app UI about 1.7 times faster
37. Do you need Browser? or Browser Engine? meta-browser will build "Browser" Full browser client app (same as desktop PCs) No o cial/standard way to use Gecko for embedding yet (Mozilla is developing GeckoView for Android but not for Linux yet) Embedded Linux need simple HTML viewer, not full-featured Browser! 37
38. Firefox and Gecko Firefox: browser as a whole Firefox is Branding name for users we port Firefox (as non brand browser) to RZ/G but UI part is not veri ed nor supported Gecko: Browser Engine part core program to load web apps We sometimes call Layout Engine part as Gecko 38 Devtools Server Browser Engine,
 Layout Engine JavaScript Engine Network, Security Rich & Heavy Full Browser UI Graphics APIs Devtools Client Linux BSP
 (HAL, Graphics backend etc...) Gecko Firefox telemetry, sync client ... etc.
39. WebViewer (simple HTML loader recipe) WebViewer slim wrapper to load HTML remove unnecessary modules no more access to Mozilla/ Google servers automatically smaller footprint (90~120MB smaller than Firefox on RZ/G) lower CPU usage (fewer background tasks in browser) fork of meta-browser includes WebViewer recipes etc https://github.com/webdino/meta-browser/ 39 Devtools Server Browser Engine,
 Layout Engine JavaScript Engine Network, Security Slim wrapper for
 Web content Graphics APIs Linux BSP
 (HAL, Graphics backend etc...) Gecko WebViewer
40. Appendix: lisence notice for JS modules
41. License of React (most famous framework) MIT License May 2013: React gets open sourced during the JSConfUS 2013 (with BSD + Patents license) Jul 2017: AFS moved Facebook BSD + Patents license to Category X (can not include) Aug 2017: Facebook explained about the license Sep 2017: React 16 (and 15.6.2) or later fainally relicensed under the MIT License 41
42. MIT License (for react) MIT License Copyright (c) Facebook, Inc. and its affiliates. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 42
43. license violation by o cial production React artifact react.development.js (OK) includes license comment of object-assign react.production.min.js (NG) doesn't include license comment of object-assign
 (code of object-assign is included) If you use production artifact of React (hosted by unpkg), you're violating the MIT license (unless you include separate license le mentioning object-assign, not only React). issue filed: https://github.com/facebook/react/issues/17492 43
44. react.development.js /** @license React v16.12.0 * react.development.js * * Copyright (c) Facebook, Inc. and its affiliates. * * This source code is licensed under the MIT license found in the * LICENSE file in the root directory of this source tree. */ 'use strict'; (function (global, factory) { typeof exports === 'object' && typeof module !== 'undefi ...snip... https://unpkg.com/browse/react@16.12.0/umd/react.development.js 44
45. react.development.js /** @license React v16.12.0 * react.development.js ...snip... /* object-assign (c) Sindre Sorhus @license MIT */ ...snip... https://unpkg.com/browse/react@16.12.0/umd/react.development.js 45
46. react.production.min.js /** @license React v16.12.0 * react.production.min.js * * Copyright (c) Facebook, Inc. and its affiliates. * * This source code is licensed under the MIT license found in the * LICENSE file in the root directory of this source tree. */ 'use strict';(function(v,m){"object"===typeof exports&&"undef ined"!==typeof module?module.exports=m():"function"===typeof define&&define.amd?define(m):v.React=m()})(this,function(){fu nction v(a){for(var b="https://reactjs.org/docs/error-decoder .html?invariant="+a,c=1;c<arguments.length;c++)b+="&args[]="+ ...snip... https://unpkg.com/browse/react@16.12.0/umd/react.production.min.js 46
47. tool: NPM License Checker license of the modules and dependencies All the npm modules or production modules installed in current dir will be listed. You can show list, count by license or output as csv/json le. note: this tool cannot automatically lter modules included in the build results (modules like web pack will be includes and you need to excludes them manually) https://github.com/davglass/license-checker 47
48. license-checker - basic usage > npx license-checker --production --unknown ├─ @babel/code-frame │ ├─ licenses: MIT │ ├─ repository: https://github.com/babel/babel/tree/master/ packages/babel-code-frame │ ├─ publisher: Sebastian McKenzie │ ├─ email: sebmck@gmail.com │ ├─ path: /.../node_modules/@babel/code-frame │ └─ licenseFile: /.../node_modules/@babel/code-frame/LICENSE ...snip... > npx license-checker --production --unknown --csv "module name","license","repository" "@babel/code-frame@7.5.5","MIT","https://github.com/babel/babel/ tree/master/packages/babel-code-frame" "@babel/core@7.6.0","MIT","https://github.com/babel/babel/tree/ master/packages/babel-core" ...snip... 48
49. Thank You! Any Questions? Feel free to contact me: dynamis@webdino.org
 @dynamitter, facebook.com/dynamis Lesser Panda illustration by @4687koSourijun