$30 off During Our Annual Pro Sale. View Details »

Web Ecosystem for Embedded Linux.pdf

dynamis
December 13, 2019

Web Ecosystem for Embedded Linux.pdf

Japan Technical Jamboree 71 で使用したスライド
https://elinux.org/Japan_Technical_Jamboree_71

dynamis

December 13, 2019
Tweet

More Decks by dynamis

Other Decks in Technology

Transcript

  1. Web Ecosystem for Embedded Linux
    @ Japan Technical Jamboree 71
    WebDINO Japan - Tomoya ASAI (dynamis)

    View Slide

  2. about:me
    Tomoya ASAI - dynamis webdino.org
    (ex-Mozilla Japan) CTO
    Lecturer at Keio University
    Lecturer at Chuo University
    dynamis @ Community
    Twitter: @dynamitter
    facebook.com/dynamis
    speakerdeck.com/dynamis
    note: my avatar is red panda (fire fox, lesser panda)
    2
    @

    View Slide

  3. about: WebDINO Japan
    Break the border of the Web!
    Non pro t org for Web, Browser, Openness
    Research & Consulting
    Research using Browsers (network, user behavior etc.)
    Building custom Browsers (Chromium, Firefox)
    Porting Browsers (for Embedded Devices etc)
    Human resource development & Event produce
    Web & Open education for students and youth
    Educating Engineers of Web, IoT and relating techs
    note: in 2017 "Mozilla Japan" renamed to "WebDINO Japan"
    https://www.webdino.org/
    3

    View Slide

  4. Topics: Overview of Web platform update
    Web Ecosystem
    Web for rich HMI
    Accessing Native from Web
    Porting Browser Engines
    appendix: License checker for JS Modules
    4

    View Slide

  5. Web Ecosystem

    View Slide

  6. "HTML5" as "Buzzword" in late 2000s
    HTML was buzzword in late 2000s
    starting from Gmail (2004), Google Maps (2005),
    many desktop apps switch to Web apps
    from 2008, Web is treated as OS independent
    platform to support new platforms such as iOS
    and Android.
    "Plateau of Productivity" now
    after too much expectations and disappointment,
    now widely used properly in many industries
    buzzword: a word or phrase that becomes very popular for a period of time
    6

    View Slide

  7. HTML5 in Hype Cycle by Gartner (2012)
    https://www.gartner.co.jp/press/html/pr20120906-01.html
    7
    HTML5
    expectations for

    HTML5 became

    max around 2012
    Big Data or IoT etc.

    follow HTML5 in 2012

    View Slide

  8. Web (HTML5) ecosystem is...
    Most Popular Technologies
    all 1st ~ 3rd popular techs are
    web (JavaScript, HTML, CSS)
    Most Popular Frameworks
    all 1st ~ 3rd popular frameworks
    are web (Node.js, Angular, React)
    Wide Platform Support
    OS independent implementation
    with Web technologies
    Easy to work with Web services
    Free to Use
    No license fees depends on the
    device sales nor engineers
    Patent free (except some video
    codecs if you use it)
    Vendor independent
    No worry to stop support by the
    vendor (good for long-term)
    Rapid improvement
    by Google, Mozilla, MS, Apple etc
    Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018
    8

    View Slide

  9. Most Popular Tech = Web Technologies
    TypeScript is meta lang of JavaScript

    Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018
    9

    View Slide

  10. Most Popular Tech = Web Technologies
    TypeScript is meta lang (type-ed version) of JavaScript

    Stack Over Flow Developer Survey Results 2019 https://insights.stackoverflow.com/survey/2019
    10

    View Slide

  11. Most Popular for OSS, Most In-Demand for job
    https://stackify.com/popular-programming-languages-2018/
    11

    View Slide

  12. Web Standard Support (as of 2018.04)
    API numbers counted with Web IDL https://github.com/MicrosoftEdge/APICatalogData
    12
    Chrome and Firefox (Gecko) lead web standard implementations
    14,194 10,225 7,537
    8,706
    11,764

    View Slide

  13. Past Web Standardization and evolution (~2015)
    ~2015, whole basic techs are updated
    HTML5, CSS 2015, ECMAScript 2015, HTTP/2 ...
    Incrementally continue to improve them...
    HTML5.1, 5.2, 5.3 ..., CSS 2017, 2018 ..., HTTP/3 ...

    ECMAScript 2016, 2017, 2018 ...
    Incremental/rapid update of Platforms
    Windows 10, Unity, Browsers... HTML/CSS/JS...
    13

    View Slide

  14. Existing Web

    APIs
    Trend: Web API evolution
    2013~ Extensible Web Manifest
    Focus on low level, primitive APIs
    fetch, stream, custom element (component) ...
    2015~ Progressive Web Apps (PWA)
    No more Gap between Native (mobile) Apps
    install, background, o ine, cache management,
    app/content packaging, worklets, webauthn ...
    14
    Low Level APIs

    (Extensible Web)
    Padding
    missing features
    (PWA etc.)
    Higher
    Lower

    View Slide

  15. Trend: Web Browser evolution
    Utilize more GPU and multi cores
    Servo/Quantum (rewrite engines with Rust-lang)
    WebGPU (WIP: Vulkan, Metal etc version of WebGL)...
    Flow Browser (WIP: new layout/rendering engine +
    SpiderMonkey) targeted for STB or similar platform
    Smaller footprint, Faster page load
    2017: Ignition+Turbofan in V8
    2019: Baseline Interpreter in SpiderMonkey
    2019: v8 Lite for low memory / embedder
    15
    memory saving by v8 Lite

    View Slide

  16. NEW: Web as System Runtime
    WebAssembly (WASM): "compile to the Web"
    binary format compiled from C/C++/Rust etc
    same code run on JS engines & standalone runtime
    similar perf as C/C++/Rust (faster than Java etc)
    WASI: POSIX like system interface for WASM
    building "secure by default" ecosystem
    Developed/hosted by ByteCode Alliance

    (Mozilla, Fastly, Intel, Red Hat and more?)
    16

    View Slide

  17. generic/speci c WASI Runtime
    Wasmtime
    (generic) standalone WebAssembly runtime
    Lucet (by Fastly, CDNs and edge compute)
    focused on low latency, high concurrency
    ~50μs with few KB overhead to instantiate

    (5ms with tens of MB memory required for V8)
    AOT compilation from Wasm to Native code
    WebAssembly Micro Runtime (WAMR)
    focused on limited resources (embedded devices)
    require 2MB by default, can run on 256KB etc
    17

    View Slide

  18. Web for rich HMI

    View Slide

  19. Where Browser Engine can be used?
    Just example device types...
    19
    Windows
    10
    Windows 10 IoT
    Consumer
    Prototype
    MCU MPU SoC
    Android
    & AOSP
    Realtime OS
    No OS
    Linux
    Qt
    Browser
    (main target)
    Arduino, RasPi etc.
    Industry
    iOS
    CPU+GPU
    Makers
    mainly Devices with rich HMI
    Win10
    mobile
    macOS
    automotive
    ATM
    medical
    kiosk
    FA
    HEMS
    PC
    intercom
    signage
    sensor
    copy
    mobile
    Robot
    projector
    home
    appliance
    Nest
    AR
    echo
    TV
    STB
    PLC

    View Slide

  20. Web use cases for embedded devices
    Printers (Richo etc)
    touch panels use HTML5
    remotely manage custom display
    for each divisions of the company
    hardware control part of the
    main body also implemented
    with Web tech (Node.js) and work
    with browser of touch panel HMI
    Door Phone (Aiphone etc)
    Including Web Browser
    Support 3rd party app with Web
    Digital Signage
    Show HTML contents to make
    content management easy
    TVs (Almost smart TVs)
    Menu implemented with HTML
    support 3rd party apps with Web
    Apps (Japan or Euro region
    speci c apps such as Hybridcast
    and Maker speci c apps for
    Panasonic (Ajax App))
    20

    View Slide

  21. HTML5 as GUI Framework
    develop Apps with HTML5
    use as GUI Framework
    run on Gecko Engine
    use with Middle ware
    when needed to communication
    native component, use web socket
    (IP) connection to localhost or API
    enhancement of Gecko engine
    no app management (install etc)
    feature (yet)
    https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html
    21

    View Slide

  22. HTML5 for Rich User Interface
    Rich design capabilities
    including i18n support or publishing level design
    Can be fast (but also can be slow)
    GPU accelerated animations/rendering
    use GPUs with WebGL (Web version of OpenGL)
    use WebAssembly for faster processing
    Many OSS framework/theme for UI design
    22

    View Slide

  23. Web support complex layouts (publishing)
    2017 IDPF join W3C
    Digital Publishing format (ePub) now
    standardized in W3C
    2019 writing mode level 3 become REC
    vertical text for Japanese, Mongolian etc
    RTL (right to left) for Arabic Hebrew, Persian etc.
    Japanese books can now made with CSS
    CSS Secrets by O'Reilly Japan & Vivliostyle
    http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja
    23

    View Slide

  24. CSS Font Features (2011~)
    https://helpx.adobe.com/jp/typekit/using/open-type-syntax.html
    24
    Various font features are supported
    by browser (especially by Gecko)

    View Slide

  25. Vertical Text (Writing Mode)
    http://tategaki.github.io/awards/ https://developer.mozilla.org/docs/Web/CSS/writing-mode
    25

    View Slide

  26. Accessing Native

    from Web

    View Slide

  27. Accessing Native libraries/devices (Gecko)
    27
    Browser Engine
    JavaScript Engine
    Graphics Backend
    Necko /
    Network
    Cache
    NSS
    OpenGL, D2D ...
    Skia
    GFX / Layout
    Thebe Azure
    Web Standard APIs
    Browser User Interface
    NPAPI

    Plug-ins
    Plug-in
    container
    NPAPI
    interpreter
    XPCOM
    Native Libraries
    js-ctypes
    Graphics Driver Device Drivers
    JIT Compilers
    Node.js
    Web Applications
    Hardwares
    Server
    & Apps
    Node

    modules
    HTML CSS JavaScript
    Custom Web APIs
    3. support web protocol
    by Native libraries
    5. add custom Web APIs
    4. use Plug-in
    1. via local server
    Cairo
    Web Extension APIs
    2. native messaging API

    View Slide

  28. Accessing Native libraries/devices from Web
    1. via local (web socket or http)
    server
    Pros: easiest to implement
    Cons: network layer and local
    server overhead
    2. Native Messaging API
    Pros: using standard extension API
    implemented in Browsers
    Cons: need communicate via
    extension' background js context
    3. support Web protocol by
    Native side
    Pros: no local server overhead
    Cons: network layer overhead
    4. Plug-ins
    Pros: better perf
    Cons: NPAPI is now deprecated
    5. custom Web APIs
    Pros: better perf, exibility
    Cons: need to hack browser engine
    28

    View Slide

  29. Example: WebGPIO/I2C by community
    Easy-to-Prototype environment
    Single JavaScript program can control both graphical
    interface and hardwares
    released for Raspberry Pi and Jetson Nano so far
    access GPIO/I2C devices from Web apps
    poly ll.js provides WebGPIO/I2C APIs proposed in W3C
    browser and Robotics community group
    Web apps will easily access hardware devices
    community provides 30+ I2C device drivers (js library to init
    and control devices) implemented with WebI2C API:
    https://github.com/chirimen-oh/chirimen-drivers
    29
    Node Server
    Web Applications
    using WebGPIO/I2C apis
    GPIO/I2C Hardwares
    poly ll.js
    GPIO/I2C backend srv
    web socket

    View Slide

  30. WebI2C API usage (via browser)
    (async function() {
    var i2cAccess = await navigator.requestI2CAccess();
    var port = i2cAccess.ports.get(1);
    var sht = new SHT30(port);
    await sht.init();
    for (;;) {
    var data = await sht.readData();
    document.getElementById("temp").innerHTML =
    data.temperature.toFixed(2) + "degree";
    document.getElementById("humidity").innerHTML =
    data.humidity.toFixed(2) + "%";
    await sleep(200);
    }
    })();
    30

    View Slide

  31. WebI2C API usage (via node)
    const { requestI2CAccess } = require("node-web-i2c");
    const VL53L0X = require("@chirimen/vl53l0x");
    (async function main() {
    const i2cAccess = await requestI2CAccess();
    const port = i2cAccess.ports.get(1);
    const vl = new VL53L0X(port, 0x29);
    await vl.init(); // for Long Range Mode (<2m) : await
    vl.init(true);
    for (;;) {
    const distance = await vl.getRange();
    console.log(`${distance} [mm]`);
    await sleep(500);
    }
    })()
    31

    View Slide

  32. Porting browser Engines
    (meta-browser)

    View Slide

  33. Building Browser engines on Yocto
    OSSystems/meta-browser
    includes recipes of Chromium/Firefox browser!
    Pain Points: not enough maintained
    Many troubles to build for your Yocto...
    GPU acceleration or JIT compiler etc will be disabled
    for many embedded SoCs (by default)
    Firefox depends on newer meta-rust/meta-clang and
    cannot build with old Yocto ... etc...
    Gecko Embedded project is working for them
    https://github.com/webdino/gecko-embedded
    33

    View Slide

  34. Gecko on RZ/G
    RZ/G Linux Platform support HTML5
    2 HMI platform = Qt / HTML5
    HMI development with most popular Web tech
    HMI platform without license fees
    Gecko as VLP (Veri ed Linux Package)
    port Gecko, browser engine of Firefox which lead
    web tech improvement with Chrome
    Gecko support enable GPU accelerated Web-based
    HMI, App development
    https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html
    34

    View Slide

  35. HTML5 Acceleration and SoC speci c porting
    Browser do JavaScript (main thread), Style and Layout calculation,
    Painting and Composting between each frame rendering
    If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow)
    SoC speci c porting needed to accelerate each steps
    for example, JIT engine(s) which use assembly code are needed to accelerate
    JavaScript (or about 10 times slower with interpreter execution)
    RZ/G use GPU for Paint (Canvas, Video) and Composite steps too!
    RZ/G support new Style engine "Stylo" and Layout engine "WebRender"
    details: https://developers.google.com/web/fundamentals/performance/rendering/
    35

    View Slide

  36. Animation perf with/without GPU
    GPU Accelerated Compositor disabled vs enabled performance test with Gecko 52 on RZ/G1M
    36
    Fish Bowl

    (fps, 50 sh)
    Speed Reading

    (average draw)
    Fish IE Tank

    (fps, 250 sh)
    Speedometer

    (runs/min)
    0 2.5 5 7.5 10
    without GPU Compositor with GPU Compositor
    Bench mainly for Composite

    about 10 times faster
    3d benchmark with fish tank
    about 2.6 times faster
    repeatedly render 96 elements
    about 3.2 times faster
    similar to real (various) app UI
    about 1.7 times faster

    View Slide

  37. Do you need Browser? or Browser Engine?
    meta-browser will build "Browser"
    Full browser client app (same as desktop PCs)
    No o cial/standard way to use Gecko for
    embedding yet (Mozilla is developing GeckoView
    for Android but not for Linux yet)
    Embedded Linux need simple HTML viewer,
    not full-featured Browser!
    37

    View Slide

  38. Firefox and Gecko
    Firefox: browser as a whole
    Firefox is Branding name for users
    we port Firefox (as non brand
    browser) to RZ/G but UI part is
    not veri ed nor supported
    Gecko: Browser Engine part
    core program to load web apps
    We sometimes call Layout Engine
    part as Gecko
    38
    Devtools
    Server
    Browser Engine,

    Layout Engine
    JavaScript
    Engine
    Network,
    Security
    Rich & Heavy Full Browser UI
    Graphics
    APIs
    Devtools
    Client
    Linux BSP

    (HAL, Graphics backend etc...)
    Gecko
    Firefox
    telemetry, sync
    client ... etc.

    View Slide

  39. WebViewer (simple HTML loader recipe)
    WebViewer
    slim wrapper to load HTML
    remove unnecessary modules
    no more access to Mozilla/
    Google servers automatically
    smaller footprint (90~120MB
    smaller than Firefox on RZ/G)
    lower CPU usage (fewer
    background tasks in browser)
    fork of meta-browser includes WebViewer recipes etc https://github.com/webdino/meta-browser/
    39
    Devtools
    Server
    Browser Engine,

    Layout Engine
    JavaScript
    Engine
    Network,
    Security
    Slim wrapper for

    Web content
    Graphics
    APIs
    Linux BSP

    (HAL, Graphics backend etc...)
    Gecko
    WebViewer

    View Slide

  40. Appendix: lisence
    notice for JS modules

    View Slide

  41. License of React (most famous framework)
    MIT License
    May 2013: React gets open sourced during the
    JSConfUS 2013 (with BSD + Patents license)
    Jul 2017: AFS moved Facebook BSD + Patents
    license to Category X (can not include)
    Aug 2017: Facebook explained about the license

    Sep 2017: React 16 (and 15.6.2) or later
    fainally relicensed under the MIT License
    41

    View Slide

  42. MIT License (for react)
    MIT License
    Copyright (c) Facebook, Inc. and its affiliates.
    Permission is hereby granted, free of charge, to any person obtaining a copy of
    this software and associated documentation files (the "Software"), to deal in
    the Software without restriction, including without limitation the rights to
    use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
    of the Software, and to permit persons to whom the Software is furnished to do
    so, subject to the following conditions:
    The above copyright notice and this permission notice
    shall be included in all copies or substantial portions
    of the Software.
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
    42

    View Slide

  43. license violation by o cial production React artifact
    react.development.js (OK)
    includes license comment of object-assign
    react.production.min.js (NG)
    doesn't include license comment of object-assign

    (code of object-assign is included)
    If you use production artifact of React (hosted
    by unpkg), you're violating the MIT license
    (unless you include separate license le
    mentioning object-assign, not only React).
    issue filed: https://github.com/facebook/react/issues/17492
    43

    View Slide

  44. react.development.js
    /** @license React v16.12.0
    * react.development.js
    *
    * Copyright (c) Facebook, Inc. and its affiliates.
    *
    * This source code is licensed under the MIT license
    found in the
    * LICENSE file in the root directory of this source tree.
    */
    'use strict';
    (function (global, factory) {
    typeof exports === 'object' && typeof module !== 'undefi
    ...snip...
    https://unpkg.com/browse/[email protected]/umd/react.development.js
    44

    View Slide

  45. react.development.js
    /** @license React v16.12.0
    * react.development.js
    ...snip...
    /*
    object-assign
    (c) Sindre Sorhus
    @license MIT
    */
    ...snip...
    https://unpkg.com/browse/[email protected]/umd/react.development.js
    45

    View Slide

  46. react.production.min.js
    /** @license React v16.12.0
    * react.production.min.js
    *
    * Copyright (c) Facebook, Inc. and its affiliates.
    *
    * This source code is licensed under the MIT license found in
    the
    * LICENSE file in the root directory of this source tree.
    */
    'use strict';(function(v,m){"object"===typeof exports&&"undef
    ined"!==typeof module?module.exports=m():"function"===typeof
    define&&define.amd?define(m):v.React=m()})(this,function(){fu
    nction v(a){for(var b="https://reactjs.org/docs/error-decoder
    .html?invariant="+a,c=1;c...snip...
    https://unpkg.com/browse/[email protected]/umd/react.production.min.js
    46

    View Slide

  47. tool: NPM License Checker
    license of the modules and dependencies
    All the npm modules or production modules
    installed in current dir will be listed.
    You can show list, count by license or output as
    csv/json le.
    note: this tool cannot automatically lter
    modules included in the build results (modules
    like web pack will be includes and you need to
    excludes them manually)
    https://github.com/davglass/license-checker
    47

    View Slide

  48. license-checker - basic usage
    > npx license-checker --production --unknown
    ├─ @babel/code-frame
    │ ├─ licenses: MIT
    │ ├─ repository: https://github.com/babel/babel/tree/master/
    packages/babel-code-frame
    │ ├─ publisher: Sebastian McKenzie
    │ ├─ email: [email protected]
    │ ├─ path: /.../node_modules/@babel/code-frame
    │ └─ licenseFile: /.../node_modules/@babel/code-frame/LICENSE
    ...snip...
    > npx license-checker --production --unknown --csv
    "module name","license","repository"
    "@babel/[email protected]","MIT","https://github.com/babel/babel/
    tree/master/packages/babel-code-frame"
    "@babel/[email protected]","MIT","https://github.com/babel/babel/tree/
    master/packages/babel-core"
    ...snip...
    48

    View Slide

  49. Thank You!
    Any Questions?
    Feel free to contact me:
    [email protected]
    @dynamitter, facebook.com/dynamis
    Lesser Panda illustration by @4687koSourijun

    View Slide