Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HTML5 on Renesas Linux Platform

dynamis
April 22, 2019

HTML5 on Renesas Linux Platform

Introduction to HTML5 (Gecko engine) support on Renesas RZ/G Linux Platform

Japanese version available:
https://speakerdeck.com/dynamis/html5-x-renesas-linux

dynamis

April 22, 2019
Tweet

More Decks by dynamis

Other Decks in Technology

Transcript

  1. HTML5 on Renesas RZ/G
    HTML5 Engine "Gecko" for RZ/G - Product Introduction
    WebDINO Japan - Tomoya ASAI

    View Slide

  2. about:me
    Tomoya ASAI
    CTO
    Tech Speaker
    mail: [email protected]
    dynamis @ Community
    Twitter: @dynamitter
    facebook.com/dynamis
    slideshare.net/dynamis
    I LOVE Firefox = Red Panda = Petit Panda = Lesser Panda
    2

    View Slide

  3. about:WebDINO Japan
    2017 June - name changed
    Mozilla Japan WebDINO Japan
    Web
    Diversity, Internationalization,
    Neutrality, Openness
    Gecko Embedded
    porting Gecko for Reneas RZ/G
    https://www.webdino.org/
    3

    View Slide

  4. Topics: HTML5 on RZ/G Introduction
    about: HTML5
    HTML5 de nition, now and future
    HTML5 and Gecko engine
    What's Gecko Engine
    HTML5 and Gecko on RZ/G
    HTML5 for IoT and RZ/G Platform
    4

    View Slide

  5. about:html5

    View Slide

  6. Point: what is HTML5?
    HTML5 in a Narrow sense = HTML4.next
    "HTML Living Standard" is the latest spec
    "HTML5" in a Broad sense = latest Web tech
    Web technologies for Application platform
    Multi-Platform App Environment
    All modern (rich) OSs support HTML5!
    Web developers recently use "PWA" to summarize latest web technologies, instead of HTML5
    6

    View Slide

  7. HTML, CSS, JavaScript = 3 basis of the Web
    HTML (Hyper Text Markup Language)
    Markup Language for the Web
    De ne contents and structure
    CSS (Cascading Style Sheet)
    Control appearance of contents
    JS (JavaScript)
    Control behavior of Web Apps
    "Java"Script come from Java because of marketing purpose (not subset of Java)
    7

    View Slide

  8. HTML example - contents marked up with
    ... (snip) ...



    height="20" alt="Close">
    My アカウント
    User Name

    My Renesas ポータル
    ユーザ情報の変更
    パスワードの変更
    お気に⼊り製品登録
    My ダウンロード




    ... (snip) ...
    from https://www.renesas.com/ja-jp/
    8

    View Slide

  9. CSS example - xxx { yyy: zzz; } format
    div.search {
    padding:0 1px 0 0;
    }
    div.searchRight {
    border-left: 1px solid
    #dddddd;
    float: right;
    width: 150px;
    padding-left: 20px;
    padding-bottom: 20px;
    }
    div.searchTrends {
    text-align: justify;
    }
    from: https://www.renesas.com/ja-jp/
    9
    div.searchRight p {
    text-align: center;
    font-weight: bold;
    margin-bottom: 5px;
    }
    div.search span.icon img {
    width: 16px;
    height: 16px;
    }
    div.search span.icon {
    padding: 0 2px 8px 0;
    background: url(../../
    designs/default/images/
    icons/default.gif) no-repeat
    }

    View Slide

  10. JavaScript example - C/Java like language
    $(document).ready(function(){
    var search = function() {
    var keyword = $('#inputText').val().trim();
    if (!keyword) return;
    var ekeyword = encodeURIComponent(keyword);
    var rekeyword = ekeyword.replace(/%20/g, '+');
    window.location.href = '/ja-jp/search/keyword-
    search.html#q=' + rekeyword;
    };
    $('#submitBtn').click(search);
    $('#inputText').keypress(function(e) {
    if (e.keyCode === 13) search();
    });
    });
    from https://www.renesas.com/ja-jp/
    10

    View Slide

  11. Then, what's HTML5?
    Tell me the de nition...

    View Slide

  12. from birth of the HTML to HTML4
    1989: 1st proposal for the Web
    1990: 1st Web Browser WorldWideWeb released
    1993: HTML published as Internet Draft
    1994: W3C start web standardization
    1995: HTML 2.0 published as RFC 1866
    1997: HTML 3.2 become W3C REC
    1997: HTML 4.0 become W3C REC
    12

    View Slide

  13. birth of WHATWG and HTML5
    1998 2006: XHTML standardization (and it's fail)
    W3C focused to re-invent HTML with XHTML

    but XHTML was not widely used since it's not compatible with
    HTML4, not supported by IE
    2004: WHATWG established
    Mozilla & Opera had proposed backward compatible
    enhancement for HTML4 but rejected by W3C. Then browser
    vendors start discussion in another organization, WHATWG
    2007: W3C start new HTML WG
    W3C started HTML5 standardization based on WHATWG HTML
    2014: HTML5 become W3C REC (recommendation)
    13

    View Slide

  14. current HTML5 = HTML Living Standard
    latest HTML spec = WHATWG HTML
    Living Standard = version-less spec
    Browser vendors only refer HTML living standard
    de ned by WHATWG, not W3C HTML5.x
    snapshot version = W3C HTML 5.x
    W3C continue to publish HTML5.x year by year
    based on WHATWG HTML living standard
    small di erence exists but implementors ignore it
    14

    View Slide

  15. summary: (narrow) HTML5 = HTML4.next
    not re-invent the Web, incremental update
    HTML4++ = HTML5
    backward compatible, step-by-step improvement
    1st step for next gen tech
    HTML5 start to summarize already implemented
    non-standard (browser speci c) technologies
    for Web Applications
    1st step to create Large Web Apps
    15

    View Slide

  16. What can "HTML5" do?

    View Slide

  17. WHATWG HTML spec - Introduction
    https://html.spec.whatwg.org/multipage/introduction.html#is-this-html5?
    17
    “the term "HTML5" is widely used
    as a buzzword to refer to modern
    Web technologies”

    View Slide

  18. "HTML5" as "Buzzword"
    HTML was buzzword in late 2000s
    starting from Gmail (2004), Google Maps (2005),
    many desktop apps switch to Web apps
    from 2008, Web is treated as OS independent
    platform to support new platforms such as iOS
    and Android.
    "Plateau of Productivity" now
    after too much expectations and disappointment,
    now widely used properly in many industries
    buzzword: a word or phrase that becomes very popular for a period of time
    18

    View Slide

  19. HTML5 in Hype Cycle by Gartner (2012)
    https://www.gartner.co.jp/press/html/pr20120906-01.html
    19
    HTML5
    expectations for

    HTML5 became

    max around 2012
    Big Data or IoT etc.

    follow HTML5 in 2012

    View Slide

  20. HTML5 for Desktop PC Platform
    2004-2005 Gmail, Google Maps
    after Gmail (2004), Google Maps (2005), Flash/Java
    apps start to switch to HTML5
    2014 Atom, Electron
    GitHub released Atom text editor and Electron
    framework to create desktop apps like Atom with
    Node.js and Chromium (and it become popular)
    2014 Unity 5.3 support HTML5 (WebGL)
    AAA games can now run on browsers
    FYI: XUL Runner with Gecko was used before Electron
    20

    View Slide

  21. AAA 3D Games come to the Web
    Unity 5 start to support WebGL
    Unity 5.3 o cially suppoprt WebGL
    Unity Web Player Plug-in become EOL
    C# => C => asm.js
    about 1/2 speed of Native Unity (2016)
    Unreal Engine 3 support WebGL
    Unreal Engine 4.9 include WebGL export by default
    about 2/3 speed of Native (2014)
    21

    View Slide

  22. Unreal Engine support WebGL (2014)
    22

    View Slide

  23. HTML5 for Mobile Platform
    ~2011: highest expectation for HTML5
    mainly as "hybrid apps" with Cordova
    to support both iOS and Android with single code
    2012: disappointed
    iOS Facebook app was too slow (because of limitation
    of WebView of iOS, no problem for Android actually...)
    2015~: HTML5 (PWA) become popular again
    Web app (currently called PWA) and new hybrid
    frameworks such as React Native become popular
    23

    View Slide

  24. Web App on 1st iPhone
    Steve Jobs Keynote MacWorld 2007 SF https://www.youtube.com/watch?v=P-a_R6ewrmM
    24
    “The full Safari engine is inside of
    iPhone. And so, you can write
    amazing Web 2.0 and Ajax apps
    that look exactly and behave
    exactly like apps on the iPhone.”

    View Slide

  25. Web App on Mobile (smartphone, ~2011)
    2007.01 iPhone release
    3rd Party apps are all Web apps (no native app support)

    iPhone = phone + iPod + internet client
    2008.08 App Store start
    support native 3rd Party apps
    2008.09 Android 1.0 release
    Android Market start on 2008.10 along with HTC Dream
    around 2010: hybrid apps become trend
    WebView was widely used to make HTML5 apps
    25

    View Slide

  26. when new iOS (native) Facebook app release
    Mark Zuckerberg at TechCrunch Disrupt SF 2012 https://www.youtube.com/watch?v=P-a_R6ewrmM
    26
    “... the biggest mistake that we
    made, as a company, is

    betting too much on HTML5

    as opposed to native …”

    View Slide

  27. Web App on Mobile (smartphone, 2012~)
    2012.08: Facebook App switch to Native
    Hybrid version facebook app was too slow on iOS
    and they switch to Native implementation
    ~2015-2016 Native apps become popular
    on Google I/O 2016, PWA was introduced and
    Android apps start to use Web widely again...
    2018.03: iOS 11.3 support PWA
    O -line, Install, Payment API support for Web apps.
    This is consumer app trend. Majority of Enterprise apps already become hybrid/html5/pwa apps.
    27

    View Slide

  28. Recent HTML5 (PWA) showcases
    2016.03 Flipkart
    usage time x 3
    re engatement rate +40%
    Trivago
    engagement +150%
    conversion +97%
    2016.05 (Suumo)
    Page loading time -75%
    read rate of push noti cation +31%
    2016.09 Alibaba
    conversion rate +76%
    Android active users +30%
    2017.03 Twitter Lite
    page view / session +65%
    number of tweets +75%
    2017.05 (Rakuten Recipe)
    Usage time +50%
    Page view x 3.1
    Access via push noti cation x 3.3~5.3
    2017.05 Lancôme
    conversion rate +17%
    conversion rate via push noti cation +8%
    2017.11 Nikkei News
    Speed index x 2 or more
    28

    View Slide

  29. sorry, a little old and include deprecated specs... http://webapi.link
    29
    Net
    Info
    Web
    Socket
    APNG
    Full-
    screen
    WebRTC
    HTML
    Components
    ECMAScript

    2015
    CSS Snapshot 2015
    WHATWG
    W3C

    HTML5
    URL
    HTTP
    TLS
    Element
    s &
    WAI-
    ARIA
    HTML5
    Parser
    Web
    Workers
    Web
    Socket
    Canvas
    2D
    multi
    media
    Content
    Model
    app
    cache
    section
    s
    HTML5
    Forms
    Server-
    Sent ev.
    geo-
    locatio
    Orient
    ation
    Notifi-
    Game-
    pad
    Media
    data
    channe
    Presen
    - Push
    SSL
    Media
    Recorder
    Opus
    mp3
    H.
    264
    Service
    Worker
    Filter
    SVG WOFF
    Math

    ML RSS
    Layout
    Media

    trans-
    form
    Tran-
    sitions &
    Anima-
    tions
    Flex
    Box
    Multi
    Colum
    Fonts
    User
    Inter-
    Shapes
    text
    decora-
    Indexed

    DB
    vibra-
    tion
    SMIL
    VP10
    HSTS
    CORS
    RDF
    RDFa
    DNT
    VP9
    Web
    Open
    Media
    VP8
    HTTP/2
    PNG
    Web
    Audio
    Batter
    y
    Generi
    c
    Shared

    Workers
    SIMD
    Web
    Crypto
    temp-
    lates
    Shadow
    DOM
    Custom
    Element
    s
    Page
    Visibilit
    FIDO
    Grid
    Layout
    File
    Pointer
    MIDI
    Speech
    Pro-
    mise
    CSP
    Class
    Modul
    e
    block
    scope
    Typed
    Array
    Arrow
    Func.
    Templat
    e
    Gene-
    rators
    Async
    Func.
    Navi.
    Timing
    Beacon
    Exclu-
    sions
    Writing
    Mode
    Web
    Anima-
    tions
    IETF Khronos
    OGP
    Game, VR & High

    Performance Apps
    Progressive
    Web Apps
    Real-Time

    Communications
    Device

    Control
    Extensible

    for Anything!
    Micro-
    data
    data
    ORTC
    WebVR
    Shared
    Mem.
    Itera-
    tors
    Proxy
    Collec-
    tions
    asm.js
    Permi-
    ssions
    DOM
    WebGL
    fetch
    Web
    Assembl
    y
    EME
    Brotli
    Mask-
    ing
    Progressive Enhancement
    Web
    App
    Manifest
    Web
    Storage
    Quarks
    Mode
    UI
    Events
    QUIC
    Origi
    Blue-
    tooth
    USB
    HPKP

    View Slide

  30. Web Standard Support (2018.04)
    API numbers counted with Web IDL https://github.com/MicrosoftEdge/APICatalogData
    30
    Chrome and Firefox (Gecko) lead web standard implementations
    14,194 10,225 7,537
    8,706
    11,764

    View Slide

  31. Recent Web Standardization
    until 2015, basic techs are completely
    updated
    HTML5, CSS 2015, ECMAScript 2015, HTTP/2 ...
    Incrementally continue to improve them...
    HTML5.1, 5.2, 5.3 ..., CSS 2017, 2018 ..., HTTP/3 ...

    ECMAScript 2016, 2017, 2018 ...
    Spec and Platforms now incrementally update
    Windows 10, Unity, Browsers... HTML/CSS/JS...
    31

    View Slide

  32. Recent Web Technologies
    No more Gap between Native Apps
    Install as an App, Run on Background, O ine and
    cache management, Push Noti cation, Payment,
    Web Authn...
    so called "PWA = Progressive Web Apps"
    Lower Level APIs
    Bluetooth, USB serial, Web Audio worklet, ...
    p-thread (by WASM) ...
    32

    View Slide

  33. Web standard (HTML5) is
    Most Popular Technology
    all 1st ~ 3rd popular techs are
    web (JavaScript, HTML, CSS)
    Most Popular Framework
    all 1st ~ 3rd popular frameworks
    are web (Node.js, Angular, React)
    Multi Platform Support
    OS independent implementation
    with Web technologies
    Easy to work with Web services
    Free to Use
    No license fees depends on the
    device sales nor engineers
    Patent free (except some video
    codecs if you use it)
    Vendor independent
    No worry to stop support by the
    vendor (good for long-term)
    Very rapid improvement
    by Google, Mozilla etc
    Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018
    33

    View Slide

  34. Most Popular Tech = Web Technologies
    TypeScript is meta lang of JavaScript

    Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018
    34

    View Slide

  35. Most Popular for OSS, Most In-Demand for job
    https://stackify.com/popular-programming-languages-2018/
    35

    View Slide

  36. HTML5 (browser engine directly) support
    Fine Layout
    vertical text, grid, Flexbox,
    regions, ow...
    Font features
    Variable Font, Ligature etc.

    HTML can control font like
    Adobe applications
    UI Framework
    React, Vue, Angular etc...
    2D Graphics
    use Canvas API to render
    Graph etc with GPU
    acceleratioon
    3D Graphics
    WebGL ~= OpenGL ES2
    Transform, Filter e ects ...
    zoom, 3d transform with
    matrix, blur etc also use GPU
    Binary Operation
    Binary (Blob) also supported
    WebAssembly can run almost
    same speed with Native
    Database
    use IndexedDB
    Network communications
    HTTP/2, TLS, WebSocket ...
    Video Camera and Mic (*)
    use WebRTC
    Video Codecs
    webm, vp8,9, h264, ogg
    theora ...
    RZ/G support H.264
    hardware decode (*)
    Audio Codecs
    opus, ac, mp3, wav ...
    Image les
    png, jpeg, svg, gif, apng, ...
    Module
    UI component and JS
    libraries are provided as
    module
    multi thread
    use multi core cpus
    note: p-thread is under dev
    (*) under development / test
    36

    View Slide

  37. Web support complex layouts for books
    2017 IDPF join W3C
    Digital Publishment format (ePub) now
    standardized in W3C
    Japanese books can now made with CSS
    CSS Secrets by O'Reilly Japan & Vivliostyle
    Complex Layout can be done with Web now
    http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja
    37

    View Slide

  38. Vertical Text Support
    http://tategaki.github.io/awards/ https://developer.mozilla.org/docs/Web/CSS/writing-mode
    38

    View Slide

  39. CSS Font Features (2011~)
    https://helpx.adobe.com/jp/typekit/using/open-type-syntax.html
    39
    Various font features are supported
    by browser (especially by Gecko)

    View Slide

  40. WebAssembly (Binary format support)
    Binary Format (!= Native Code)
    compatible with asm.js (and js)
    Super Fast "Web" program
    near C speed (faster than Java etc)
    Shorter load time (than asm.js or js)
    Compiled from many languages
    C, C++, Rust, Python etc...
    existing libraries can be ported to the Web!
    40

    View Slide

  41. WebGL2+WebAssembly (Unreal Engine 4, 2017/02~)
    Note: run on PC, not RZ/G https://www.youtube.com/watch?v=TwuIRcpeUWE
    41

    View Slide

  42. HTML5 (JavaScript) libraries support
    I18n (multi language) support
    gettext, uent etc format
    i18n API can handle time, units, etc.
    Graphics Library
    Edit or lter images, Convert formats,
    Motion detection...
    Audio Visual
    Mixer, Filter, Text-to-Speech ...
    3D/Game Engine
    Unity, Unreal Engine etc...
    AI Frameworks (TensorFlow etc)
    Face recognition, object detection,
    audio recognition etc...
    P2P network
    Video chat, P2P data...
    porting from Native
    C, Rust, Python etc can be converted
    Physical Simulation, Encryption,
    Encoder/Decoder, Data Science etc...
    other
    many many OSS libraries ...
    42

    View Slide

  43. Latest Example (Porting python to the Web)
    Mozilla is porting full standard
    Python interpreter to Wasm.
    also port data science stack,
    including NumPy, Pandas, Scipy,
    and Matplotlib.
    few times slower than native
    https://hacks.mozilla.org/2019/04/pyodide-bringing-the-scientific-python-stack-to-the-browser/
    43

    View Slide

  44. HTML5 as GUI Framework
    develop Apps with HTML5
    use as GUI Framework
    run on Gecko Engine
    use with Middle ware
    when needed to communication
    native component, use web socket
    (IP) connection to localhost or API
    enhancement of Gecko engine
    no app management (install etc)
    feature (yet)
    https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html
    44

    View Slide

  45. What HTML5 can / cannot
    45
    feature HTML5 can HTML5 cannot note
    GUI
    Any 2D, 3D Contents
    No limitations
    Embed Native GUI frame
    Such as Font features or vertical text
    support are better than Native
    frameworks like Qt
    GPU

    usage
    Animations, Filters, Layer
    Compositions, Video decode etc. can
    be done with GPU acceleration
    Cannot use GPU directly.

    Cannot use GPU speci c features
    Shaders etc can be used exactly same
    as OpenGL ES2,3
    performan
    ce
    Near Native (C lang) with WASM

    (WebAssembly is faster than Java)
    JavaScript can be fast, can be slow
    depends on how you write JS
    WebAssembly usage is still limited
    App Install
    Install and manage background
    application (with Web Manifest)
    Packaged (and signed) App
    distribution and install
    Packaging as Apps are still under
    development and standardization
    other
    Push noti cation, multi thread,

    background exec
    direct access to hardwares or le
    systems, SQL database etc.
    need other native implementation if
    you need hardware speci c features

    View Slide

  46. HTML5 & Gecko Engine

    View Slide

  47. HTML5, Browser, Browser Engine
    HTML5 is a technology for the Web
    HTML5 App = Web App
    Web Browser is an App to view Web contents
    Chrome, Edge, Firefox, IE, Safari etc.
    Browser Engines execute Web Apps
    Browser - UI and User Data = Browser Engine
    Blink, EdgeHTML, Gecko, Trident, WebKit etc.
    RZ/G Veri ed Linux Package include Gecko
    47

    View Slide

  48. Browser / Engine / Contents
    Browser (Firefox ...)
    Browser Engine + UI part
    Browser Engine (Gecko ...)
    core program to load contents
    to content area (viewport)
    Contents
    Web Page / Apps
    48
    content area
    browser
    UI

    View Slide

  49. Firefox and Gecko
    Firefox: browser as a whole
    Firefox is Branding name for users
    we port Firefox (as non brand
    browser) to RZ/G but UI part is
    not veri ed nor supported
    Gecko: Browser Engine part
    core program to load web apps
    We sometimes call Layout Engine
    part as Gecko
    49
    Devtools
    Server
    Browser Engine,

    Layout Engine
    JavaScript
    Engine
    Network,
    Security
    Browser UI
    User Data Storage etc.
    Graphics
    APIs
    Devtools
    Client
    Linux BSP

    (HAL, Graphics backend etc...)
    Gecko
    Firefox

    View Slide

  50. Browser Engine (more details)
    Engine = Core parts
    Rendering, Layout,
    JavaScript, Network etc
    UI (so-called "Chrome") is
    implemented over Engine
    Web App Platform
    You can create Web Apps
    soon RZ/G with Gecko Engine
    50
    Browser Engine (Gecko, in case of Firefox)
    JavaScript Engine Necko /
    Network
    NSS
    GFX / Layout
    Thebe Azure
    Web Standard APIs
    Browser User Interface
    (so-called Chrome)
    Plug-in
    container
    NPAPI
    interpreter
    XPCOM etc...
    JIT Compilers
    Web Applications (Contents)
    Hardwares
    HTML CSS JavaScript
    Linux BSP (Kernel, Driver, Graphic Backend, Codecs etc...
    Plug-in
    Compositor
    Style Engine

    View Slide

  51. Web Browser and Browser Engine
    Browser Side
    Navigation UI

    (Location bar, back/forward
    button, tab/window switcher...)
    User Data Management such
    as bookmarks, history, saved
    password, downloads,
    extensions etc.
    Application Management
    DevTools (devtools client)
    Browser Engine Side
    Render Content and Run Apps
    exec JavaScript, Network
    management, encryption, video
    decode...
    Provide WebAPIs for Apps
    Save App data cache, o ine API
    support
    provide API for DevTools
    (devtools server)
    51

    View Slide

  52. Popular Browsers and their engines
    Renesas RZ/G support Gecko Engine to creat HTML5 Apps/HMI
    52
    Web Browser Browser Engine JavaScript Engine
    Chrome Blink V8
    Edge
    EdgeHTML

    (will switch to Chromium)
    Chakra (chakra.dll)
    Firefox Gecko SpiderMonkey
    Safari WebKit JavaScriptCore
    Internet Explorer Trident Chakra (JScript9.dll)

    View Slide

  53. https://www.mozilla.jp/blog/entry/10552/

    View Slide

  54. View Slide

  55. View Slide

  56. https://www.mozilla.jp/blog/entry/10552/

    View Slide

  57. HTML5 & Gecko Engine
    on RZ/G Linux Platform

    View Slide

  58. Why HTML5 needed for Embedded devices?
    HTML5 now become popular for various industry
    Most popular Language and Framework should be used for
    Embedded devices too
    Free to Use technology
    No worry about vendor lock-in nor patent/license issue
    Suitable platform for very long term products
    Work with Internet and the Web
    Share same client code for embedded devices and PC,
    smart devices
    Easy to update apps when your app loaded from network
    58

    View Slide

  59. IoT products (devices with network)
    Devices with Web Tech
    Devices with Web Engine
    background: Web usage in IoT products
    In the IoT era, almost products will
    connect to the Internet and some
    of them use web technologies to
    communicate or control. Not only
    smart devices apps, various layers
    of technologies are web and Web
    engine (Browsers or JavaScript
    engine) suit for the case.
    Even edge devices like sensors
    some can be controlled from Web
    directly or via controller devices.
    59

    View Slide

  60. Where Gecko can be used?
    Just example device types...
    60
    Windows
    10
    Windows 10 IoT
    Consumer
    Prototype
    MCU MPU SoC
    Android
    & AOSP
    Realtime OS
    No OS
    Linux
    Qt
    Gecko on Linux
    (main target)
    Arduino, RasPi etc.
    Industry
    iOS
    CPU+GPU
    Makers
    mainly Devices with rich HMI
    Win10
    mobile
    macOS
    automotive
    ATM
    medical
    kiosk
    FA
    HEMS
    PC
    intercom
    signage
    sensor
    copy
    mobile
    Robot
    projector
    home
    appliance
    Nest
    AR
    echo
    TV
    STB
    PLC

    View Slide

  61. Embedded devices with Web
    Printers (Richo etc)
    touch panels use HTML5
    remotely manage custom display
    for each divisions of the company
    hardware control part of the
    main body also implemented
    with Web tech (Node.js) and work
    with browser of touch panel HMI
    Door Phone (Aiphone etc)
    Including Web Browser
    Support 3rd party app with Web
    Digital Signage
    Show HTML contents to make
    content management easy
    TVs (Almost smart TVs)
    Menu implemented with HTML
    support 3rd party apps with Web
    Apps (Japan or Euro region
    speci c apps such as Hybridcast
    and Maker speci c apps for
    Panasonic (Ajax App))
    61

    View Slide

  62. Gecko on RZ/G
    RZ/G Linux Platform support HTML5
    2 HMI platform = Qt / HTML5
    HMI development with most popular Web tech
    HMI platform without license fees
    Gecko as VLP (Veri ed Linux Package)
    port Gecko, browser engine of Firefox which lead
    web tech improvement with Chrome
    Gecko support enable GPU accelerated Web-based
    HMI, App development
    https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html
    62

    View Slide

  63. HTML5 Acceleration and SoC speci c porting
    Browser do JavaScript (main thread), Style and Layout calculation,
    Painting and Composting between each frame rendering
    If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow)
    SoC speci c porting needed to accelerate each steps
    for example, JIT engine(s) which use assembly code are needed to accelerate
    JavaScript (or about 10 times slower with interpreter execution)
    RZ/G1 use GPU for Paint (Canvas, Video) and Composite steps too!
    RZ/G1 support latest Style engine "Stylo" which is introduced from Gecko 57
    details: https://developers.google.com/web/fundamentals/performance/rendering/
    63

    View Slide

  64. HTML5 speed with/without GPU
    GPU Accelerated Compositor disabled vs enabled performance test with Gecko 52 on RZ/G1M
    64
    Fish Bowl

    (fps, 50 sh)
    Speed Reading

    (average draw)
    Fish IE Tank

    (fps, 250 sh)
    Speedometer

    (runs/min)
    0 2.5 5 7.5 10
    without GPU Compositor with GPU Compositor
    Bench mainly for Composite

    about 10 times faster
    3d benchmark with fish tank
    about 2.6 times faster
    repeatedly render 96 elements
    about 3.2 times faster
    similar to real (various) app UI
    about 1.7 times faster

    View Slide

  65. same HTML5 contents for RZ/G
    simple HMI with for touch panel
    see GPU accelerated animations and video decode
    example using Framework (React)
    like Control panel with many images or graph
    General Web Apps (for PC or mobile)
    Really complex and heavy app like Google Maps
    3D contents (HelloRacer, PhiloGL etc...)

    PlayCanvas Apps (BMW Car, Home Lift customize)
    see https://github.com/webdino/gecko-embedded/wiki/Sample-Contents
    65

    View Slide

  66. Web Engine Comparison
    based on feedback from Japanese Makers
    66
    Platform Pros Cons
    Gecko
    Free License (MPL), Edge Web tech,

    Share code, Support in Japan
    Footprint
    No Support for embedded (except RZ/G!)
    Chromium
    Free License (BSD), Edge Web tech,
    share code for PC and smart devices
    Footprint, no support in Japan

    No support for embedded
    WebKit
    Free License (BSD), Edge Web tech,
    share code for PC and smart devices
    Footprint, no support
    Less web tech support than Chromium/Gecko
    Qt WebEngine
    reuse QML code
    OS support with Qt EE
    Commercial: Too expensive

    OSS version: license (GPLv3), no support
    Android
    Performance, edge tech, OS support

    Google services support (except AOSP)
    Footprint, Too many complex update, support

    license (contract with Google)

    View Slide

  67. Memory Usage of Web apps
    Simple, hand written contents (Tanzanite)
    memory usage (of contents): 1MB 3MB
    A little Complex one with Frameworks
    memory usage: 5MB 20MB
    Complex Web App (Google Maps Lite)
    memory usage: 30MB 60MB
    not intended for Embedded devices
    In addition to this, around 100+MB needed for browser
    depends on UI part and window size etc.
    (not like android) no limit of memory for Web content. You can run 1GB over AAA game on browser
    67

    View Slide

  68. What can you use for RZ/G (currently)
    Gecko 60 Browser Engine (included in Renesas VLP)
    Runtime Engine for Web Apps
    Same engine as Firefox 60 (ESR version)
    We may support Gecko 68 too for RZ/G2 release (not sure yet)
    No brand version build of Firefox (not included in VLP)
    Full version (same as Linux desktop) of browser app
    No warranty for browser features except to load HTML content
    Too heavy browser for Embedded devices actually
    Remote debug tools included (same as PC version)
    68

    View Slide

  69. Gecko Engine for Embedded
    Full featured browser is not needed
    no or minimal UI for embedded devices needed
    use Gecko to load HTML content
    Note: need to implement wrapper apps (with XUL)
    to use Gecko to load HTML content.
    What wrapper or browser is preapred?
    Amethyst (under development, OSS, free to use)
    Corundum (commercial browser, need license fees)
    69
    content area
    browser
    UI

    View Slide

  70. What is needed?
    Wrapper or Browser App to load HTML content/apps
    full feature browser Firefox (not included in VLP) contains too many features and heavy
    In general, you need to use simple wrapper app of Gecko Engine
    We are now preparing simple minimal wrapper app "Amethyst"
    If you need device speci c Browser app, need to develop (or buy) it
    Application Management system
    Packaging Web Apps, Delivery and update via network, Signature for the package, OTA
    update system are not included in VLP and Gecko
    Communication with Native Libraries or Apps
    Web Apps cannot dynamically "linked" nor communicated with unix socket etc. Your
    native component need to support some Web API or you need enhance Gecko
    WebDINO Japan etc can provide consulting support for feasibility study and development
    70

    View Slide

  71. Amethyst (Project code name)
    Creating simplest WebViewer with Gecko
    Almost No Browser UI (No tabs, navigation bar etc)
    Minimal memory usage to load HTML
    No unintentional network connection (to Mozilla/
    Google etc servers)
    Schedule
    Developer (internal) release: End of June
    Public Release: End of Sep.
    71

    View Slide

  72. Thank You!
    Any Questions?
    Feel free to contact me:
    [email protected]
    @dynamitter, facebook.com/dynamis
    by @4687koSourijun

    View Slide