about:me Tomoya ASAI CTO Tech Speaker mail: [email protected] dynamis @ Community Twitter: @dynamitter facebook.com/dynamis slideshare.net/dynamis I LOVE Firefox = Red Panda = Petit Panda = Lesser Panda 2
about:WebDINO Japan 2017 June - name changed Mozilla Japan WebDINO Japan Web Diversity, Internationalization, Neutrality, Openness Gecko Embedded porting Gecko for Reneas RZ/G https://www.webdino.org/ 3
Topics: HTML5 on RZ/G Introduction about: HTML5 HTML5 de nition, now and future HTML5 and Gecko engine What's Gecko Engine HTML5 and Gecko on RZ/G HTML5 for IoT and RZ/G Platform 4
Point: what is HTML5? HTML5 in a Narrow sense = HTML4.next "HTML Living Standard" is the latest spec "HTML5" in a Broad sense = latest Web tech Web technologies for Application platform Multi-Platform App Environment All modern (rich) OSs support HTML5! Web developers recently use "PWA" to summarize latest web technologies, instead of HTML5 6
HTML, CSS, JavaScript = 3 basis of the Web HTML (Hyper Text Markup Language) Markup Language for the Web De ne contents and structure CSS (Cascading Style Sheet) Control appearance of contents JS (JavaScript) Control behavior of Web Apps "Java"Script come from Java because of marketing purpose (not subset of Java) 7
JavaScript example - C/Java like language $(document).ready(function(){ var search = function() { var keyword = $('#inputText').val().trim(); if (!keyword) return; var ekeyword = encodeURIComponent(keyword); var rekeyword = ekeyword.replace(/%20/g, '+'); window.location.href = '/ja-jp/search/keyword- search.html#q=' + rekeyword; }; $('#submitBtn').click(search); $('#inputText').keypress(function(e) { if (e.keyCode === 13) search(); }); }); from https://www.renesas.com/ja-jp/ 10
from birth of the HTML to HTML4 1989: 1st proposal for the Web 1990: 1st Web Browser WorldWideWeb released 1993: HTML published as Internet Draft 1994: W3C start web standardization 1995: HTML 2.0 published as RFC 1866 1997: HTML 3.2 become W3C REC 1997: HTML 4.0 become W3C REC 12
birth of WHATWG and HTML5 1998 2006: XHTML standardization (and it's fail) W3C focused to re-invent HTML with XHTML but XHTML was not widely used since it's not compatible with HTML4, not supported by IE 2004: WHATWG established Mozilla & Opera had proposed backward compatible enhancement for HTML4 but rejected by W3C. Then browser vendors start discussion in another organization, WHATWG 2007: W3C start new HTML WG W3C started HTML5 standardization based on WHATWG HTML 2014: HTML5 become W3C REC (recommendation) 13
current HTML5 = HTML Living Standard latest HTML spec = WHATWG HTML Living Standard = version-less spec Browser vendors only refer HTML living standard de ned by WHATWG, not W3C HTML5.x snapshot version = W3C HTML 5.x W3C continue to publish HTML5.x year by year based on WHATWG HTML living standard small di erence exists but implementors ignore it 14
summary: (narrow) HTML5 = HTML4.next not re-invent the Web, incremental update HTML4++ = HTML5 backward compatible, step-by-step improvement 1st step for next gen tech HTML5 start to summarize already implemented non-standard (browser speci c) technologies for Web Applications 1st step to create Large Web Apps 15
WHATWG HTML spec - Introduction https://html.spec.whatwg.org/multipage/introduction.html#is-this-html5? 17 “the term "HTML5" is widely used as a buzzword to refer to modern Web technologies”
"HTML5" as "Buzzword" HTML was buzzword in late 2000s starting from Gmail (2004), Google Maps (2005), many desktop apps switch to Web apps from 2008, Web is treated as OS independent platform to support new platforms such as iOS and Android. "Plateau of Productivity" now after too much expectations and disappointment, now widely used properly in many industries buzzword: a word or phrase that becomes very popular for a period of time 18
HTML5 in Hype Cycle by Gartner (2012) https://www.gartner.co.jp/press/html/pr20120906-01.html 19 HTML5 expectations for HTML5 became max around 2012 Big Data or IoT etc. follow HTML5 in 2012
HTML5 for Desktop PC Platform 2004-2005 Gmail, Google Maps after Gmail (2004), Google Maps (2005), Flash/Java apps start to switch to HTML5 2014 Atom, Electron GitHub released Atom text editor and Electron framework to create desktop apps like Atom with Node.js and Chromium (and it become popular) 2014 Unity 5.3 support HTML5 (WebGL) AAA games can now run on browsers FYI: XUL Runner with Gecko was used before Electron 20
AAA 3D Games come to the Web Unity 5 start to support WebGL Unity 5.3 o cially suppoprt WebGL Unity Web Player Plug-in become EOL C# => C => asm.js about 1/2 speed of Native Unity (2016) Unreal Engine 3 support WebGL Unreal Engine 4.9 include WebGL export by default about 2/3 speed of Native (2014) 21
HTML5 for Mobile Platform ~2011: highest expectation for HTML5 mainly as "hybrid apps" with Cordova to support both iOS and Android with single code 2012: disappointed iOS Facebook app was too slow (because of limitation of WebView of iOS, no problem for Android actually...) 2015~: HTML5 (PWA) become popular again Web app (currently called PWA) and new hybrid frameworks such as React Native become popular 23
Web App on 1st iPhone Steve Jobs Keynote MacWorld 2007 SF https://www.youtube.com/watch?v=P-a_R6ewrmM 24 “The full Safari engine is inside of iPhone. And so, you can write amazing Web 2.0 and Ajax apps that look exactly and behave exactly like apps on the iPhone.”
Web App on Mobile (smartphone, ~2011) 2007.01 iPhone release 3rd Party apps are all Web apps (no native app support) iPhone = phone + iPod + internet client 2008.08 App Store start support native 3rd Party apps 2008.09 Android 1.0 release Android Market start on 2008.10 along with HTC Dream around 2010: hybrid apps become trend WebView was widely used to make HTML5 apps 25
when new iOS (native) Facebook app release Mark Zuckerberg at TechCrunch Disrupt SF 2012 https://www.youtube.com/watch?v=P-a_R6ewrmM 26 “... the biggest mistake that we made, as a company, is betting too much on HTML5 as opposed to native …”
Web App on Mobile (smartphone, 2012~) 2012.08: Facebook App switch to Native Hybrid version facebook app was too slow on iOS and they switch to Native implementation ~2015-2016 Native apps become popular on Google I/O 2016, PWA was introduced and Android apps start to use Web widely again... 2018.03: iOS 11.3 support PWA O -line, Install, Payment API support for Web apps. This is consumer app trend. Majority of Enterprise apps already become hybrid/html5/pwa apps. 27
sorry, a little old and include deprecated specs... http://webapi.link 29 Net Info Web Socket APNG Full- screen WebRTC HTML Components ECMAScript 2015 CSS Snapshot 2015 WHATWG W3C HTML5 URL HTTP TLS Element s & WAI- ARIA HTML5 Parser Web Workers Web Socket Canvas 2D multi media Content Model app cache section s HTML5 Forms Server- Sent ev. geo- locatio Orient ation Notifi- Game- pad Media data channe Presen - Push SSL Media Recorder Opus mp3 H. 264 Service Worker Filter SVG WOFF Math ML RSS Layout Media trans- form Tran- sitions & Anima- tions Flex Box Multi Colum Fonts User Inter- Shapes text decora- Indexed DB vibra- tion SMIL VP10 HSTS CORS RDF RDFa DNT VP9 Web Open Media VP8 HTTP/2 PNG Web Audio Batter y Generi c Shared Workers SIMD Web Crypto temp- lates Shadow DOM Custom Element s Page Visibilit FIDO Grid Layout File Pointer MIDI Speech Pro- mise CSP Class Modul e block scope Typed Array Arrow Func. Templat e Gene- rators Async Func. Navi. Timing Beacon Exclu- sions Writing Mode Web Anima- tions IETF Khronos OGP Game, VR & High Performance Apps Progressive Web Apps Real-Time Communications Device Control Extensible for Anything! Micro- data data ORTC WebVR Shared Mem. Itera- tors Proxy Collec- tions asm.js Permi- ssions DOM WebGL fetch Web Assembl y EME Brotli Mask- ing Progressive Enhancement Web App Manifest Web Storage Quarks Mode UI Events QUIC Origi Blue- tooth USB HPKP
Web Standard Support (2018.04) API numbers counted with Web IDL https://github.com/MicrosoftEdge/APICatalogData 30 Chrome and Firefox (Gecko) lead web standard implementations 14,194 10,225 7,537 8,706 11,764
Recent Web Technologies No more Gap between Native Apps Install as an App, Run on Background, O ine and cache management, Push Noti cation, Payment, Web Authn... so called "PWA = Progressive Web Apps" Lower Level APIs Bluetooth, USB serial, Web Audio worklet, ... p-thread (by WASM) ... 32
Web standard (HTML5) is Most Popular Technology all 1st ~ 3rd popular techs are web (JavaScript, HTML, CSS) Most Popular Framework all 1st ~ 3rd popular frameworks are web (Node.js, Angular, React) Multi Platform Support OS independent implementation with Web technologies Easy to work with Web services Free to Use No license fees depends on the device sales nor engineers Patent free (except some video codecs if you use it) Vendor independent No worry to stop support by the vendor (good for long-term) Very rapid improvement by Google, Mozilla etc Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 33
Most Popular Tech = Web Technologies TypeScript is meta lang of JavaScript Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 34
HTML5 (browser engine directly) support Fine Layout vertical text, grid, Flexbox, regions, ow... Font features Variable Font, Ligature etc. HTML can control font like Adobe applications UI Framework React, Vue, Angular etc... 2D Graphics use Canvas API to render Graph etc with GPU acceleratioon 3D Graphics WebGL ~= OpenGL ES2 Transform, Filter e ects ... zoom, 3d transform with matrix, blur etc also use GPU Binary Operation Binary (Blob) also supported WebAssembly can run almost same speed with Native Database use IndexedDB Network communications HTTP/2, TLS, WebSocket ... Video Camera and Mic (*) use WebRTC Video Codecs webm, vp8,9, h264, ogg theora ... RZ/G support H.264 hardware decode (*) Audio Codecs opus, ac, mp3, wav ... Image les png, jpeg, svg, gif, apng, ... Module UI component and JS libraries are provided as module multi thread use multi core cpus note: p-thread is under dev (*) under development / test 36
Web support complex layouts for books 2017 IDPF join W3C Digital Publishment format (ePub) now standardized in W3C Japanese books can now made with CSS CSS Secrets by O'Reilly Japan & Vivliostyle Complex Layout can be done with Web now http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja 37
CSS Font Features (2011~) https://helpx.adobe.com/jp/typekit/using/open-type-syntax.html 39 Various font features are supported by browser (especially by Gecko)
WebAssembly (Binary format support) Binary Format (!= Native Code) compatible with asm.js (and js) Super Fast "Web" program near C speed (faster than Java etc) Shorter load time (than asm.js or js) Compiled from many languages C, C++, Rust, Python etc... existing libraries can be ported to the Web! 40
HTML5 (JavaScript) libraries support I18n (multi language) support gettext, uent etc format i18n API can handle time, units, etc. Graphics Library Edit or lter images, Convert formats, Motion detection... Audio Visual Mixer, Filter, Text-to-Speech ... 3D/Game Engine Unity, Unreal Engine etc... AI Frameworks (TensorFlow etc) Face recognition, object detection, audio recognition etc... P2P network Video chat, P2P data... porting from Native C, Rust, Python etc can be converted Physical Simulation, Encryption, Encoder/Decoder, Data Science etc... other many many OSS libraries ... 42
Latest Example (Porting python to the Web) Mozilla is porting full standard Python interpreter to Wasm. also port data science stack, including NumPy, Pandas, Scipy, and Matplotlib. few times slower than native https://hacks.mozilla.org/2019/04/pyodide-bringing-the-scientific-python-stack-to-the-browser/ 43
HTML5 as GUI Framework develop Apps with HTML5 use as GUI Framework run on Gecko Engine use with Middle ware when needed to communication native component, use web socket (IP) connection to localhost or API enhancement of Gecko engine no app management (install etc) feature (yet) https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 44
What HTML5 can / cannot 45 feature HTML5 can HTML5 cannot note GUI Any 2D, 3D Contents No limitations Embed Native GUI frame Such as Font features or vertical text support are better than Native frameworks like Qt GPU usage Animations, Filters, Layer Compositions, Video decode etc. can be done with GPU acceleration Cannot use GPU directly. Cannot use GPU speci c features Shaders etc can be used exactly same as OpenGL ES2,3 performan ce Near Native (C lang) with WASM (WebAssembly is faster than Java) JavaScript can be fast, can be slow depends on how you write JS WebAssembly usage is still limited App Install Install and manage background application (with Web Manifest) Packaged (and signed) App distribution and install Packaging as Apps are still under development and standardization other Push noti cation, multi thread, background exec direct access to hardwares or le systems, SQL database etc. need other native implementation if you need hardware speci c features
HTML5, Browser, Browser Engine HTML5 is a technology for the Web HTML5 App = Web App Web Browser is an App to view Web contents Chrome, Edge, Firefox, IE, Safari etc. Browser Engines execute Web Apps Browser - UI and User Data = Browser Engine Blink, EdgeHTML, Gecko, Trident, WebKit etc. RZ/G Veri ed Linux Package include Gecko 47
Firefox and Gecko Firefox: browser as a whole Firefox is Branding name for users we port Firefox (as non brand browser) to RZ/G but UI part is not veri ed nor supported Gecko: Browser Engine part core program to load web apps We sometimes call Layout Engine part as Gecko 49 Devtools Server Browser Engine, Layout Engine JavaScript Engine Network, Security Browser UI User Data Storage etc. Graphics APIs Devtools Client Linux BSP (HAL, Graphics backend etc...) Gecko Firefox
Browser Engine (more details) Engine = Core parts Rendering, Layout, JavaScript, Network etc UI (so-called "Chrome") is implemented over Engine Web App Platform You can create Web Apps soon RZ/G with Gecko Engine 50 Browser Engine (Gecko, in case of Firefox) JavaScript Engine Necko / Network NSS GFX / Layout Thebe Azure Web Standard APIs Browser User Interface (so-called Chrome) Plug-in container NPAPI interpreter XPCOM etc... JIT Compilers Web Applications (Contents) Hardwares HTML CSS JavaScript Linux BSP (Kernel, Driver, Graphic Backend, Codecs etc... Plug-in Compositor Style Engine
Web Browser and Browser Engine Browser Side Navigation UI (Location bar, back/forward button, tab/window switcher...) User Data Management such as bookmarks, history, saved password, downloads, extensions etc. Application Management DevTools (devtools client) Browser Engine Side Render Content and Run Apps exec JavaScript, Network management, encryption, video decode... Provide WebAPIs for Apps Save App data cache, o ine API support provide API for DevTools (devtools server) 51
Popular Browsers and their engines Renesas RZ/G support Gecko Engine to creat HTML5 Apps/HMI 52 Web Browser Browser Engine JavaScript Engine Chrome Blink V8 Edge EdgeHTML (will switch to Chromium) Chakra (chakra.dll) Firefox Gecko SpiderMonkey Safari WebKit JavaScriptCore Internet Explorer Trident Chakra (JScript9.dll)
Why HTML5 needed for Embedded devices? HTML5 now become popular for various industry Most popular Language and Framework should be used for Embedded devices too Free to Use technology No worry about vendor lock-in nor patent/license issue Suitable platform for very long term products Work with Internet and the Web Share same client code for embedded devices and PC, smart devices Easy to update apps when your app loaded from network 58
IoT products (devices with network) Devices with Web Tech Devices with Web Engine background: Web usage in IoT products In the IoT era, almost products will connect to the Internet and some of them use web technologies to communicate or control. Not only smart devices apps, various layers of technologies are web and Web engine (Browsers or JavaScript engine) suit for the case. Even edge devices like sensors some can be controlled from Web directly or via controller devices. 59
Where Gecko can be used? Just example device types... 60 Windows 10 Windows 10 IoT Consumer Prototype MCU MPU SoC Android & AOSP Realtime OS No OS Linux Qt Gecko on Linux (main target) Arduino, RasPi etc. Industry iOS CPU+GPU Makers mainly Devices with rich HMI Win10 mobile macOS automotive ATM medical kiosk FA HEMS PC intercom signage sensor copy mobile Robot projector home appliance Nest AR echo TV STB PLC
Embedded devices with Web Printers (Richo etc) touch panels use HTML5 remotely manage custom display for each divisions of the company hardware control part of the main body also implemented with Web tech (Node.js) and work with browser of touch panel HMI Door Phone (Aiphone etc) Including Web Browser Support 3rd party app with Web Digital Signage Show HTML contents to make content management easy TVs (Almost smart TVs) Menu implemented with HTML support 3rd party apps with Web Apps (Japan or Euro region speci c apps such as Hybridcast and Maker speci c apps for Panasonic (Ajax App)) 61
Gecko on RZ/G RZ/G Linux Platform support HTML5 2 HMI platform = Qt / HTML5 HMI development with most popular Web tech HMI platform without license fees Gecko as VLP (Veri ed Linux Package) port Gecko, browser engine of Firefox which lead web tech improvement with Chrome Gecko support enable GPU accelerated Web-based HMI, App development https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 62
HTML5 Acceleration and SoC speci c porting Browser do JavaScript (main thread), Style and Layout calculation, Painting and Composting between each frame rendering If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow) SoC speci c porting needed to accelerate each steps for example, JIT engine(s) which use assembly code are needed to accelerate JavaScript (or about 10 times slower with interpreter execution) RZ/G1 use GPU for Paint (Canvas, Video) and Composite steps too! RZ/G1 support latest Style engine "Stylo" which is introduced from Gecko 57 details: https://developers.google.com/web/fundamentals/performance/rendering/ 63
HTML5 speed with/without GPU GPU Accelerated Compositor disabled vs enabled performance test with Gecko 52 on RZ/G1M 64 Fish Bowl (fps, 50 sh) Speed Reading (average draw) Fish IE Tank (fps, 250 sh) Speedometer (runs/min) 0 2.5 5 7.5 10 without GPU Compositor with GPU Compositor Bench mainly for Composite about 10 times faster 3d benchmark with fish tank about 2.6 times faster repeatedly render 96 elements about 3.2 times faster similar to real (various) app UI about 1.7 times faster
same HTML5 contents for RZ/G simple HMI with for touch panel see GPU accelerated animations and video decode example using Framework (React) like Control panel with many images or graph General Web Apps (for PC or mobile) Really complex and heavy app like Google Maps 3D contents (HelloRacer, PhiloGL etc...) PlayCanvas Apps (BMW Car, Home Lift customize) see https://github.com/webdino/gecko-embedded/wiki/Sample-Contents 65
Web Engine Comparison based on feedback from Japanese Makers 66 Platform Pros Cons Gecko Free License (MPL), Edge Web tech, Share code, Support in Japan Footprint No Support for embedded (except RZ/G!) Chromium Free License (BSD), Edge Web tech, share code for PC and smart devices Footprint, no support in Japan No support for embedded WebKit Free License (BSD), Edge Web tech, share code for PC and smart devices Footprint, no support Less web tech support than Chromium/Gecko Qt WebEngine reuse QML code OS support with Qt EE Commercial: Too expensive OSS version: license (GPLv3), no support Android Performance, edge tech, OS support Google services support (except AOSP) Footprint, Too many complex update, support license (contract with Google)
Memory Usage of Web apps Simple, hand written contents (Tanzanite) memory usage (of contents): 1MB 3MB A little Complex one with Frameworks memory usage: 5MB 20MB Complex Web App (Google Maps Lite) memory usage: 30MB 60MB not intended for Embedded devices In addition to this, around 100+MB needed for browser depends on UI part and window size etc. (not like android) no limit of memory for Web content. You can run 1GB over AAA game on browser 67
What can you use for RZ/G (currently) Gecko 60 Browser Engine (included in Renesas VLP) Runtime Engine for Web Apps Same engine as Firefox 60 (ESR version) We may support Gecko 68 too for RZ/G2 release (not sure yet) No brand version build of Firefox (not included in VLP) Full version (same as Linux desktop) of browser app No warranty for browser features except to load HTML content Too heavy browser for Embedded devices actually Remote debug tools included (same as PC version) 68
Gecko Engine for Embedded Full featured browser is not needed no or minimal UI for embedded devices needed use Gecko to load HTML content Note: need to implement wrapper apps (with XUL) to use Gecko to load HTML content. What wrapper or browser is preapred? Amethyst (under development, OSS, free to use) Corundum (commercial browser, need license fees) 69 content area browser UI
What is needed? Wrapper or Browser App to load HTML content/apps full feature browser Firefox (not included in VLP) contains too many features and heavy In general, you need to use simple wrapper app of Gecko Engine We are now preparing simple minimal wrapper app "Amethyst" If you need device speci c Browser app, need to develop (or buy) it Application Management system Packaging Web Apps, Delivery and update via network, Signature for the package, OTA update system are not included in VLP and Gecko Communication with Native Libraries or Apps Web Apps cannot dynamically "linked" nor communicated with unix socket etc. Your native component need to support some Web API or you need enhance Gecko WebDINO Japan etc can provide consulting support for feasibility study and development 70
Amethyst (Project code name) Creating simplest WebViewer with Gecko Almost No Browser UI (No tabs, navigation bar etc) Minimal memory usage to load HTML No unintentional network connection (to Mozilla/ Google etc servers) Schedule Developer (internal) release: End of June Public Release: End of Sep. 71
dynamis
ad5jp
tamaiyutaro
oracle4engineer
mamohacy
nikkei_engineer_recruiting
koic
odasho
carta_engineering
chiroruxx
katzueno
jacopen
oogfranz
soudai
notwaldorf
trallard
myddelton
jacobian
tanoku
jlugia
scottboms
keathley
quramy
destraynor
sugarenia