HTML4.next "HTML Living Standard" is the latest spec "HTML5" in a Broad sense = latest Web tech Web technologies for Application platform Multi-Platform App Environment All modern (rich) OSs support HTML5! Web developers recently use "PWA" to summarize latest web technologies, instead of HTML5 6
(Hyper Text Markup Language) Markup Language for the Web De ne contents and structure CSS (Cascading Style Sheet) Control appearance of contents JS (JavaScript) Control behavior of Web Apps "Java"Script come from Java because of marketing purpose (not subset of Java) 7
for the Web 1990: 1st Web Browser WorldWideWeb released 1993: HTML published as Internet Draft 1994: W3C start web standardization 1995: HTML 2.0 published as RFC 1866 1997: HTML 3.2 become W3C REC 1997: HTML 4.0 become W3C REC 12
it's fail) W3C focused to re-invent HTML with XHTML but XHTML was not widely used since it's not compatible with HTML4, not supported by IE 2004: WHATWG established Mozilla & Opera had proposed backward compatible enhancement for HTML4 but rejected by W3C. Then browser vendors start discussion in another organization, WHATWG 2007: W3C start new HTML WG W3C started HTML5 standardization based on WHATWG HTML 2014: HTML5 become W3C REC (recommendation) 13
WHATWG HTML Living Standard = version-less spec Browser vendors only refer HTML living standard de ned by WHATWG, not W3C HTML5.x snapshot version = W3C HTML 5.x W3C continue to publish HTML5.x year by year based on WHATWG HTML living standard small di erence exists but implementors ignore it 14
update HTML4++ = HTML5 backward compatible, step-by-step improvement 1st step for next gen tech HTML5 start to summarize already implemented non-standard (browser speci c) technologies for Web Applications 1st step to create Large Web Apps 15
from Gmail (2004), Google Maps (2005), many desktop apps switch to Web apps from 2008, Web is treated as OS independent platform to support new platforms such as iOS and Android. "Plateau of Productivity" now after too much expectations and disappointment, now widely used properly in many industries buzzword: a word or phrase that becomes very popular for a period of time 18
Gmail (2004), Google Maps (2005), Flash/Java apps start to switch to HTML5 2014 Atom, Electron GitHub released Atom text editor and Electron framework to create desktop apps like Atom with Node.js and Chromium (and it become popular) 2014 Unity 5.3 support HTML5 (WebGL) AAA games can now run on browsers FYI: XUL Runner with Gecko was used before Electron 20
to support WebGL Unity 5.3 o cially suppoprt WebGL Unity Web Player Plug-in become EOL C# => C => asm.js about 1/2 speed of Native Unity (2016) Unreal Engine 3 support WebGL Unreal Engine 4.9 include WebGL export by default about 2/3 speed of Native (2014) 21
as "hybrid apps" with Cordova to support both iOS and Android with single code 2012: disappointed iOS Facebook app was too slow (because of limitation of WebView of iOS, no problem for Android actually...) 2015~: HTML5 (PWA) become popular again Web app (currently called PWA) and new hybrid frameworks such as React Native become popular 23
SF https://www.youtube.com/watch?v=P-a_R6ewrmM 24 “The full Safari engine is inside of iPhone. And so, you can write amazing Web 2.0 and Ajax apps that look exactly and behave exactly like apps on the iPhone.”
Party apps are all Web apps (no native app support) iPhone = phone + iPod + internet client 2008.08 App Store start support native 3rd Party apps 2008.09 Android 1.0 release Android Market start on 2008.10 along with HTC Dream around 2010: hybrid apps become trend WebView was widely used to make HTML5 apps 25
TechCrunch Disrupt SF 2012 https://www.youtube.com/watch?v=P-a_R6ewrmM 26 “... the biggest mistake that we made, as a company, is betting too much on HTML5 as opposed to native …”
to Native Hybrid version facebook app was too slow on iOS and they switch to Native implementation ~2015-2016 Native apps become popular on Google I/O 2016, PWA was introduced and Android apps start to use Web widely again... 2018.03: iOS 11.3 support PWA O -line, Install, Payment API support for Web apps. This is consumer app trend. Majority of Enterprise apps already become hybrid/html5/pwa apps. 27
Net Info Web Socket APNG Full- screen WebRTC HTML Components ECMAScript 2015 CSS Snapshot 2015 WHATWG W3C HTML5 URL HTTP TLS Element s & WAI- ARIA HTML5 Parser Web Workers Web Socket Canvas 2D multi media Content Model app cache section s HTML5 Forms Server- Sent ev. geo- locatio Orient ation Notifi- Game- pad Media data channe Presen - Push SSL Media Recorder Opus mp3 H. 264 Service Worker Filter SVG WOFF Math ML RSS Layout Media trans- form Tran- sitions & Anima- tions Flex Box Multi Colum Fonts User Inter- Shapes text decora- Indexed DB vibra- tion SMIL VP10 HSTS CORS RDF RDFa DNT VP9 Web Open Media VP8 HTTP/2 PNG Web Audio Batter y Generi c Shared Workers SIMD Web Crypto temp- lates Shadow DOM Custom Element s Page Visibilit FIDO Grid Layout File Pointer MIDI Speech Pro- mise CSP Class Modul e block scope Typed Array Arrow Func. Templat e Gene- rators Async Func. Navi. Timing Beacon Exclu- sions Writing Mode Web Anima- tions IETF Khronos OGP Game, VR & High Performance Apps Progressive Web Apps Real-Time Communications Device Control Extensible for Anything! Micro- data data ORTC WebVR Shared Mem. Itera- tors Proxy Collec- tions asm.js Permi- ssions DOM WebGL fetch Web Assembl y EME Brotli Mask- ing Progressive Enhancement Web App Manifest Web Storage Quarks Mode UI Events QUIC Origi Blue- tooth USB HPKP
as an App, Run on Background, O ine and cache management, Push Noti cation, Payment, Web Authn... so called "PWA = Progressive Web Apps" Lower Level APIs Bluetooth, USB serial, Web Audio worklet, ... p-thread (by WASM) ... 32
3rd popular techs are web (JavaScript, HTML, CSS) Most Popular Framework all 1st ~ 3rd popular frameworks are web (Node.js, Angular, React) Multi Platform Support OS independent implementation with Web technologies Easy to work with Web services Free to Use No license fees depends on the device sales nor engineers Patent free (except some video codecs if you use it) Vendor independent No worry to stop support by the vendor (good for long-term) Very rapid improvement by Google, Mozilla etc Stack Over Flow Developer Survey Results 2018 https://insights.stackoverflow.com/survey/2018 33
Flexbox, regions, ow... Font features Variable Font, Ligature etc. HTML can control font like Adobe applications UI Framework React, Vue, Angular etc... 2D Graphics use Canvas API to render Graph etc with GPU acceleratioon 3D Graphics WebGL ~= OpenGL ES2 Transform, Filter e ects ... zoom, 3d transform with matrix, blur etc also use GPU Binary Operation Binary (Blob) also supported WebAssembly can run almost same speed with Native Database use IndexedDB Network communications HTTP/2, TLS, WebSocket ... Video Camera and Mic (*) use WebRTC Video Codecs webm, vp8,9, h264, ogg theora ... RZ/G support H.264 hardware decode (*) Audio Codecs opus, ac, mp3, wav ... Image les png, jpeg, svg, gif, apng, ... Module UI component and JS libraries are provided as module multi thread use multi core cpus note: p-thread is under dev (*) under development / test 36
Digital Publishment format (ePub) now standardized in W3C Japanese books can now made with CSS CSS Secrets by O'Reilly Japan & Vivliostyle Complex Layout can be done with Web now http://www.oreilly.co.jp/books/9784873117669/ https://www.w3.org/2016/05/digpub.html.ja 37
with asm.js (and js) Super Fast "Web" program near C speed (faster than Java etc) Shorter load time (than asm.js or js) Compiled from many languages C, C++, Rust, Python etc... existing libraries can be ported to the Web! 40
etc format i18n API can handle time, units, etc. Graphics Library Edit or lter images, Convert formats, Motion detection... Audio Visual Mixer, Filter, Text-to-Speech ... 3D/Game Engine Unity, Unreal Engine etc... AI Frameworks (TensorFlow etc) Face recognition, object detection, audio recognition etc... P2P network Video chat, P2P data... porting from Native C, Rust, Python etc can be converted Physical Simulation, Encryption, Encoder/Decoder, Data Science etc... other many many OSS libraries ... 42
full standard Python interpreter to Wasm. also port data science stack, including NumPy, Pandas, Scipy, and Matplotlib. few times slower than native https://hacks.mozilla.org/2019/04/pyodide-bringing-the-scientific-python-stack-to-the-browser/ 43
GUI Framework run on Gecko Engine use with Middle ware when needed to communication native component, use web socket (IP) connection to localhost or API enhancement of Gecko engine no app management (install etc) feature (yet) https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 44
cannot note GUI Any 2D, 3D Contents No limitations Embed Native GUI frame Such as Font features or vertical text support are better than Native frameworks like Qt GPU usage Animations, Filters, Layer Compositions, Video decode etc. can be done with GPU acceleration Cannot use GPU directly. Cannot use GPU speci c features Shaders etc can be used exactly same as OpenGL ES2,3 performan ce Near Native (C lang) with WASM (WebAssembly is faster than Java) JavaScript can be fast, can be slow depends on how you write JS WebAssembly usage is still limited App Install Install and manage background application (with Web Manifest) Packaged (and signed) App distribution and install Packaging as Apps are still under development and standardization other Push noti cation, multi thread, background exec direct access to hardwares or le systems, SQL database etc. need other native implementation if you need hardware speci c features
Web HTML5 App = Web App Web Browser is an App to view Web contents Chrome, Edge, Firefox, IE, Safari etc. Browser Engines execute Web Apps Browser - UI and User Data = Browser Engine Blink, EdgeHTML, Gecko, Trident, WebKit etc. RZ/G Veri ed Linux Package include Gecko 47
Branding name for users we port Firefox (as non brand browser) to RZ/G but UI part is not veri ed nor supported Gecko: Browser Engine part core program to load web apps We sometimes call Layout Engine part as Gecko 49 Devtools Server Browser Engine, Layout Engine JavaScript Engine Network, Security Browser UI User Data Storage etc. Graphics APIs Devtools Client Linux BSP (HAL, Graphics backend etc...) Gecko Firefox
JavaScript, Network etc UI (so-called "Chrome") is implemented over Engine Web App Platform You can create Web Apps soon RZ/G with Gecko Engine 50 Browser Engine (Gecko, in case of Firefox) JavaScript Engine Necko / Network NSS GFX / Layout Thebe Azure Web Standard APIs Browser User Interface (so-called Chrome) Plug-in container NPAPI interpreter XPCOM etc... JIT Compilers Web Applications (Contents) Hardwares HTML CSS JavaScript Linux BSP (Kernel, Driver, Graphic Backend, Codecs etc... Plug-in Compositor Style Engine
bar, back/forward button, tab/window switcher...) User Data Management such as bookmarks, history, saved password, downloads, extensions etc. Application Management DevTools (devtools client) Browser Engine Side Render Content and Run Apps exec JavaScript, Network management, encryption, video decode... Provide WebAPIs for Apps Save App data cache, o ine API support provide API for DevTools (devtools server) 51
to creat HTML5 Apps/HMI 52 Web Browser Browser Engine JavaScript Engine Chrome Blink V8 Edge EdgeHTML (will switch to Chromium) Chakra (chakra.dll) Firefox Gecko SpiderMonkey Safari WebKit JavaScriptCore Internet Explorer Trident Chakra (JScript9.dll)
for various industry Most popular Language and Framework should be used for Embedded devices too Free to Use technology No worry about vendor lock-in nor patent/license issue Suitable platform for very long term products Work with Internet and the Web Share same client code for embedded devices and PC, smart devices Easy to update apps when your app loaded from network 58
with Web Engine background: Web usage in IoT products In the IoT era, almost products will connect to the Internet and some of them use web technologies to communicate or control. Not only smart devices apps, various layers of technologies are web and Web engine (Browsers or JavaScript engine) suit for the case. Even edge devices like sensors some can be controlled from Web directly or via controller devices. 59
Windows 10 Windows 10 IoT Consumer Prototype MCU MPU SoC Android & AOSP Realtime OS No OS Linux Qt Gecko on Linux (main target) Arduino, RasPi etc. Industry iOS CPU+GPU Makers mainly Devices with rich HMI Win10 mobile macOS automotive ATM medical kiosk FA HEMS PC intercom signage sensor copy mobile Robot projector home appliance Nest AR echo TV STB PLC
HTML5 remotely manage custom display for each divisions of the company hardware control part of the main body also implemented with Web tech (Node.js) and work with browser of touch panel HMI Door Phone (Aiphone etc) Including Web Browser Support 3rd party app with Web Digital Signage Show HTML contents to make content management easy TVs (Almost smart TVs) Menu implemented with HTML support 3rd party apps with Web Apps (Japan or Euro region speci c apps such as Hybridcast and Maker speci c apps for Panasonic (Ajax App)) 61
platform = Qt / HTML5 HMI development with most popular Web tech HMI platform without license fees Gecko as VLP (Veri ed Linux Package) port Gecko, browser engine of Firefox which lead web tech improvement with Chrome Gecko support enable GPU accelerated Web-based HMI, App development https://www.renesas.com/ja-jp/products/rzg-linux-platform/linux-package.html 62
(main thread), Style and Layout calculation, Painting and Composting between each frame rendering If some of these are too heavy, cannot keep 60 (or 30) fps (= looks slow) SoC speci c porting needed to accelerate each steps for example, JIT engine(s) which use assembly code are needed to accelerate JavaScript (or about 10 times slower with interpreter execution) RZ/G1 use GPU for Paint (Canvas, Video) and Composite steps too! RZ/G1 support latest Style engine "Stylo" which is introduced from Gecko 57 details: https://developers.google.com/web/fundamentals/performance/rendering/ 63
performance test with Gecko 52 on RZ/G1M 64 Fish Bowl (fps, 50 sh) Speed Reading (average draw) Fish IE Tank (fps, 250 sh) Speedometer (runs/min) 0 2.5 5 7.5 10 without GPU Compositor with GPU Compositor Bench mainly for Composite about 10 times faster 3d benchmark with fish tank about 2.6 times faster repeatedly render 96 elements about 3.2 times faster similar to real (various) app UI about 1.7 times faster
panel see GPU accelerated animations and video decode example using Framework (React) like Control panel with many images or graph General Web Apps (for PC or mobile) Really complex and heavy app like Google Maps 3D contents (HelloRacer, PhiloGL etc...) PlayCanvas Apps (BMW Car, Home Lift customize) see https://github.com/webdino/gecko-embedded/wiki/Sample-Contents 65
Platform Pros Cons Gecko Free License (MPL), Edge Web tech, Share code, Support in Japan Footprint No Support for embedded (except RZ/G!) Chromium Free License (BSD), Edge Web tech, share code for PC and smart devices Footprint, no support in Japan No support for embedded WebKit Free License (BSD), Edge Web tech, share code for PC and smart devices Footprint, no support Less web tech support than Chromium/Gecko Qt WebEngine reuse QML code OS support with Qt EE Commercial: Too expensive OSS version: license (GPLv3), no support Android Performance, edge tech, OS support Google services support (except AOSP) Footprint, Too many complex update, support license (contract with Google)
memory usage (of contents): 1MB 3MB A little Complex one with Frameworks memory usage: 5MB 20MB Complex Web App (Google Maps Lite) memory usage: 30MB 60MB not intended for Embedded devices In addition to this, around 100+MB needed for browser depends on UI part and window size etc. (not like android) no limit of memory for Web content. You can run 1GB over AAA game on browser 67
Engine (included in Renesas VLP) Runtime Engine for Web Apps Same engine as Firefox 60 (ESR version) We may support Gecko 68 too for RZ/G2 release (not sure yet) No brand version build of Firefox (not included in VLP) Full version (same as Linux desktop) of browser app No warranty for browser features except to load HTML content Too heavy browser for Embedded devices actually Remote debug tools included (same as PC version) 68
no or minimal UI for embedded devices needed use Gecko to load HTML content Note: need to implement wrapper apps (with XUL) to use Gecko to load HTML content. What wrapper or browser is preapred? Amethyst (under development, OSS, free to use) Corundum (commercial browser, need license fees) 69 content area browser UI
content/apps full feature browser Firefox (not included in VLP) contains too many features and heavy In general, you need to use simple wrapper app of Gecko Engine We are now preparing simple minimal wrapper app "Amethyst" If you need device speci c Browser app, need to develop (or buy) it Application Management system Packaging Web Apps, Delivery and update via network, Signature for the package, OTA update system are not included in VLP and Gecko Communication with Native Libraries or Apps Web Apps cannot dynamically "linked" nor communicated with unix socket etc. Your native component need to support some Web API or you need enhance Gecko WebDINO Japan etc can provide consulting support for feasibility study and development 70
No Browser UI (No tabs, navigation bar etc) Minimal memory usage to load HTML No unintentional network connection (to Mozilla/ Google etc servers) Schedule Developer (internal) release: End of June Public Release: End of Sep. 71
dynamis
chanyou0311
bbrbbq
kniino
hogelog
cyberagentdevelopers
hiyanger
abemii
shigashiyama
tacck
shumei_ito
nssv
cmusudakeisuke
bcantrill
matthewcrist
destraynor
kevinliebholz
lauravandoore
bluesmoon
gr2m
davidbonilla
pedronauck
caitiem20
iamctodd
kastner
![about:me Tomoya ASAI CTO Tech Speaker mail: [email protected] dynamis @](https://files.speakerdeck.com/presentations/18bdd18477504d1580e2c2b802abfdbe/slide_1.jpg){kind=link}
![Thank You! Any Questions? Feel free to contact me: [email protected]](https://files.speakerdeck.com/presentations/18bdd18477504d1580e2c2b802abfdbe/slide_71.jpg){kind=link}