AR Rendering Engine, SDK, Content, and Future.

Yongseon Yim
LINE Plus Yuki Core Dev Senior Engineer
https://linedevday.linecorp.com/2020/ja/sessions/9322
https://linedevday.linecorp.com/2020/en/sessions/9322

Transcript

2. Agenda › AR Introduction › AR on LINE › AR Rendering Engine › Engine Renewal
3. Augmented Reality
4. Augmented Reality Overview
5. Augmented Reality Samples Game Video Call Selfie IUUQTXXXJOEFQFOEFOUDPVLFYUSBTJOEZCFTUHBEHFUTUFDI
   WJEFPHBNFTDPOTPMFTBVHNFOUFESFBMJUZHBNFTBSBOESPJEJPT
   BQQMFHPPHMFQMBZUFDIBIUNM
6. Trend
7. AR on LINE
8. Selfie Camera
9. Video Call AR Effects
10. LINE Avatar Let's make an avatar that resembles me
11. Start of Engine Dev.
12. AR Rendering Stack -*/& AR Rendering GRAPHICS STACK RENDERER CAMERA STACK RESOURCE MANAGEMENT CAMERA MANAGEMENT CONTENT MANAGEMENT SENSOR MANAGEMENT
13. YUKI YUKI -*/& AR Rendering YUKI is our AR Rendering Engine.
14. Requirements Light weight High Performance Cross Platform
15. YUKI Engine/SDK Start of YUKI Engine/SDK Dev. 2016.12 :6,*
    &OHJOF
    $ 0QFO(-&4 "OESPJE
    ,PUMJO J04
    4XJGU .BD04
    0CKFDUJWF$ 8JOEPXT
    $ Applied to LINE LIVE 2017.05 Applied to LINE Video Call ~2018.03 Applied to LINE Family Apps 2018.12
16. LINE LIVE First release Start of YUKI Engine/SDK Dev. 2016.12 Applied to LINE LIVE 2017.05 Applied to LINE Video Call ~2018.03 Applied to LINE Family Apps 2018.12 YUKI FOR LINE LIVE AR RENDERING BROADCASTER
17. LINE Video Call And, FacePlay was activated. Start of YUKI Engine/SDK Dev. 2016.12 Applied to LINE LIVE 2017.05 Applied to LINE Video Call ~2018.03 Applied to LINE Family Apps 2018.12
18. LINE Family Apps Start of YUKI Engine/SDK Dev. 2016.12 Applied to LINE LIVE 2017.05 Applied to LINE Video Call ~2018.03 Applied to LINE Family Apps 2018.12 Toward a more portable engine.
19. Faced Problems We faced several problems… Start of YUKI Engine/SDK Dev. 2016.12 Applied to LINE LIVE 2017.05 Applied to LINE Video Call ~2018.03 Applied to LINE Family Apps 2018.12 2019.01
20. What Problems Next gen. graphics API - Apple deprecates OpenGL and recommends using Metal. - Vulkan/DirectX12 needs to be used to support next-generation graphics. - The existing design had no consideration for using multiple graphics APIs. Performance Issue - High GPU utilizations Engine Size Issue - The size reached 7MB More powerful contents - We wanted to create content with more advanced technologies.
21. Renewal
22. Layout -*/& AR Rendering YUKI GRAPHICS STACK CAMERA STACK
23. ELSA -*/& AR Rendering YUKI ELSA CAMERA STACK
24. ELSA Goals Multiple Graphics API (Metal, Vulkan, OpenGL, DirectX) Cross Platform (iOS, Android, Mac, Windows) Light weight (Lower than 3MB) High Performance Better Code Quality Produce powerful contents by combining new technologies
25. How to Achieve Our Goals Graphics Abstraction Layer (GAL) › It’s part of the design and allows to handle multiple graphics APIs using one interface. › Simply put, it's possible to create interfaces through multiple APIs through a concise abstraction.
26. How to Achieve Our Goals C++ › One of the most portable languages. › A good language to choose from in terms of performance. › Productivity is also improved in C++17 and above.
27. How to Achieve Our Goals SPIR-V › Mediation role for common shader. VKSL (VULKAN GLSL) SPIR-V GLSL COMPILER HLSL COMPILER GLSL ESSL HLSL MSL COMPILER MSL OPENGL OPENGLES DIRECTX METAL VULKAN
28. Frame Graph GRAPH VERTEX &OUJUZ GRAPH VERTEX GRAPH VERTEX GRAPH VERTEX GRAPH VERTEX &OUJUZ &OUJUZ GPU Frame
29. RENDER RENDER RENDER POSITION VELOCITY SPRITE HEALTH MANA How to Achieve Our Goals Frame Graph › Hybrid Entity-Component-System(ECS) › Improved performance by increasing cache hit rate › Improve performance by minimizing state change 6OJRVF
    &OUJUZ
    *% &OUJUZ POSITION VELOCITY SPRITE HEALTH MANA 6OJRVF
    &OUJUZ
    *% &OUJUZ POSITION VELOCITY SPRITE HEALTH SKILL 6OJRVF
    &OUJUZ
    *% &OUJUZ RENDER RENDER
30. How to Achieve Our Goals Visualized Tool
31. So, we made it!
32. GPU Frame Time ms 0 5 10 15 20 25 30 Existing ELSA -PXFS
    JT
    CFUUFS
33. Light weight MB 0 1 2 3 4 5 6 7 Existing ELSA -PXFS
    JT
    CFUUFS
34. Now and Next Plan AI Filter Open source AR Rendering Engine YUKI ELSA (GRAPHICS) CAMERA STACK
35. Summary › However, there are many difficulties in doing AR rendering directly. › YUKI / ELSA : Engine already built, Powerful performance, Small size. › The need for AR content is growing.
36. Thank you