Building VR Apps On Android

During this talk you will learn how to build VR applications for Android starting with a basic Open GL ES prototype and moving towards a more complex one using Unity.

Julien will start by explaining how to build simple prototypes with Java, OpenGL ES and the Google VR library. Then, you will learn about Unity, which will allow you to build complex VR applications. Julien will also explore how to build Android native plugins (Java, JNI) in order to explain you how Unity can interact with the Android components (we will take our player & WebView plugins as examples). You'll get some tips on the UI/UX best practices for your Cardboard or Daydream apps, and some hints on 3D optimization for mobile platforms (in order for you to always keep your app at 60fps) which will give your users a smoother experience.

All Julien's examples will be based on CINEVR, The VR application he's been working on at Cinemur.

Transcript

1. Julien Salvi Building VR Apps On Android
2. Julien Salvi Android addict since Froyo Always looking for the best IPA! Twitter: @JulienSalvi Creative Technologist & Android Engineer @ CINEMUR
3. • Virtual Reality on Android • First steps in VR development • Going further with Unity • Best practices for VR on Android
4. 1 Virtual Reality on Android A global overview
5. John Carmack It's a moral imperative that we must create this “ NOT SO BAD
6. Why going into VR?
7. Be immersive
8. Go beyond reality
9. Bring new app to a new level
10. Share your experience, be social
11. Cardboard with Google VR (1.170.0) ◦ API Level 19 or later ◦ Native development, OpenGL ES ◦ Viewers with one-button input or gaze-base UI ◦ Hard to build complex applications (UI components, object loaders…)
12. Daydream with Google VR (1.170.0) ◦ API Level 24 or later ◦ A limited number of devices ◦ Daydream controller that allows 3DoF ◦ High quality, low latency ◦ Hard to build complex applications (UI components, object loaders…)
13. Oculus VR SDK (1.18.0) ◦ Developing for Oculus Go and Gear VR ◦ API Level 21 or later ◦ Native libraries (C/C++) ◦ Hard to prototype without Unity or Unreal ◦ Avantage: low level optimizations (JNI)
14. 2 First steps in VR development GVR and OpenGL ES
15. A GVR overview
16. …… …… Gradle dependencies
17. Build Cardboard and Daydream applications Widgets for watching 360 videos or panoramic images (easy implementation) Spatial audio support All .aar ~11Mo GVR in few words
18. GVR Developer Tools
19. Use PerformanceLogger information to check the temperature, frame rate, head position... Enable the HUD to monitor your app in VR (Daydream only) Performance monitoring
20. A bit of code? Yes please!
21. Player based on ExoPlayer Google VR SDK OpenGL ES Prototyping a video player in VR
22. Init the GvrView in the GvrActivity
23. StereoRenderer in details
24. StereoRenderer in details
25. … StereoRenderer in details
26. StereoRenderer in details
27. All together!
28. Adding panoramic images to your app • Render mono or stereo panorama • Fullscreen or Cardboard mode • Bitmap loading only
29. Watching 360 videos • Render mono or stereo 360 video • Fullscreen or Cardboard mode • Local or remote content • DASH, HLS, MP4, Ogg, webm support • Basic player control
31. Unleash the power of the GVR https://codelabs.developers.google.com/?cat=Virtual+Reality Google VR Codelabs https://github.com/googlevr/gvr-android-sdk/tree/master/samples (VrViews, 360 player, VR player, ndk sample) Google VR samples
32. 3 Going further with Unity Sh*t got serious!
33. Cross-platform (Android, iOS, Windows and most of the VR headsets) Fast prototyping, fast scripting Supporting most of the 3D and 2D formats (compression included) Optimization tools: Occlusion culling, Lightmap baking…
34. Control on the rendering pipeline Cross-platform shader compilation Bridge to native platforms Android Studio project export
36. Building plugins for Unity
37. Build your plugins using Java or C/C++ Android plugins might be : .jar, .aar ou .so Build an Android plugin for Unity
38. The new VideoPlayer component compatible with Android but limited Link the Unity texture and the Android surface to render the video Android Player For Unity
39. Let’s eat some C# and C++ Unity and JNI are best friends!
40. J N I Java code C/C++ code JavaPlugin
41. Calling a JAVA object from Unity
42. C/C++ Plugin
43. Better performances when you call your Java plugin through the JNI Important: JVM loading Java and C/C++ Plugin
44. Java and C/C++ Plugin
45. Java and C/C++ Plugin
46. Java and C/C++ Plugin
47. 4 VR Best Practices Perf matter!
48. UI tips & tricks • Simplify UI layouts when possible, dynamic UI requires costly layout recalculations. • Beware of the overdraws! • No transparent UI. • Use simple geometry and baked texture
49. UI/UX tip: think about your user
50. GPU optimizations • Avoid shadows, they have an expensive cost. • Use forward rendering. Do not use deferred rendering on mobile hardware. • Use ETC2 or ASTC texture compression for Android • Use simple shader, move expensive fragment operations to the vertex shader.
51. 3D optimizations • Create LOD (level of detail) meshes with lower vertex counts. • Break up large models into parts and take advantage of the occlusion culling • Similar to the previous tip, disable static batching. • Reduce model detail in a modeling application.
52. Dynamic Lighting • Get the average color of a frame directly from the CPU. • Use last layers of mipmaps to get color thanks to a resampling of the video texture. • Extract the color from the last layer of mipmaps using the native plugin.
53. Screen Drifting ツ • Hardware defects (gyroscope, accelerometer…) • Software correction but complex
54. https://developers.google.com/vr https://medium.com/@cinemur/73ec7e83dd5c About Google VR ExoPlayer for building powerful VR players on Cardboard and GearVR https://docs.unity3d.com/Manual/PluginsForAndroid.html Building Unity plugins for Android Want to know more on VR? https://www.youtube.com/watch?v=ES9jArHRFHQ Designing Screen Interfaces for VR (Google I/O ‘17)
55. "VR can be tremendously fun and beautiful. It's been frustrating that more people haven't been able to enjoy it for so many years. I hope lots of people will soon find VR to be as fascinating as I have." Jaron Lanier - term of Virtual Reality Inventor Merci !