Slide 1

Slide 1 text

July 26, 2024 CHA Jerome Safie Inc. Development Division 2 - Mobile Team Safie Viewer Android & Sceneview

Slide 2

Slide 2 text

© Safie Inc.| Self-Introduction - Jerome CHA, French - Joined Safie in November 2023 as Android Engineer - Previously worked at Kubell (ex Chatwork) and Enigmo (BUYMA) - Hobbies : Workout (ex-JBBF Bodybuilder) / Making my own robot - Twitter @yujiro45 2

Slide 3

Slide 3 text

© Safie Inc.| Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 3

Slide 4

Slide 4 text

© Safie Inc.| Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 4

Slide 5

Slide 5 text

© Safie Inc.| Sceneview Recap - https://sceneview.github.io/ - Library that uses Google’s ARCore and Filament to display 3D / AR contents - Included 2 views : Sceneview & ARSceneview - My previous presentation @ Mobile Dev Japan #1 https://speakerdeck.com/jeromecha/ar-with-sceneview-on-android 5 ARSceneview Sceneview

Slide 6

Slide 6 text

© Safie Inc.| Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 6

Slide 7

Slide 7 text

© Safie Inc.| Safie Viewer - Cloud service for security cameras - Clear images, easy to setup, easy to use - Videos are in HD 30fps 7

Slide 8

Slide 8 text

© Safie Inc.| Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 8

Slide 9

Slide 9 text

© Safie Inc.| Safie’s new camera - Safie GO 360 - Launched in February 1st 2024 - Compact and lightweight, easy to transport - Built-in LTE for cloud recording anytime, anywhere - Horizontal and vertical viewing angle of 182° 9 https://safie.co.jp/news/3033/

Slide 10

Slide 10 text

© Safie Inc.| Web Demo 10 speed x3

Slide 11

Slide 11 text

© Safie Inc.| 3D model - 2 installations 11 Wall Ceiling

Slide 12

Slide 12 text

© Safie Inc.| 3D model on Android 12 - 3D Model created via Blender - iOS : .dae format → convert to .scn format with Xcode - Android : .glb format

Slide 13

Slide 13 text

© Safie Inc.| 3D model on Android 13

Slide 14

Slide 14 text

© Safie Inc.| Texture Mapping 14 How can we map the camera’s stream on the texture of the 3D model

Slide 15

Slide 15 text

© Safie Inc.| Texture Mapping 15 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt

Slide 16

Slide 16 text

© Safie Inc.| Texture Mapping 16 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt It is a PlaneNode, so we cannot use it

Slide 17

Slide 17 text

© Safie Inc.| Texture Mapping 17 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt We need VideoMaterial & ExoPlayer

Slide 18

Slide 18 text

© Safie Inc.| Texture Mapping 18 We need to make our own VideoMaterial with ExoPlayer!

Slide 19

Slide 19 text

© Safie Inc.| Texture Mapping 19 VideoMaterial ExoPlayer

Slide 20

Slide 20 text

© Safie Inc.| Texture Mapping 20 ExoPlayerVideoMaterial!

Slide 21

Slide 21 text

© Safie Inc.| Texture Mapping 21 Engine : Google Filament’s engine Exoplayer : for the video’s stream materialLoader : for the texture’s creation

Slide 22

Slide 22 text

© Safie Inc.| Texture Mapping 22 Create a SurfaceTexture

Slide 23

Slide 23 text

© Safie Inc.| Texture Mapping 23 Create a Stream from Filament based on the surfaceTexture

Slide 24

Slide 24 text

© Safie Inc.| Texture Mapping 24 Create a Sceneview Texture for our modelNode based on Filament’s Stream

Slide 25

Slide 25 text

© Safie Inc.| Texture Mapping 25 Create a VideoInstance for our modelNode later

Slide 26

Slide 26 text

© Safie Inc.| Texture Mapping 26 Setup our ExoPlayer

Slide 27

Slide 27 text

© Safie Inc.| Texture Mapping 27 Use our own CustomVideoMaterial

Slide 28

Slide 28 text

© Safie Inc.| Texture Mapping 28

Slide 29

Slide 29 text

© Safie Inc.| User’s interaction 29 How can we limit the 3D model rotation?

Slide 30

Slide 30 text

© Safie Inc.| User’s interaction 30 To limit the 3D model rotation : - Remove the default’s user interaction - Develop your own behavior from scratch

Slide 31

Slide 31 text

© Safie Inc.| User’s interaction 31 To remove the default behavior : - Create a subclass of SceneView - Setup cameraManipulator to null

Slide 32

Slide 32 text

© Safie Inc.| User’s interaction - Drag (Wall position) 32

Slide 33

Slide 33 text

© Safie Inc.| User’s interaction - Drag (Ceiling position) 33

Slide 34

Slide 34 text

© Safie Inc.| User’s interaction - Zoom 34

Slide 35

Slide 35 text

© Safie Inc.| User’s interaction - Zoom 35 Zoom in a specific point on the 3D model - Pinch in/out : Take the middle point between the two fingers and zoom to that position - Double Tap : Double tap to zoom in/out - If tap outside the 3D model, then zoom in the middle

Slide 36

Slide 36 text

© Safie Inc.| User’s interaction - Zoom 36

Slide 37

Slide 37 text

© Safie Inc.| User’s interaction - Zoom 37 Screen

Slide 38

Slide 38 text

© Safie Inc.| User’s interaction - Zoom 38 Screen

Slide 39

Slide 39 text

© Safie Inc.| User’s interaction - Zoom 39 Screen

Slide 40

Slide 40 text

© Safie Inc.| User’s interaction - Zoom 40 Screen This is called a HitResult

Slide 41

Slide 41 text

© Safie Inc.| User’s interaction - Zoom 41

Slide 42

Slide 42 text

© Safie Inc.| Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 42

Slide 43

Slide 43 text

© Safie Inc.| Problems Problem 1 3D Model hit box 43

Slide 44

Slide 44 text

© Safie Inc.| Problem 1 : 3D Model hit box 44 Tapping outside is considered as the edge of the 3D model (but only once)

Slide 45

Slide 45 text

© Safie Inc.| Problem 1 : 3D Model hit box 45 Swiping outside also triggered the 3D Model to move

Slide 46

Slide 46 text

© Safie Inc.| Problem 1 : 3D Model hit box 46 https://github.com/SceneView/sceneview-android/issues/522

Slide 47

Slide 47 text

© Safie Inc.| Problem 1 : 3D Model hit box 47 Resolved by putting a PlaneNode on the top of the 3D model to intercept the hit : Diameter of hemisphere *assuming that the center of the hemisphere is at (0,0) *

Slide 48

Slide 48 text

© Safie Inc.| Problem 1 : 3D Model hit box 48

Slide 49

Slide 49 text

© Safie Inc.| Problems 2 Problem 2 ExoPlayer on multiple surfaces 49

Slide 50

Slide 50 text

© Safie Inc.| Problems 2 : ExoPlayer on multiple surfaces 50 Switching Fisheye ⟷ 360° view with a button

Slide 51

Slide 51 text

© Safie Inc.| Problems 2 : ExoPlayer on multiple surfaces 51 https://github.com/google/ExoPlayer/issues/6939

Slide 52

Slide 52 text

© Safie Inc.| Problems 2 : ExoPlayer on multiple surfaces 52 The only solution was to reinit ExoPlayer each time you press the button

Slide 53

Slide 53 text

© Safie Inc.| We are hiring! More info : https://open.talentio.com/r/1/c/safie/pages/84134 53 We are looking for Android / iOS engineers!