What's new in ARCore from IO19

Slide 1

Slide 1 text

What's new in ARCore From Google I/O 2019

Slide 2

Slide 2 text

2 2018.2- Mercari, Intern(Android Engineer) 2019.4- Mercari, New Graduate (Backend Engineer) sasami(@sasamihoo)

Slide 3

Slide 3 text

Sessions in I/O’19 ● What’s New in ARCore ● Designing AR Applications ● Increasing AR Realism with Lighting ● Developing the First AR Experience for Google Maps ● Augmenting Faces and Images ● AR as a Feature: How to Supercharge Products Using Augmented Reality

Slide 4

Slide 4 text

Outline for today's presentation ● What is ARCore? ● what's new in ARCore ● How can we develop using ARCore?

Slide 5

Slide 5 text

What is ARCore?

Slide 6

Slide 6 text

What is ARCore? ● Software development kit for building an AR application ○ Motion tracking ○ Environmental understanding ○ Light estimation ● Platform ○ Android Studio ← today’s talk ○ iOS(with ARKit) ○ Unity ○ Unreal

Slide 7

Slide 7 text

Sceneform ● render 3D realistic scenes without having to learn OpenGL ● develop in AndroidStudio ○ SDK ○ Plugins for AndroidStudio ● pre-requirement ○ Android Studio 3.1 or higher ○ Android SDK Platform version 7.0 (API level 24) or higher

Slide 8

Slide 8 text

what's new in ARCore?

Slide 9

Slide 9 text

What’s new in ARCore? ● HDR environmental API ● Updates in Augmented faces and images ● Model Viewer ● AR navigation in Google Map

Slide 10

Slide 10 text

HDR environmental API Increasing AR Realism with Lighting

Slide 11

Slide 11 text

HDR environmental API ● Increase AR Realism with Lightning (Google I/O’19) ● Shows AR object with accurate shadows, highlights and reﬂections ○ Estimate using machine learning with a single camera frame Before and after Environmental HDR is applied to the digital mannequin on the left, featuring 3D printed designs from Julia Koerner(https://developers.googleblog.com/2019/05/ARCore-IO19.html)

Slide 12

Slide 12 text

HDR environmental API ● Main Directional Light ● Ambient Spherical Harmonics ● HDR Cubemap

Slide 13

Slide 13 text

Main Directional Light ● detecting light direction ○ Show the reﬂection of light from the particular of point which is not straight down

Slide 14

Slide 14 text

Ambient Spherical Harmonics ● gives the object a sense of the overall intensity of light ○ makes add more shading of the object

Slide 15

Slide 15 text

HDR Cubemap ● high dynamically cubemap ○ full specular hightlights and reﬂection ※cubemap...a way of mapping texture of surrounding environment

Slide 16

Slide 16 text

How to develop? ● Using Sceneform ※This api has not published yet(late summer for this year)

Slide 17

Slide 17 text

Directional Light API Config config = session.getConfig(); Config.setLightEstimationMode[LightEstimationMode,ENVIRONMENTAL_HDR]; Session,configure(config); // Get light estimation from frame LightEstimate lightEstimate = frame.getLightEstimate(); If (lightEstimate.getState() != LightEstimate_state.VALID) { return; } // Get main light intensity & direction float[] direction = lightEstimate.getMainLightDirection(); float[] intensity = lightEstimate.getMainLightIntensity();

Slide 18

Slide 18 text

Ambient Spherical Harmonics API Config config = session.getConfig(); Config.setLightEstimationMode[LightEstimationMode,ENVIRONMENTAL_HDR]; Session,configure(config); // Get light estimation from frame LightEstimate lightEstimate = frame.getLightEstimate(); If (lightEstimate.getState() != LightEstimate_state.VALID) { return; } float[] harmonics = lightEstimate.getAmbientSphericalHarmonics();

Slide 19

Slide 19 text

HDR Cubemap Config config = session.getConfig(); Config.setLightEstimationMode[LightEstimationMode,ENVIRONMENTAL_HDR]; Session,configure(config); // Get light estimation from frame LightEstimate lightEstimate = frame.getLightEstimate(); If (lightEstimate.getState() != LightEstimate_state.VALID) { return; } Image[] lightmaps = lightEstimate.getHdrCubeMap(); for (int i = 0; i < lightmaps.length; ++1) { app.uploadToTexture(i, lightmaps[i]); }

Slide 20

Slide 20 text

Augmented Images Augmenting Faces and Images

Slide 21

Slide 21 text

Augment images ● Point 2d images and render 3D models according to the images ● Enable to track those images simultaneously ○ Multiple images can be tracked at the same time An example of how the Augmented Images API can be used with moving targets by JD.com(https://developers.googleblog.com/2019/05/ARCore-IO19.html)

Slide 22

Slide 22 text

Augmented images in Google I/O ● Demo at Sandbox ○ Shows instruction of coﬀee machine ● AR navigation(from Google I/O 2019 app) ○ Show 3D signs of names for each spot/building

Slide 23

Slide 23 text

How it works? ● Recognize/track images ● Provide 6DoF Pose with device camera ● Anchor content of image

Slide 24

Slide 24 text

Recognize/track images ● Enable register images as augmented images inside app ● 20 images simultaneously trackable ● Up to 1000 images can be stored in augmented db ● All tracking happen on the device

Slide 25

Slide 25 text

Provide 6DoF Pose with device camera ● six degree of freedom pose with respect to the image ○ x,y,z and orientation(pitch, yaw and roll) → this enables to show 3D augmented images with right position (https://ja.wikipedia.org/wiki/6DoF)

Slide 26

Slide 26 text

Anchor content of image ● Overlay images with the right position ex) signposts of Google I/O How 3D images can be shown in each position of landmark? landmark position wrt signpost + device position wrt signpost = overlay images with the right position

Slide 27

Slide 27 text

Steps to develop ● Create Augmented Images D/B ● Create & Conﬁgure ARCore session ● Query pose, Render content

Slide 28

Slide 28 text

Create Augmented Images D/B ● Augmented Images database ○ enable create a database of storing up to 1,000 images ○ oﬄine tool enable to use to generate this database ○ enable to execute it at runtime → Adding images to the database enables to use to trigger showing 3D images

Slide 29

Slide 29 text

Create & Conﬁgure ARCore session private void configureSession() { Config config = new Config(session); augImgDatabase = AugumentedImageDatabase.deserialize(session, getAssets().open("sample_database.imgdb")) config.setAugumentedImageDatabase(augImgDatabase); session.configure(config); }

Slide 30

Slide 30 text

Query pose, Render content public void onFrameUpdate() { Collection updateImages = frame.getUpdatedTrackables(AugmentedImage.class); for (AugmentedImage image : updatedImages) { if (image.getTrackingState() == TRACKING) { drawAugumentedImages(image.getIndex(), image.getCenterPose()) ...

Slide 31

Slide 31 text

Codelab ● ARCore Augmented Images

Slide 32

Slide 32 text

Augmented faces Augmenting Faces and Images

Slide 33

Slide 33 text

Augmented faces ● identify diﬀerent regions of detected faces ● overlay assets such as textures and models

Slide 34

Slide 34 text

Augmented faces ● Center Pose ● Region poses ● 468 points 3D face mesh

Slide 35

Slide 35 text

Center Pose ● a center pose for a face(x,y,z) which marks the middle of a head ○ it’s useful when you render the hat

Slide 36

Slide 36 text

Region poses ● poses of region(such as nodes, ears and around faces)

Slide 37

Slide 37 text

468 point 3D face mesh ● dense 3D mesh ○ useful to paint detailed textures that accurately follow a face ○ without specialize hardware(depth sensor) by using machine learning

Slide 38

Slide 38 text

App using Augmented Faces ● MakeupPlus ○ Face eﬀect based on ARCore https://play.google.com/store/apps/details?id=com.meitu.makeup&h l=en

Slide 39

Slide 39 text

Augmented Faces on iOS ● one asset runs across both iOS and Android ○ APIs for iOS will be available later this summer

Slide 40

Slide 40 text

How it works? ● machine learning models built on top of TensorFlow Lite platform ○ transfer training ■ way of training the dataset using neural network ■ predict both 3D vertices & 2D contours ● run on device in real time

Slide 41

Slide 41 text

How to develop? ● create virtual content ● conﬁgure ARCore ● Render virtual content

Slide 42

Slide 42 text

Create virtual content ● 3D: FBX template ﬁle is prepared in the SDK ○ developer-guides/creating-assets-for-augmented-faces ● 2D: template texture ﬁle is prepared as well ○ Flatter version of face mesh ○ `reference_face_texture.png`

Slide 43

Slide 43 text

Create virtual content ● Covert .fbx ﬁle into model using AndroidStudio plugin

Slide 44

Slide 44 text

Conﬁgure augmented face mode public class FaceArFragment extends ArFragment { @Override protected Set getSessionFeatures() { return EnumSet.of(Session.Feature.FRONT_CAMERA); } @Override protected Config getSessionConfiguration(Session session) { Config config = new Config(session); config.setAugumentedFaceMode(AugumentedFaceMode.MESH3D); return config } }

Slide 45

Slide 45 text

Conﬁgure augmented face mode(3D content) protected void onCreate(Bundle saveInstanceState) {} ... // Load the face regions renderable. // Load 3D Contain into scene form ModelRenderable.bundler() .setSource(this, R.raw.fox_faces).build().thenAccept( modelRenderable -> { faceRegionsRenderable = modelRenderable; ... });

Slide 46

Slide 46 text

Conﬁgure augumented face mode(2D) protected void onCreate(Bundle saveInstanceState) {} ... // Load the face mesh texture // 2D texture you designed Texture.builder() .setSource(this, R.drawable.flag_texture) .build() .thenAccept(texture -> faceMeshTexture = texture);

Slide 47

Slide 47 text

Import virtual content OnUpdatedListener onUpdateListener() { ... for (AugumentedFace face: faceList) { // Create a face node and add it to the scene. AugumentedFaceNode faceNode = new AugumentedFaceNode(face); faceNode.setParent(scene); } } // Overlay the 3D assets on the face. faceNode.setFaceRegionsRenderable(faceRegionsRenderable);

Slide 48

Slide 48 text

Other resources ● Codelab ○ Augmented Faces ● Sample resources(.fbx) ○ Poly

Slide 49

Slide 49 text

Model Viewer What’s New in ARCore

Slide 50

Slide 50 text

Model Viewer ● enable show VR Object directly from web

Slide 51

Slide 51 text

Codelab ● Load .gltf file with tag ○ .gltf is an asset format for WebGL, OpenGL ○ Enable to use it at runtime

Slide 52

Slide 52 text

Codelab ● Add 3D Models to The Web with ○ https://awake-pigeon.glitch.me/ ○

Slide 53

Slide 53 text

● You can try on Pixel phones! ○ Playground for camera

Slide 54

Slide 54 text

AR navigation in Google Map ● Developing the First AR Experience for Google Maps ○ How to get more accuracy to apply AR navigation into Google Map? ■ Speed of Light ○ You can try on Pixel smartphones!

Slide 55

Slide 55 text

Thank you!