Slide 1

Slide 1 text

Julien Salvi Place your screenshot here Please draw me a sheep … in ar!

Slide 2

Slide 2 text

Android addict since Froyo PAUG, Punk & IPA! You can find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder

Slide 3

Slide 3 text

× Augmented Reality on mobile × First steps with ARCore & Sceneform × Always more with ARCore! × AR with Unity on Android

Slide 4

Slide 4 text

1. Augmented reality on android Real Reality with more fun!

Slide 5

Slide 5 text

“The AR market grew an active install base that approached 900 million”* *According to Digi-Capital

Slide 6

Slide 6 text

Building Ar APPS FOR... Place your screenshot here Housing, real estate Gaming and leisure ⚗ Science, education, industry A new way to enjoy sport Enhance your daily life! FUN!

Slide 7

Slide 7 text

Place your screenshot here Augmented reality on Android

Slide 8

Slide 8 text

× ARCore started in 2017, Sceneform in 2018 × Lots of feature: Motion tracking, environmental understanding, and light estimation × Bring Android UI to AR with Sceneform × 200+ Android devices supported Arcore & sceneform Build awesome AR experiences

Slide 9

Slide 9 text

2. First steps with Arcore and sceneform Sheep it!

Slide 10

Slide 10 text

× Learn the basics of ARCore & Sceneform × Display basic shape and 3D assets × Use Android UI components to interact with your assets × Bring your assets to life with some animations First steps with Arcore and sceneform Let’s draw a sheep in AR!

Slide 11

Slide 11 text

First steps with Arcore and sceneform

Slide 12

Slide 12 text

× MinSDK 24 if ARCore is required, 14 otherwise × OpenGL 3.0.0 is required × “Google Play Services for AR” must be installed. Optional if your app contains non AR features × Java 8 is required when using Sceneform First steps with Arcore and sceneform Some requirements before starting

Slide 13

Slide 13 text

Android Manifest

Slide 14

Slide 14 text

Add ARCore & Sceneform dependencies to your project repositories { google() jcenter() } dependencies { implementation 'com.google.ar:core:1.15.0' implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.15.0' } ARCore and Sceneform ⚠ The latest version 1.16.0 is no more available as a maven dependency

Slide 15

Slide 15 text

× Sceneform is now open-source! × SFB assets and plugin are now deprecated × Use glTF assets instead × Be careful when updating to 1.16.0 ⚠sceneform 1.16.0 Break changes ahead!

Slide 16

Slide 16 text

Sceneform First steps into the AR world!

Slide 17

Slide 17 text

Add ArFragment in your layout

Slide 18

Slide 18 text

ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }

Slide 19

Slide 19 text

ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }

Slide 20

Slide 20 text

× Always think you are in a 3D world × Use Vector3 for position, translation, scale × Use Quaternion for rotations × The coordinate system is right-handed like OpenGL conventions Sceneform: Sheep 101 Into the 3D world y x z

Slide 21

Slide 21 text

× Load color or image texture asynchronously × Support for transparent colors and textures × Use MaterialFactory to attach the generated material to your shape Sceneform: Sheep 101 Loading Texture for your 3D shapes

Slide 22

Slide 22 text

× 3 basic shapes with ShapeFactory: Cube, Sphere and Cylinder. × Every shape accepts a Material × You must provide a center, a radius or a size for the shape you are building Sceneform: Sheep 101 Basic shapes with Sceneform

Slide 23

Slide 23 text

Create an opaque colored cube private fun makeColoredSheepBox() { MaterialFactory.makeOpaqueWithColor(this, Color(getColor(R.color.red))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }

Slide 24

Slide 24 text

Create a transparent cube private fun makeTransparentColoredSheepBox() { MaterialFactory.makeTransparentWithColor(this, Color(getColor(R.color.colorAccent50))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }

Slide 25

Slide 25 text

Create a textured cube private fun makeTextureSheepBox() { Texture.builder().setSource(BitmapFactory.decodeResource( resources, R.drawable.texture_cardboard)) .build() .thenAccept { texture -> MaterialFactory.makeOpaqueWithTexture(this, texture) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } } }

Slide 26

Slide 26 text

Now add the shape to the scene arFragment?.setOnTapArPlaneListener { hitResult, _, _ -> if (sheepBoxRenderable == null) { return@setOnTapArPlaneListener } // Create the Anchor: describes a fixed locations and orientation in the real world val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) // Create a TransformableNode and add it to the anchor. TransformableNode(arFragment!!.transformationSystem).apply { setParent(anchorNode) renderable = sheepBoxRenderable select() } }

Slide 27

Slide 27 text

tada! You just draw 3 sheeps... in boxes!

Slide 28

Slide 28 text

Sceneform Bring the fun to life!

Slide 29

Slide 29 text

× Let’s build a more realistic sheep with basic shapes, color and a wool texture × Learn how to compose Nodes or display Android UI components in AR × Add animations to bring the sheep to life Sceneform: Sheep 102 Let’s go deeper with Sceneform

Slide 30

Slide 30 text

× A Node represents a transformation within the scene graph’s hierarchy × Nodes have 1 parent and 0 to n child nodes and contain 1 renderable × To build a more realistic sheep we have to compose with several nodes Anchor Sceneform: Sheep 102 Node all the things! Sheep Node Sheep Node Sheep Node Sheep Node

Slide 31

Slide 31 text

Sheep scene graph’s hierarchy y x y z Body Part Sheep Node Head Left ear Right ear Main Body Body Part Body Part Body Part Leg Leg Leg Leg

Slide 32

Slide 32 text

Let’s now build our sheep // Create the Anchor. val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) sheepNode = SheepNode(this).apply { setParent(anchorNode) } val legPositions = arrayOf( Vector3(-0.1f, 0f, 0.1f), Vector3(0.1f, 0f, 0.1f), Vector3(-0.1f, 0f, -0.1f), Vector3(0.1f, 0f, -0.1f) ) val legRenderables = arrayOf( sheepRightBackLeg, sheepRightFrontLeg, sheepLeftBackLeg, sheepLeftFrontLeg ) legPositions.forEachIndexed { index, legPosition -> Node().apply { renderable = legRenderables[index] localPosition = legPosition setParent(sheepNode) } } Here is a code sample to add the 4 legs of our sheep

Slide 33

Slide 33 text

tada! We have now an awesome sheep in ar!

Slide 34

Slide 34 text

You can even take selfies with him!

Slide 35

Slide 35 text

But we can do more!

Slide 36

Slide 36 text

× With Sceneform you can use Android UI components in AR × Build layouts like classic Android application (XML layouts) × All view interactions are working as usual (OnClick listener for example) Sceneform: Sheep 102 Let’s use Android UI components in AR

Slide 37

Slide 37 text

Easily render Android UI sheepCard = Node() sheepCard!!.setParent(this) sheepCard!!.isEnabled = false sheepCard!!.localPosition = Vector3(0f, 0.5f, 0f) sheepCard!!.localRotation = Quaternion.axisAngle(Vector3(0f, 1f, 0f), 90f) sheepCard!!.localScale = Vector3(0.5f, 0.5f, 0.5f) ViewRenderable.builder() .setView(context, R.layout.layout_sheep_card) .build() .thenAccept { renderable -> sheepCard!!.renderable = renderable bindViews(renderable.view) animator = createAnimator(localPosition, Vector3(0f, 0f, 0.5f)) }

Slide 38

Slide 38 text

And here is an ANdroid layout in AR!

Slide 39

Slide 39 text

× For animating your 3D assets, play with position, scale and rotation × Use ObjectAnimator for animating Nodes × Compose animations thanks to AnimationSet × Let’s create a headbanging animation! Sceneform: Sheep 102 I like to move it! Move it!

Slide 40

Slide 40 text

Let’s compose our animation private fun headRotationAnimator(headRotation: Quaternion) = ObjectAnimator().apply { setObjectValues(headRotation) setPropertyName("localRotation") setEvaluator(QuaternionEvaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE } private fun headTranslationAnimator(startPosition: Vector3, endPosition: Vector3) = ObjectAnimator().apply { setObjectValues(startPosition, endPosition) setPropertyName("localPosition") setEvaluator(Vector3Evaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE }

Slide 41

Slide 41 text

Let’s compose our animation private fun headbangAnimation(headRotation: Quaternion) = AnimatorSet().apply { val rotation = Quaternion.multiply( headRotation, Quaternion.axisAngle(Vector3(0f, 0.5f, 0f), 45f)) val startPosition = sheepHeadNode!!.localPosition val endPosition = startPosition + Vector3(0f, -0.05f, 0f) playTogether( headRotationAnimator(rotation), headTranslationAnimator(startPosition, endPosition) ) }

Slide 42

Slide 42 text

DEMO

Slide 43

Slide 43 text

Sceneform Let’s play with awesome 3D assets

Slide 44

Slide 44 text

× When you want to use complex 3D objects go for glTF format × Before Sceneform 1.16.0, you can use OBJ or FBX formats which will be converted into *.sfa or *.sfb formats × Unfortunately the Android Studio plugin is broken since 3.6.0 Sceneform: Sheep 201 Let’s play with more complex 3D assets

Slide 45

Slide 45 text

Place your screenshot here Where to find nice 3D assets?

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

dependencies { classpath 'com.google.ar.sceneform:plugin:1.15.0' } apply plugin: 'com.google.ar.sceneform.plugin' sceneform.asset('sampledata/models/sheep.obj', 'default', 'sampledata/models/sheep.sfa', 'src/main/res/raw/sheep') Add 3D assets with Android Studio

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Load your 3D assets in you scene ModelRenderable.builder() .setSource(this, Uri.parse("sheep1.sfb")) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } } // From Sceneform 1.16.0, load your glTF asset remotely or locally ModelRenderable.builder() .setSource(this, Uri.parse("sheep1.gltf")) .setIsFilamentGltf(true) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } }

Slide 50

Slide 50 text

× ObjectAnimator can still be used to animate your asset × Before Sceneform 1.16.0, FBX format was the only supported format for animations × From Sceneform 1.16.0, glTF loading and animations are available thanks to Filament library Sceneform: Sheep 201 Animations with 3D assets

Slide 51

Slide 51 text

Playing 3D asset animations // With Sceneform 1.15.0 val danceData = sheepRenderable.getAnimationData("sheep_move") val numAnimations = sheepRenderable.getAnimationDataCount() val moveData = sheepRenderable.getAnimationData(0) ModelAnimator(moveData, sheepRenderable).start() // From Sceneform 1.16.0, get the animation from the glTF asset thanks to Filament val filamentAsset = model.getRenderableInstance().getFilamentAsset(); if (filamentAsset.getAnimator().getAnimationCount() > 0) { animator = AnimationInstance(filamentAsset.getAnimator(), 0, System.nanoTime()) } animator.animator.applyAnimation(animator.index, time % animator.duration) animator.animator.updateBoneMatrices();

Slide 52

Slide 52 text

CONgrats! You are now ready to build awesome AR apps!

Slide 53

Slide 53 text

3. Always more with ARcore Unlimited poweeeeer!

Slide 54

Slide 54 text

Always more with arcore Augmented images Augmented faces Cloud anchors ⚓

Slide 55

Slide 55 text

× Bring images to life × Use ARCore directly without Sceneform × ARCore can store up to 1,000 images references per image database × Use the arcoreimg command line tool to build your image reference database Augmented images Bring images to a new level!

Slide 56

Slide 56 text

× Time to build your own filter! × Use face recognition to transform your face into a 3D mesh × Place your augmented face assets thanks to identified regions (NOSE_TIP, LEFT_FOREHEAD…) Augmented faces Selfie time!

Slide 57

Slide 57 text

× Create collaborative AR experiences × Anchor data is stored on the cloud to be shared with multiple users × You can use Firebase for sharing Cloud Anchor or use your own solution Cloud anchors AR together! ☁ ⚓

Slide 58

Slide 58 text

4. AR with unity on android Amazing stuff ahead!

Slide 59

Slide 59 text

AR with unity on android Powerful game engine (AR/VR, 2D, 3D…) Cross-platform development Apps and modules for Android

Slide 60

Slide 60 text

Unity as a LiBrary!

Slide 61

Slide 61 text

AR with Unity on Android Unity as a Library × Available with Unity 2019.3 × Add Unity content (AR/VR, 2D/3D games) directly to your Android apps × Proper library integration

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

class MainUnityActivity : OverrideUnityActivity() { protected fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) goBackBtn.setOnClickListener { _ -> showHomeActivity() } sendMsgBtn.setOnClickListener { _ -> UnitySendMessage("AndroidBuddy", "ChangeColor", "Blue") } finishBtn.setOnClickListener { _ -> finish() } } } Override the OverrideUnityActivity

Slide 64

Slide 64 text

AR with Unity on Android Unity as a Library limitations × Fullscreen rendering only × Only one instance of the Unity runtime is supported × You may need to adapt your plugins to work properly

Slide 65

Slide 65 text

AR with Unity on Android Unity as a library 01 02 03 https://blogs.unity3d.com/2019/06/17/add-features-powered-by -unity-to-native-mobile-apps Unity as a Library https://forum.unity.com/threads/integration-unity-as-a-library- in-native-android-app.685240/ Integration Unity as a library in native Android app https://github.com/Unity-Technologies/uaal-example Unity as a library example

Slide 66

Slide 66 text

Want to know more? https://developers.google.com/ar/develop ARCore documentation & CodeLabs https://codelabs.developers.google.com Design guidelines for AR https://github.com/Oleur/SheepAR SheepAR source code https://designguidelines.withgoogle.com/ar-design

Slide 67

Slide 67 text

Merci ! Have fun with ARCore on Android! Any questions? @JulienSalvi