Please Draw Me A Sheep... in AR!

Augmented Reality (AR) is booming on mobile with ARCore / ARKit and new tools are emerging for building awesome experiences.

Like The Little Prince you will learn the foundations of ARCore and Sceneform for actually drawing a sheep in AR using basic forms and tools, and its rendering pipeline.

Through this talk, you will have an overview of what these libraries can offer from dealing with simple 3D shapes to advanced 3D assets or bringing Android UI components in AR. In the end, we will go further and see how to bring some AR fun with newly introduced features and Unity.

Transcript

1. Julien Salvi Place your screenshot here Please draw me a sheep … in ar!
2. Android addict since Froyo PAUG, Punk & IPA! You can find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder
3. × Augmented Reality on mobile × First steps with ARCore & Sceneform × Always more with ARCore! × AR with Unity on Android
4. 1. Augmented reality on android Real Reality with more fun!
5. “The AR market grew an active install base that approached 900 million”* *According to Digi-Capital
6. Building Ar APPS FOR... Place your screenshot here Housing, real estate Gaming and leisure ⚗ Science, education, industry A new way to enjoy sport Enhance your daily life! FUN!
7. Place your screenshot here Augmented reality on Android
8. × ARCore started in 2017, Sceneform in 2018 × Lots of feature: Motion tracking, environmental understanding, and light estimation × Bring Android UI to AR with Sceneform × 200+ Android devices supported Arcore & sceneform Build awesome AR experiences
9. 2. First steps with Arcore and sceneform Sheep it!
10. × Learn the basics of ARCore & Sceneform × Display basic shape and 3D assets × Use Android UI components to interact with your assets × Bring your assets to life with some animations First steps with Arcore and sceneform Let’s draw a sheep in AR!
11. First steps with Arcore and sceneform
12. × MinSDK 24 if ARCore is required, 14 otherwise × OpenGL 3.0.0 is required × “Google Play Services for AR” must be installed. Optional if your app contains non AR features × Java 8 is required when using Sceneform First steps with Arcore and sceneform Some requirements before starting
13. Android Manifest <uses-permission android:name="android.permission.CAMERA" /> <uses-feature android:glEsVersion="0x00030000" android:required="true" /> <uses-feature android:name="android.hardware.camera.ar" android:required="true" /> <application> <meta-data android:name="com.google.ar.core" android:value="required" /> </application>
14. Add ARCore & Sceneform dependencies to your project repositories { google() jcenter() } dependencies { implementation 'com.google.ar:core:1.15.0' implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.15.0' } ARCore and Sceneform ⚠ The latest version 1.16.0 is no more available as a maven dependency
15. × Sceneform is now open-source! × SFB assets and plugin are now deprecated × Use glTF assets instead × Be careful when updating to 1.16.0 ⚠sceneform 1.16.0 Break changes ahead!
16. Sceneform First steps into the AR world!
17. Add ArFragment in your layout <androidx.constraintlayout.widget.ConstraintLayout android:layout_width="match_parent" android:layout_height="match_parent"> <fragment android:id="@+id/sheepBoxFragment" android:name="com.google.ar.sceneform.ux.ArFragment" android:layout_width="0dp" android:layout_height="0dp" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent"/> </androidx.constraintlayout.widget.ConstraintLayout>
18. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
19. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
20. × Always think you are in a 3D world × Use Vector3 for position, translation, scale × Use Quaternion for rotations × The coordinate system is right-handed like OpenGL conventions Sceneform: Sheep 101 Into the 3D world y x z
21. × Load color or image texture asynchronously × Support for transparent colors and textures × Use MaterialFactory to attach the generated material to your shape Sceneform: Sheep 101 Loading Texture for your 3D shapes
22. × 3 basic shapes with ShapeFactory: Cube, Sphere and Cylinder. × Every shape accepts a Material × You must provide a center, a radius or a size for the shape you are building Sceneform: Sheep 101 Basic shapes with Sceneform
23. Create an opaque colored cube private fun makeColoredSheepBox() { MaterialFactory.makeOpaqueWithColor(this, Color(getColor(R.color.red))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
24. Create a transparent cube private fun makeTransparentColoredSheepBox() { MaterialFactory.makeTransparentWithColor(this, Color(getColor(R.color.colorAccent50))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
25. Create a textured cube private fun makeTextureSheepBox() { Texture.builder().setSource(BitmapFactory.decodeResource( resources, R.drawable.texture_cardboard)) .build() .thenAccept { texture -> MaterialFactory.makeOpaqueWithTexture(this, texture) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } } }
26. Now add the shape to the scene arFragment?.setOnTapArPlaneListener { hitResult, _, _ -> if (sheepBoxRenderable == null) { return@setOnTapArPlaneListener } // Create the Anchor: describes a fixed locations and orientation in the real world val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) // Create a TransformableNode and add it to the anchor. TransformableNode(arFragment!!.transformationSystem).apply { setParent(anchorNode) renderable = sheepBoxRenderable select() } }
27. tada! You just draw 3 sheeps... in boxes!
28. Sceneform Bring the fun to life!
29. × Let’s build a more realistic sheep with basic shapes, color and a wool texture × Learn how to compose Nodes or display Android UI components in AR × Add animations to bring the sheep to life Sceneform: Sheep 102 Let’s go deeper with Sceneform
30. × A Node represents a transformation within the scene graph’s hierarchy × Nodes have 1 parent and 0 to n child nodes and contain 1 renderable × To build a more realistic sheep we have to compose with several nodes Anchor Sceneform: Sheep 102 Node all the things! Sheep Node Sheep Node Sheep Node Sheep Node
31. Sheep scene graph’s hierarchy y x y z Body Part Sheep Node Head Left ear Right ear Main Body Body Part Body Part Body Part Leg Leg Leg Leg
32. Let’s now build our sheep // Create the Anchor. val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) sheepNode = SheepNode(this).apply { setParent(anchorNode) } val legPositions = arrayOf( Vector3(-0.1f, 0f, 0.1f), Vector3(0.1f, 0f, 0.1f), Vector3(-0.1f, 0f, -0.1f), Vector3(0.1f, 0f, -0.1f) ) val legRenderables = arrayOf( sheepRightBackLeg, sheepRightFrontLeg, sheepLeftBackLeg, sheepLeftFrontLeg ) legPositions.forEachIndexed { index, legPosition -> Node().apply { renderable = legRenderables[index] localPosition = legPosition setParent(sheepNode) } } Here is a code sample to add the 4 legs of our sheep
33. tada! We have now an awesome sheep in ar!
34. You can even take selfies with him!
35. But we can do more!
36. × With Sceneform you can use Android UI components in AR × Build layouts like classic Android application (XML layouts) × All view interactions are working as usual (OnClick listener for example) Sceneform: Sheep 102 Let’s use Android UI components in AR
37. Easily render Android UI sheepCard = Node() sheepCard!!.setParent(this) sheepCard!!.isEnabled = false sheepCard!!.localPosition = Vector3(0f, 0.5f, 0f) sheepCard!!.localRotation = Quaternion.axisAngle(Vector3(0f, 1f, 0f), 90f) sheepCard!!.localScale = Vector3(0.5f, 0.5f, 0.5f) ViewRenderable.builder() .setView(context, R.layout.layout_sheep_card) .build() .thenAccept { renderable -> sheepCard!!.renderable = renderable bindViews(renderable.view) animator = createAnimator(localPosition, Vector3(0f, 0f, 0.5f)) }
38. And here is an ANdroid layout in AR!
39. × For animating your 3D assets, play with position, scale and rotation × Use ObjectAnimator for animating Nodes × Compose animations thanks to AnimationSet × Let’s create a headbanging animation! Sceneform: Sheep 102 I like to move it! Move it!
40. Let’s compose our animation private fun headRotationAnimator(headRotation: Quaternion) = ObjectAnimator().apply { setObjectValues(headRotation) setPropertyName("localRotation") setEvaluator(QuaternionEvaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE } private fun headTranslationAnimator(startPosition: Vector3, endPosition: Vector3) = ObjectAnimator().apply { setObjectValues(startPosition, endPosition) setPropertyName("localPosition") setEvaluator(Vector3Evaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE }
41. Let’s compose our animation private fun headbangAnimation(headRotation: Quaternion) = AnimatorSet().apply { val rotation = Quaternion.multiply( headRotation, Quaternion.axisAngle(Vector3(0f, 0.5f, 0f), 45f)) val startPosition = sheepHeadNode!!.localPosition val endPosition = startPosition + Vector3(0f, -0.05f, 0f) playTogether( headRotationAnimator(rotation), headTranslationAnimator(startPosition, endPosition) ) }
42. DEMO
43. Sceneform Let’s play with awesome 3D assets
44. × When you want to use complex 3D objects go for glTF format × Before Sceneform 1.16.0, you can use OBJ or FBX formats which will be converted into *.sfa or *.sfb formats × Unfortunately the Android Studio plugin is broken since 3.6.0 Sceneform: Sheep 201 Let’s play with more complex 3D assets
45. Place your screenshot here Where to find nice 3D assets?
47. dependencies { classpath 'com.google.ar.sceneform:plugin:1.15.0' } apply plugin: 'com.google.ar.sceneform.plugin' sceneform.asset('sampledata/models/sheep.obj', 'default', 'sampledata/models/sheep.sfa', 'src/main/res/raw/sheep') Add 3D assets with Android Studio
49. Load your 3D assets in you scene ModelRenderable.builder() .setSource(this, Uri.parse("sheep1.sfb")) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } } // From Sceneform 1.16.0, load your glTF asset remotely or locally ModelRenderable.builder() .setSource(this, Uri.parse("sheep1.gltf")) .setIsFilamentGltf(true) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } }
50. × ObjectAnimator can still be used to animate your asset × Before Sceneform 1.16.0, FBX format was the only supported format for animations × From Sceneform 1.16.0, glTF loading and animations are available thanks to Filament library Sceneform: Sheep 201 Animations with 3D assets
51. Playing 3D asset animations // With Sceneform 1.15.0 val danceData = sheepRenderable.getAnimationData("sheep_move") val numAnimations = sheepRenderable.getAnimationDataCount() val moveData = sheepRenderable.getAnimationData(0) ModelAnimator(moveData, sheepRenderable).start() // From Sceneform 1.16.0, get the animation from the glTF asset thanks to Filament val filamentAsset = model.getRenderableInstance().getFilamentAsset(); if (filamentAsset.getAnimator().getAnimationCount() > 0) { animator = AnimationInstance(filamentAsset.getAnimator(), 0, System.nanoTime()) } animator.animator.applyAnimation(animator.index, time % animator.duration) animator.animator.updateBoneMatrices();
52. CONgrats! You are now ready to build awesome AR apps!
53. 3. Always more with ARcore Unlimited poweeeeer!
54. Always more with arcore Augmented images Augmented faces Cloud anchors ⚓
55. × Bring images to life × Use ARCore directly without Sceneform × ARCore can store up to 1,000 images references per image database × Use the arcoreimg command line tool to build your image reference database Augmented images Bring images to a new level!
56. × Time to build your own filter! × Use face recognition to transform your face into a 3D mesh × Place your augmented face assets thanks to identified regions (NOSE_TIP, LEFT_FOREHEAD…) Augmented faces Selfie time!
57. × Create collaborative AR experiences × Anchor data is stored on the cloud to be shared with multiple users × You can use Firebase for sharing Cloud Anchor or use your own solution Cloud anchors AR together! ☁ ⚓
58. 4. AR with unity on android Amazing stuff ahead!
59. AR with unity on android Powerful game engine (AR/VR, 2D, 3D…) Cross-platform development Apps and modules for Android
60. Unity as a LiBrary!
61. AR with Unity on Android Unity as a Library × Available with Unity 2019.3 × Add Unity content (AR/VR, 2D/3D games) directly to your Android apps × Proper library integration
63. class MainUnityActivity : OverrideUnityActivity() { protected fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) goBackBtn.setOnClickListener { _ -> showHomeActivity() } sendMsgBtn.setOnClickListener { _ -> UnitySendMessage("AndroidBuddy", "ChangeColor", "Blue") } finishBtn.setOnClickListener { _ -> finish() } } } Override the OverrideUnityActivity
64. AR with Unity on Android Unity as a Library limitations × Fullscreen rendering only × Only one instance of the Unity runtime is supported × You may need to adapt your plugins to work properly
65. AR with Unity on Android Unity as a library 01 02 03 https://blogs.unity3d.com/2019/06/17/add-features-powered-by -unity-to-native-mobile-apps Unity as a Library https://forum.unity.com/threads/integration-unity-as-a-library- in-native-android-app.685240/ Integration Unity as a library in native Android app https://github.com/Unity-Technologies/uaal-example Unity as a library example
66. Want to know more? https://developers.google.com/ar/develop ARCore documentation & CodeLabs https://codelabs.developers.google.com Design guidelines for AR https://github.com/Oleur/SheepAR SheepAR source code https://designguidelines.withgoogle.com/ar-design
67. Merci ! Have fun with ARCore on Android! Any questions? @JulienSalvi