Please Draw Me A Sheep... in AR!

Augmented Reality (AR) is booming on mobile with ARCore / ARKit and new tools are emerging for building awesome experiences.

Like The Little Prince you will learn the foundations of ARCore and Sceneform for actually drawing a sheep in AR using basic forms and tools, and its rendering pipeline.

Through this talk, you will have an overview of what these libraries can offer from dealing with simple 3D shapes to advanced 3D assets or bringing Android UI components in AR. In the end, we will go further and see how to bring some AR fun with newly introduced features.

Transcript

1. Julien Salvi Place your screenshot here Please draw me a sheep … in ar!
2. Android addict since Froyo PAUG, Punk & IPA! You can find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder
3. × Augmented Reality on mobile × First steps with ARCore & Sceneform × Always more with ARCore!
4. 1. Augmented reality on android Real Reality with more fun!
5. “The AR market grew an active install base that approached 900 million”* *According to Digi-Capital
6. Building Ar APPS FOR... Place your screenshot here Housing, real estate Gaming and leisure ⚗ Science, education, industry A new way to enjoy sport Enhance your daily life! FUN!
7. Place your screenshot here Augmented reality on Android
8. × ARCore started in 2017, Sceneform in 2018 × Lots of feature: Motion tracking, environmental understanding, and light estimation × Bring Android UI to AR with Sceneform × 250+ Android devices supported Arcore & sceneform Build awesome AR experiences
9. 2. First steps with Arcore and sceneform Sheep it!
10. × Learn the basics of ARCore & Sceneform × Display basic shape and 3D assets × Use Android UI components to interact with your assets × Bring your assets to life with some animations First steps with Arcore and sceneform Let’s draw a sheep in AR!
11. First steps with Arcore and sceneform
12. × MinSDK 24 if ARCore is required, 14 otherwise × OpenGL 3.0.0 is required × “Google Play Services for AR” must be installed. Optional if your app contains non AR features × Java 8 is required when using Sceneform First steps with Arcore and sceneform Some requirements before starting
13. Android Manifest <uses-permission android:name="android.permission.CAMERA" /> <uses-feature android:glEsVersion="0x00030000" android:required="true" /> <uses-feature android:name="android.hardware.camera.ar" android:required="true" /> <application> <meta-data android:name="com.google.ar.core" android:value="required" /> </application>
14. Add ARCore & Sceneform dependencies to your project repositories { google() jcenter() } dependencies { implementation 'com.google.ar:core:1.20.0' implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1' } ARCore and Sceneform ⚠ Sceneform 1.16.0 is not available as a maven dependency
15. × Sceneform is now open-source! × SFA/SFB assets and plugin are now deprecated and can’t be used × Use glTF assets instead × Be careful when going for 1.16.0 ⚠sceneform 1.16.0 Break changes ahead!
16. Sceneform First steps into the AR world!
17. Add ArFragment to your layout <androidx.constraintlayout.widget.ConstraintLayout android:layout_width="match_parent" android:layout_height="match_parent"> <fragment android:id="@+id/sheepBoxFragment" android:name="com.google.ar.sceneform.ux.ArFragment" android:layout_width="0dp" android:layout_height="0dp" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent"/> </androidx.constraintlayout.widget.ConstraintLayout>
18. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
19. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
20. × Always think you are in a 3D world × Use Vector3 for position, translation, scale × Use Quaternion for rotations × The coordinate system is right-handed like OpenGL conventions Sceneform: Sheep 101 Into the 3D world y x z
21. × Load color or image texture asynchronously × Support for transparent colors and textures × Use MaterialFactory to attach the generated material to your shape Sceneform: Sheep 101 Loading Texture for your 3D shapes
22. × 3 basic shapes with ShapeFactory: Cube, Sphere and Cylinder. × Every shape accepts a Material × You must provide a center, a radius or a size for the shape you are building Sceneform: Sheep 101 Basic shapes with Sceneform
23. Create an opaque colored cube private fun makeColoredSheepBox() { MaterialFactory.makeOpaqueWithColor(this, Color(getColor(R.color.red))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
24. Create a transparent cube private fun makeTransparentColoredSheepBox() { MaterialFactory.makeTransparentWithColor(this, Color(getColor(R.color.colorAccent50))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
25. Create a textured cube private fun makeTextureSheepBox() { Texture.builder().setSource(BitmapFactory.decodeResource( resources, R.drawable.texture_cardboard)) .build() .thenAccept { texture -> MaterialFactory.makeOpaqueWithTexture(this, texture) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } } }
26. Now add the shape to the scene arFragment?.setOnTapArPlaneListener { hitResult, _, _ -> if (sheepBoxRenderable == null) { return@setOnTapArPlaneListener } // Create the Anchor: describes a fixed locations and orientation in the real world val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) // Create a TransformableNode and add it to the anchor. TransformableNode(arFragment!!.transformationSystem).apply { setParent(anchorNode) renderable = sheepBoxRenderable select() } }
27. tada! You just draw 3 sheeps... in boxes!
28. Sceneform Bring the fun to life!
29. × Let’s build a more realistic sheep with basic shapes, color and a wool texture × Learn how to compose Nodes or display Android UI components in AR × Add animations to bring the sheep to life Sceneform: Sheep 102 Let’s go deeper with Sceneform
30. × A Node represents a transformation within the scene graph’s hierarchy × Nodes have 1 parent and 0 to n child nodes and contain 1 renderable × To build a more realistic sheep we have to compose with several nodes Anchor Sceneform: Sheep 102 Node all the things! Sheep Node Sheep Node Sheep Node Sheep Node
31. Sheep scene graph hierarchy y x y z Body Part Sheep Node Head Left ear Right ear Main Body Body Part Body Part Body Part Leg Leg Leg Leg
32. Let’s now build our sheep // Create the Anchor. val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) sheepNode = SheepNode(this).apply { setParent(anchorNode) } val legPositions = arrayOf( Vector3(-0.1f, 0f, 0.1f), Vector3(0.1f, 0f, 0.1f), Vector3(-0.1f, 0f, -0.1f), Vector3(0.1f, 0f, -0.1f) ) val legRenderables = arrayOf( sheepRightBackLeg, sheepRightFrontLeg, sheepLeftBackLeg, sheepLeftFrontLeg ) legPositions.forEachIndexed { index, legPosition -> Node().apply { renderable = legRenderables[index] localPosition = legPosition setParent(sheepNode) } } Here is a code sample to add the 4 legs of our sheep
33. tada! We have now an awesome sheep in ar!
34. You can even take selfies with him!
35. But we can do more!
36. × With Sceneform you can use Android UI components in AR × Build layouts like classic Android application (XML layouts) × All view interactions are working as usual (OnClick listener for example) Sceneform: Sheep 102 Let’s use Android UI components in AR
37. Easily render Android UI sheepCard = Node() sheepCard!!.setParent(this) sheepCard!!.isEnabled = false sheepCard!!.localPosition = Vector3(0f, 0.5f, 0f) sheepCard!!.localRotation = Quaternion.axisAngle(Vector3(0f, 1f, 0f), 90f) sheepCard!!.localScale = Vector3(0.5f, 0.5f, 0.5f) ViewRenderable.builder() .setView(context, R.layout.layout_sheep_card) .build() .thenAccept { renderable -> sheepCard!!.renderable = renderable bindViews(renderable.view) animator = createAnimator(localPosition, Vector3(0f, 0f, 0.5f)) }
38. And here is an ANdroid layout in AR!
39. × For animating your 3D assets, play with position, scale and rotation × Use ObjectAnimator for animating Nodes × Compose animations thanks to AnimationSet × Let’s create a headbanging animation! Sceneform: Sheep 102 I like to move it! Move it!
40. Let’s compose our animation private fun headRotationAnimator(headRotation: Quaternion) = ObjectAnimator().apply { setObjectValues(headRotation) setPropertyName("localRotation") setEvaluator(QuaternionEvaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE } private fun headTranslationAnimator(startPosition: Vector3, endPosition: Vector3) = ObjectAnimator().apply { setObjectValues(startPosition, endPosition) setPropertyName("localPosition") setEvaluator(Vector3Evaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE }
41. Let’s compose our animation private fun headbangAnimation(headRotation: Quaternion) = AnimatorSet().apply { val rotation = Quaternion.multiply( headRotation, Quaternion.axisAngle(Vector3(0f, 0.5f, 0f), 45f)) val startPosition = sheepHeadNode!!.localPosition val endPosition = startPosition + Vector3(0f, -0.05f, 0f) playTogether( headRotationAnimator(rotation), headTranslationAnimator(startPosition, endPosition) ) }
42. DEMO
43. Sceneform Let’s play with awesome 3D assets
44. × When you want to use complex 3D objects go for glTF format × Before Sceneform 1.16.0, you could use OBJ or FBX formats which was converted into *.sfa or *.sfb formats × Unfortunately the Android Studio plugin is broken since 3.6.0 Sceneform: Sheep 201 Let’s play with more complex 3D assets
45. Place your screenshot here Where to find nice 3D assets?
48. Load your 3D assets in you scene // For Sceneform 1.16.0, load your glTF asset remotely or locally ModelRenderable.builder() .setSource(this, Uri.parse("https://my.api.com/sheep1.gltf")) .setIsFilamentGltf(true) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } }
49. × ObjectAnimator can still be used to animate your asset × Before Sceneform 1.16.0, FBX format was the only supported format for animations × From Sceneform 1.16.0, glTF loading and animations are available thanks to Filament library Sceneform: Sheep 201 Animations with 3D assets
50. Playing 3D asset animations // With Sceneform 1.15.0 val danceData = sheepRenderable.getAnimationData("sheep_move") val numAnimations = sheepRenderable.getAnimationDataCount() val moveData = sheepRenderable.getAnimationData(0) ModelAnimator(moveData, sheepRenderable).start() // For Sceneform 1.16.0, get the animation from the glTF asset thanks to Filament val filamentAsset = model.getRenderableInstance().getFilamentAsset(); if (filamentAsset.getAnimator().getAnimationCount() > 0) { animator = AnimationInstance(filamentAsset.getAnimator(), 0, System.nanoTime()) } animator.animator.applyAnimation(animator.index, time % animator.duration) animator.animator.updateBoneMatrices();
51. CONgrats! You are now ready to build awesome AR apps!
52. 3. Always more with ARcore Unlimited poweeeeer!
53. Always more with arcore Augmented images Augmented faces Cloud anchors ⚓ And more!
54. × Bring images to life × Use ARCore directly without Sceneform × ARCore can store up to 1,000 images references per image database × Use the arcoreimg command line tool to build your image reference database Augmented images Bring images to a new level!
55. × Time to build your own filter! × Use face recognition to transform your face into a 3D mesh × Place your augmented face assets thanks to identified regions (NOSE_TIP, LEFT_FOREHEAD…) Augmented faces Selfie time!
56. ☁ × Create collaborative AR experiences × Anchor data is stored on the cloud to be shared with multiple users × You can use Firebase for sharing Cloud Anchor or use your own solution × Permanent cloud anchor! Cloud anchors AR together! ⚓
57. × Accurate mapping of your environment × Map the depth of your scene so your 3D assets might be hidden behind a box × Controlling object occlusion Depth API Mapping the world
58. × Provides details info about the lighting in your scene × Enable immersive and realistic experiences × Two lighting mode available: ENVIRONMENTAL_HDR and AMBIENT_INTENSITY Lighting estimation Shine like a diamond!
59. Want to know more? https://developers.google.com/ar/develop ARCore documentation & CodeLabs https://codelabs.developers.google.com Design guidelines for AR https://github.com/Oleur/SheepAR SheepAR source code https://designguidelines.withgoogle.com/ar-design
60. Thanks ! Have fun with ARCore on Android! Any questions? @JulienSalvi