Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Please Draw Me A Sheep... in AR!

Please Draw Me A Sheep... in AR!

Augmented Reality (AR) is booming on mobile with ARCore / ARKit and new tools are emerging for building awesome experiences.

Like The Little Prince you will learn the foundations of ARCore and Sceneform for actually drawing a sheep in AR using basic forms and tools, and its rendering pipeline.

Through this talk, you will have an overview of what these libraries can offer from dealing with simple 3D shapes to advanced 3D assets or bringing Android UI components in AR. In the end, we will go further and see how to bring some AR fun with newly introduced features.

56047a7b11797f42c2e7030d771fe803?s=128

Julien Salvi

October 08, 2020
Tweet

Transcript

  1. Julien Salvi Place your screenshot here Please draw me a

    sheep … in ar!
  2. Android addict since Froyo PAUG, Punk & IPA! You can

    find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder
  3. × Augmented Reality on mobile × First steps with ARCore

    & Sceneform × Always more with ARCore!
  4. 1. Augmented reality on android Real Reality with more fun!

  5. “The AR market grew an active install base that approached

    900 million”* *According to Digi-Capital
  6. Building Ar APPS FOR... Place your screenshot here Housing, real

    estate Gaming and leisure ⚗ Science, education, industry A new way to enjoy sport Enhance your daily life! FUN!
  7. Place your screenshot here Augmented reality on Android

  8. × ARCore started in 2017, Sceneform in 2018 × Lots

    of feature: Motion tracking, environmental understanding, and light estimation × Bring Android UI to AR with Sceneform × 250+ Android devices supported Arcore & sceneform Build awesome AR experiences
  9. 2. First steps with Arcore and sceneform Sheep it!

  10. × Learn the basics of ARCore & Sceneform × Display

    basic shape and 3D assets × Use Android UI components to interact with your assets × Bring your assets to life with some animations First steps with Arcore and sceneform Let’s draw a sheep in AR!
  11. First steps with Arcore and sceneform

  12. × MinSDK 24 if ARCore is required, 14 otherwise ×

    OpenGL 3.0.0 is required × “Google Play Services for AR” must be installed. Optional if your app contains non AR features × Java 8 is required when using Sceneform First steps with Arcore and sceneform Some requirements before starting
  13. Android Manifest <uses-permission android:name="android.permission.CAMERA" /> <uses-feature android:glEsVersion="0x00030000" android:required="true" /> <uses-feature

    android:name="android.hardware.camera.ar" android:required="true" /> <application> <meta-data android:name="com.google.ar.core" android:value="required" /> </application>
  14. Add ARCore & Sceneform dependencies to your project repositories {

    google() jcenter() } dependencies { implementation 'com.google.ar:core:1.20.0' implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1' } ARCore and Sceneform ⚠ Sceneform 1.16.0 is not available as a maven dependency
  15. × Sceneform is now open-source! × SFA/SFB assets and plugin

    are now deprecated and can’t be used × Use glTF assets instead × Be careful when going for 1.16.0 ⚠sceneform 1.16.0 Break changes ahead!
  16. Sceneform First steps into the AR world!

  17. Add ArFragment to your layout <androidx.constraintlayout.widget.ConstraintLayout android:layout_width="match_parent" android:layout_height="match_parent"> <fragment android:id="@+id/sheepBoxFragment"

    android:name="com.google.ar.sceneform.ux.ArFragment" android:layout_width="0dp" android:layout_height="0dp" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent"/> </androidx.constraintlayout.widget.ConstraintLayout>
  18. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override

    fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
  19. ArFragment in your Activity class SheepBoxActivity : AppCompatActivity() { override

    fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (!checkIsSupportedDeviceOrFinish()) { return } binding = ActivitySheepBoxBinding.inflate(layoutInflater) setContentView(binding.root) arFragment = supportFragmentManager.findFragmentById(R.id.sheepBoxFragment) as ArFragment } }
  20. × Always think you are in a 3D world ×

    Use Vector3 for position, translation, scale × Use Quaternion for rotations × The coordinate system is right-handed like OpenGL conventions Sceneform: Sheep 101 Into the 3D world y x z
  21. × Load color or image texture asynchronously × Support for

    transparent colors and textures × Use MaterialFactory to attach the generated material to your shape Sceneform: Sheep 101 Loading Texture for your 3D shapes
  22. × 3 basic shapes with ShapeFactory: Cube, Sphere and Cylinder.

    × Every shape accepts a Material × You must provide a center, a radius or a size for the shape you are building Sceneform: Sheep 101 Basic shapes with Sceneform
  23. Create an opaque colored cube private fun makeColoredSheepBox() { MaterialFactory.makeOpaqueWithColor(this,

    Color(getColor(R.color.red))) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
  24. Create a transparent cube private fun makeTransparentColoredSheepBox() { MaterialFactory.makeTransparentWithColor(this, Color(getColor(R.color.colorAccent50)))

    .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } }
  25. Create a textured cube private fun makeTextureSheepBox() { Texture.builder().setSource(BitmapFactory.decodeResource( resources,

    R.drawable.texture_cardboard)) .build() .thenAccept { texture -> MaterialFactory.makeOpaqueWithTexture(this, texture) .thenAccept { material -> sheepBoxRenderable = ShapeFactory.makeCube( Vector3(0.2f, 0.2f, 0.2f), Vector3(0.0f, 0.15f, 0.0f), material) } } }
  26. Now add the shape to the scene arFragment?.setOnTapArPlaneListener { hitResult,

    _, _ -> if (sheepBoxRenderable == null) { return@setOnTapArPlaneListener } // Create the Anchor: describes a fixed locations and orientation in the real world val anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) // Create a TransformableNode and add it to the anchor. TransformableNode(arFragment!!.transformationSystem).apply { setParent(anchorNode) renderable = sheepBoxRenderable select() } }
  27. tada! You just draw 3 sheeps... in boxes!

  28. Sceneform Bring the fun to life!

  29. × Let’s build a more realistic sheep with basic shapes,

    color and a wool texture × Learn how to compose Nodes or display Android UI components in AR × Add animations to bring the sheep to life Sceneform: Sheep 102 Let’s go deeper with Sceneform
  30. × A Node represents a transformation within the scene graph’s

    hierarchy × Nodes have 1 parent and 0 to n child nodes and contain 1 renderable × To build a more realistic sheep we have to compose with several nodes Anchor Sceneform: Sheep 102 Node all the things! Sheep Node Sheep Node Sheep Node Sheep Node
  31. Sheep scene graph hierarchy y x y z Body Part

    Sheep Node Head Left ear Right ear Main Body Body Part Body Part Body Part Leg Leg Leg Leg
  32. Let’s now build our sheep // Create the Anchor. val

    anchorNode = AnchorNode(hitResult.createAnchor()) anchorNode.setParent(arFragment!!.arSceneView.scene) sheepNode = SheepNode(this).apply { setParent(anchorNode) } val legPositions = arrayOf( Vector3(-0.1f, 0f, 0.1f), Vector3(0.1f, 0f, 0.1f), Vector3(-0.1f, 0f, -0.1f), Vector3(0.1f, 0f, -0.1f) ) val legRenderables = arrayOf( sheepRightBackLeg, sheepRightFrontLeg, sheepLeftBackLeg, sheepLeftFrontLeg ) legPositions.forEachIndexed { index, legPosition -> Node().apply { renderable = legRenderables[index] localPosition = legPosition setParent(sheepNode) } } Here is a code sample to add the 4 legs of our sheep
  33. tada! We have now an awesome sheep in ar!

  34. You can even take selfies with him!

  35. But we can do more!

  36. × With Sceneform you can use Android UI components in

    AR × Build layouts like classic Android application (XML layouts) × All view interactions are working as usual (OnClick listener for example) Sceneform: Sheep 102 Let’s use Android UI components in AR
  37. Easily render Android UI sheepCard = Node() sheepCard!!.setParent(this) sheepCard!!.isEnabled =

    false sheepCard!!.localPosition = Vector3(0f, 0.5f, 0f) sheepCard!!.localRotation = Quaternion.axisAngle(Vector3(0f, 1f, 0f), 90f) sheepCard!!.localScale = Vector3(0.5f, 0.5f, 0.5f) ViewRenderable.builder() .setView(context, R.layout.layout_sheep_card) .build() .thenAccept { renderable -> sheepCard!!.renderable = renderable bindViews(renderable.view) animator = createAnimator(localPosition, Vector3(0f, 0f, 0.5f)) }
  38. And here is an ANdroid layout in AR!

  39. × For animating your 3D assets, play with position, scale

    and rotation × Use ObjectAnimator for animating Nodes × Compose animations thanks to AnimationSet × Let’s create a headbanging animation! Sceneform: Sheep 102 I like to move it! Move it!
  40. Let’s compose our animation private fun headRotationAnimator(headRotation: Quaternion) = ObjectAnimator().apply

    { setObjectValues(headRotation) setPropertyName("localRotation") setEvaluator(QuaternionEvaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE } private fun headTranslationAnimator(startPosition: Vector3, endPosition: Vector3) = ObjectAnimator().apply { setObjectValues(startPosition, endPosition) setPropertyName("localPosition") setEvaluator(Vector3Evaluator()) interpolator = AccelerateInterpolator() target = sheepHeadNode duration = 200 repeatMode = ObjectAnimator.REVERSE repeatCount = ObjectAnimator.INFINITE }
  41. Let’s compose our animation private fun headbangAnimation(headRotation: Quaternion) = AnimatorSet().apply

    { val rotation = Quaternion.multiply( headRotation, Quaternion.axisAngle(Vector3(0f, 0.5f, 0f), 45f)) val startPosition = sheepHeadNode!!.localPosition val endPosition = startPosition + Vector3(0f, -0.05f, 0f) playTogether( headRotationAnimator(rotation), headTranslationAnimator(startPosition, endPosition) ) }
  42. DEMO

  43. Sceneform Let’s play with awesome 3D assets

  44. × When you want to use complex 3D objects go

    for glTF format × Before Sceneform 1.16.0, you could use OBJ or FBX formats which was converted into *.sfa or *.sfb formats × Unfortunately the Android Studio plugin is broken since 3.6.0 Sceneform: Sheep 201 Let’s play with more complex 3D assets
  45. Place your screenshot here Where to find nice 3D assets?

  46. None
  47. None
  48. Load your 3D assets in you scene // For Sceneform

    1.16.0, load your glTF asset remotely or locally ModelRenderable.builder() .setSource(this, Uri.parse("https://my.api.com/sheep1.gltf")) .setIsFilamentGltf(true) .build().thenAccept { assetRendereable -> TransformableNode(arFragment!!.transformationSystem).apply { renderable = assetRendereable setParent(anchorNode) select() } }
  49. × ObjectAnimator can still be used to animate your asset

    × Before Sceneform 1.16.0, FBX format was the only supported format for animations × From Sceneform 1.16.0, glTF loading and animations are available thanks to Filament library Sceneform: Sheep 201 Animations with 3D assets
  50. Playing 3D asset animations // With Sceneform 1.15.0 val danceData

    = sheepRenderable.getAnimationData("sheep_move") val numAnimations = sheepRenderable.getAnimationDataCount() val moveData = sheepRenderable.getAnimationData(0) ModelAnimator(moveData, sheepRenderable).start() // For Sceneform 1.16.0, get the animation from the glTF asset thanks to Filament val filamentAsset = model.getRenderableInstance().getFilamentAsset(); if (filamentAsset.getAnimator().getAnimationCount() > 0) { animator = AnimationInstance(filamentAsset.getAnimator(), 0, System.nanoTime()) } animator.animator.applyAnimation(animator.index, time % animator.duration) animator.animator.updateBoneMatrices();
  51. CONgrats! You are now ready to build awesome AR apps!

  52. 3. Always more with ARcore Unlimited poweeeeer!

  53. Always more with arcore Augmented images Augmented faces Cloud anchors

    ⚓ And more!
  54. × Bring images to life × Use ARCore directly without

    Sceneform × ARCore can store up to 1,000 images references per image database × Use the arcoreimg command line tool to build your image reference database Augmented images Bring images to a new level!
  55. × Time to build your own filter! × Use face

    recognition to transform your face into a 3D mesh × Place your augmented face assets thanks to identified regions (NOSE_TIP, LEFT_FOREHEAD…) Augmented faces Selfie time!
  56. ☁ × Create collaborative AR experiences × Anchor data is

    stored on the cloud to be shared with multiple users × You can use Firebase for sharing Cloud Anchor or use your own solution × Permanent cloud anchor! Cloud anchors AR together! ⚓
  57. × Accurate mapping of your environment × Map the depth

    of your scene so your 3D assets might be hidden behind a box × Controlling object occlusion Depth API Mapping the world
  58. × Provides details info about the lighting in your scene

    × Enable immersive and realistic experiences × Two lighting mode available: ENVIRONMENTAL_HDR and AMBIENT_INTENSITY Lighting estimation Shine like a diamond!
  59. Want to know more? https://developers.google.com/ar/develop ARCore documentation & CodeLabs https://codelabs.developers.google.com

    Design guidelines for AR https://github.com/Oleur/SheepAR SheepAR source code https://designguidelines.withgoogle.com/ar-design
  60. Thanks ! Have fun with ARCore on Android! Any questions?

    @JulienSalvi