Herding Llamas in your own flat - A deep dive into ARCore and Sceneform (Droidcon Berlin 2018)

ARCore left the developer preview in February and is now available for production apps. Given his experience in developing the yourhome AR app that was presented at Google’s ARCore booth at the MWC 2018 Steven will give a deep dive into creating an native AR Android app using ARCore. The talk will be accompanied by an open source sample app that allows users to place llamas in the AR scene and interact with them. Based on this example app the talk will cover how to setup an AR scene beginning with calibration to find the first real-world planes and placing virtual objects (llamas) on these planes. Using animation and collision detection these llamas will move on the detected floor in the room. The user will also be able to place fences that will keep the llamas in place. By touching the llamas and fences the user can pet them resp. move the fences. As the last missing piece the 3D assets for the llamas and the fences will be loaded from Google's Poly.

Transcript

1. Herding Llamas in your own flat - A deep dive into ARCore and Sceneform Steven Mohr, Smart As Apps GmbH Droidcon Berlin 2018, 26 June 2018
2. Agenda • Short introduction about ARCore and Sceneform • Model import • AR Calibration • Object placement and creation • Shadows and Animations • All code samples are from the example app: 
 github.com/SmartAsApps/HerdingLlamas
3. Who’s that guy? • Steven Mohr, Android developer since Android 1.5 • Founded Smart As Apps in 2014, doing freelance and app development since then • Started to work with Sceneform in January 2018 as part of the EAP for my client otto group • The app yourhome Augmented reality was presented at Google’s AR booth at MWC 2018 and was part of different presentations at Google I/O 2018
4. The sample app • Place llamas and fences in AR • Allow to touch llamas and get a reaction • Llamas should move but avoid other llamas and fences
6. What is ARCore? • Google’s Augmented Reality framework • Successor of Project Tango but without special hardware • Finds features in the real world and keeps them in sync with the virtual world
7. What is Sceneform? • Released in May at Google I/O 2018 • Lightweight 3D rendering library • Handles ARCore init • Provides an easy start for new AR projects and helps to link ARCore with the renderer • More details about the renderer: Building AR apps with the Sceneform SDK, Google I/O 18 (https://youtu.be/jzaMMV6w_OE)
8. How to start? • ArFragment • Handling camera permissions • Checks whether ARCore is installed and handles installation if necessary • Provides standard gesture support • ArSceneView • A SurfaceView for Sceneform • ARCore has to be setup independently
9. Are you a 3D artist? I’m not…
11. Model loading • Import via Sceneform Android Studio Plugin • Adds tasks to build.gradle • For every asset there a two new Gradle tasks • One to create the SFA file (Sceneform Asset) • One to create the SFB file (Sceneform Bundle) • At least one of them is called with every build
12. Scriptable model loading $ ./gradlew app:tasks —all compileAsset-src_47main_47res_47raw_47llama createAsset-src_47main_47res_47raw_47llama $ ./gradlew createAsset-src_47main_47res_47raw_47llama —-info […] Info: Wrote SFA (json) to /Users/steven/Business/UrbanLlamaFarmer/app/ build/.sfa6838821559768885463/Llama.sfa (1240 bytes) > Task :app:createAsset-src_47main_47res_47raw_47llama Command Line: /Users/steven/UrbanLlamaFarmer/app/build/sceneform_sdk/mac/ converter --no_sfb --mat build/sceneform_sdk/default_materials/ obj_material.sfm -a --matc /Users/steven/UrbanLlamaFarmer/app/build/ sceneform_sdk/mac/matc --outdir /Users/steven/UrbanLlamaFarmer/app/ build/.sfa6838821559768885463 sampledata/models/llama/Llama.obj
13. Scriptable model loading • The SFAs are a best-guess so for more complex ones you have to fine-tune them. • The SFAs are JSON-like: Use Python and the hjson module for updating the SFAs • Calculate a real bounding box • Assign textures to right channels • Maybe down-size your textures
14. Scriptable model loading $ ./gradlew compileAsset-src_47main_47res_47raw_47llama —-info […] Info: Wrote SFB to src/main/res/raw/llama.sfb (1777024 bytes) > Task :app:compileAsset-src_47main_47res_47raw_47llama Command Line: /Users/steven/Business/UrbanLlamaFarmer/app/build/ sceneform_sdk/mac/converter sampledata/models/llama/Llama.sfa --desktop -- output src/main/res/raw/llama
15. Let’s give ARCore a little help …
16. Calibration • Included in ArFragment • You can update the shown view via val myView = … arFragment.planeDiscoveryController.setInstructionView(myView) • Automatically disappears as soon as a plane is detected
18. Calibration • With ArSceneView : Do it on your own • Register an Scene.OnUpdateListener with your ARSceneView • Call on every frame: arSceneView.arFrame.getUpdatedTrackables(Plane::class.java). firstOrNull { it.trackingState == TrackingState.TRACKING && it.type == Plane.Type.HORIZONTAL_UPWARD_FACING }?.let { // Finish onboarding }
20. Free the llamas!
21. Object placement • Included in ArFragment arFragment.setOnTapArPlaneListener { hitResult, plane, motionEvent -> val scene = arFragment.arSceneView?.scene ?: return@setOnTapArPlaneListener val node = objectPlacement.placeObject(hitResult) scene.addChild(node) } class ObjectPlacement(private val nodeCreator: NodeCreator) { fun placeObject(hitResult: HitResult): Node { val anchor = hitResult.createAnchor() val anchorNode = AnchorNode(anchor) placementType.createNode(nodeCreator).setParent(anchorNode) return anchorNode } }
22. Object creation class NodeCreator constructor(private val transformationSystem: TransformationSystem) { private lateinit var llamaRenderable: ModelRenderable fun init(context: Context) { ModelRenderable.builder().setSource(context, R.raw.llama).build().thenAccept { llamaRenderable = it } } fun createLlama() : Node { val llamaNode = LlamaNode({ createHeart() }, transformationSystem) llamaNode.renderable = llamaRenderable return llamaNode } }
23. TransformableNode class LlamaNode constructor(private val heartProvider: () -> HeartNode, transformationSystem: TransformationSystem) : TransformableNode(transformationSystem) { init { scaleController.isEnabled = false // scaleController.minScale = 0.5f // scaleController.maxScale = 1.5f translationController.isEnabled = false rotationController.isEnabled = false } override fun onTap(hitTestResult: HitTestResult?, motionEvent: MotionEvent?) { //select() startPetAnimation() } }
24. Llamas are no ghosts!
25. Collision detection • Very basic -> just tests for overlaps class LlamaNode … { override fun onUpdate(frameTime: FrameTime?) { super.onUpdate(frameTime) resolveOverlap() } } fun Node.resolveOverlap() { val overlappingNode = scene?.overlapTest(this) ?: return if (overlappingNode is HeartNode) return val direction = Vector3.subtract(worldPosition, overlappingNode.worldPosition) direction.y = 0f val rejection = 0.1f worldPosition = Vector3.add(worldPosition, direction.normalized().scaled(rejection)) }
26. Shadows • Renderables can be shadow caster and/or receiver llamaRenderable.isShadowCaster = true llamaRenderable.isShadowReceiver = false • Detected planes can be shadow receiver or not arFragment.arSceneView.planeRenderer.isShadowReceiver = false
28. Animations • Can be done „manually“ or by using Android Animations
30. Android Animation class LlamaNode … { private fun startPetAnimation() { val heartNode = heartProvider() addChild(heartNode) val animator = createAnimator() animator.target = heartNode animator.addListener(object : SimpleAnimatorListener() { override fun onAnimationEnd(animation: Animator) { removeChild(heartNode) } }) animator.start() } private fun createAnimator(): ObjectAnimator { val animator = ObjectAnimator() animator.setObjectValues(Vector3(0f, 1.5f, 0f), Vector3(0f, 2.5f, 0f)) animator.propertyName = "localPosition" animator.duration = 750 animator.repeatCount = 1 animator.setEvaluator(Vector3Evaluator()) return animator } }
31. Sceneform Animation class HeartNode : Node() { override fun onUpdate(frameTime: FrameTime) { super.onUpdate(frameTime) localRotation = Quaternion.axisAngle(Vector3(0f, 1f, 0f), (frameTime.startSeconds*500) % 360) } }
32. Let’s look at the sample app ✓Place llamas and fences in AR ✓Allow to touch llamas and get a reaction ✓Llamas should move but avoid other llamas and fences
33. Summary • Sceneform is a great start for AR apps • Hopefully Google will add support for animated renderables in the next releases • Documentation is a bit sparse at the moment
34. Thank you! • Sample app: https://github.com/SmartAsApps/HerdingLlamas • Twitter: @smartasapps • Website: www.smartasapps.de • E-Mail: steven.mohr@smartasapps.de