Let's add AR to your Android app

Sources: https://github.com/volodia-chornenkyy/arcore-basic-demo

Transcript

1. Let's add AR to your Android app! Volodia Chornenkyy @TechMagic @Good&Co
2. Augmented is not Virtual...
4. AR over VR • Cheaper (Oculus Rift - VR helmet - 12k UAH) • Greater audience (millions of mobile devices) • Easier to use (no need to use additional hardware)
5. AR use cases for mobile • Games • Navigation • Engineering • Marketing • Rest of the ideas that are not listed…
6. N minutes of theory
7. Fundamental concepts - Odometry Odometry is the use of data from motion sensors to estimate change in position over time.
8. Fundamental concepts - Feature points Feature point is a distinctive location in images – for example corners, blobs or T-junctions are usually good candidates.
9. Fundamental concepts - Planes Clusters of feature points that appear to lie on common horizontal or vertical surfaces, like tables or walls, and makes these surfaces available to your app as planes.
10. Fundamental concepts - Hit testing A hit test is generally a way to cast out a straight line from a point in space in some direction and determine if it intersects with any objects we're interested in.
11. Fundamental concepts - Anchors & Nodes Define an anchor to ensure that ARCore tracks the object's position over time. Nodes should be attached to an anchor.
12. Fundamental concepts - Vector3 A Vector3 is an object representing points and vectors in 3D space. Vector3 used in translation and scaling operations Shift y by 1: Vector3(0f,1f,0f) More: https://proandroiddev.com/arcore-cupcakes-3-understanding-vector3-93ec6bd08071
13. Fundamental concepts - Quaternion Quaternions are used for the object rotation (mechanics) in 3D space. Rotate y by 10 degrees: Quaternion.axisAngle(Vector3(0.0f, 1.0f, 0.0f), 10f) More: https://proandroiddev.com/arcore-cupcakes-4-understanding-quaternion-rotations-f90703f3966e
16. ARCore platforms support + NDK
17. ARCore features • Display 3D models • Light estimation • Augmented faces • Augmented images • Cloud Anchors • Sceneform
18. TODO https://github.com/volodia-chornenkyy/arcore-basic-demo
19. Show me the code
20. Integrate ARCore p.1 • Configure project build.gradle file ◦ minSdk 24 ◦ sourceCompatibility JavaVersion.VERSION_1_8 ◦ implementation 'com.google.ar:core:1.11.0' • Update AndroidManifest.xml ◦ uses-permission CAMERA ◦ (OPTIONAL) uses-feature android.hardware.camera.ar ◦ (OPTIONAL) meta-data com.google.ar.core required
21. Integrate ARCore p.2 - Official runtime check https://developers.google.com/ar/develop/java/enable-arcore fun maybeEnableArButton() { val availability = ArCoreApk.getInstance().checkAvailability(this@MainActivity) if (availability.isTransient) { // GooglePlay check Handler().postDelayed({ maybeEnableArButton() }, 200) } if (availability.isSupported) { // APK check mArButton.visibility = View.VISIBLE mArButton.isEnabled = true } else { // Unsupported or unknown. mArButton.visibility = View.INVISIBLE mArButton.isEnabled = false } } ArCoreHelper#askToInstallArCore() ArCoreApk.getInstance().requestInstall(activity, alwaysAskToInstall)
22. Integrate ARCore p.2 - Own runtime check if (status == ARCoreAvailabilityStatus.AVAILABLE) { if (!CameraPermissionHelper.hasCameraPermission(this@MainActivity)) { CameraPermissionHelper.requestCameraPermission(this@MainActivity) } else { showArContent() } } else if (status == ARCoreAvailabilityStatus.NEED_INSTALL) { arCoreHelper.askToInstallArCore() } fun checkIfArCoreAvailable(availabilityCallback: ARCoreAvailabilityCallback) { if (ArCoreApk.getInstance().checkAvailability(activity).isSupported.not()) { availabilityCallback.handleARCoreAvailability(ARCoreAvailabilityStatus.NEED_INSTALL) } else { availabilityCallback.handleARCoreAvailability(ARCoreAvailabilityStatus.AVAILABLE) } } ARCoreAvailabilityCallback
24. Integrate Sceneform p.1 • Import the Sceneform plugin • Configure project build.gradle file ◦ implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.11.0' ◦ OR implementation 'com.google.ar.sceneform:core:1.11.0'
25. Integrate Sceneform p.2 - Setup Session fun createArSession(): Session { val session = Session(activity) val config = Config(session) config.updateMode = UpdateMode.LATEST_CAMERA_IMAGE config.focusMode = Config.FocusMode.AUTO config.planeFindingMode = Config.PlaneFindingMode.HORIZONTAL session.configure(config) return session } arSceneView.setupSession(arCoreHelper.createArSession())
26. Integrate Sceneform p.3 - Setup interaction gestureDetector = GestureDetector( this, object : GestureDetector.SimpleOnGestureListener() { override fun onSingleTapUp(e: MotionEvent): Boolean { onSingleTap(e) return true } override fun onDown(e: MotionEvent): Boolean { return true } }) arSceneView.scene.setOnTouchListener { _: HitTestResult, event: MotionEvent -> return@setOnTouchListener gestureDetector.onTouchEvent(event) }
27. Integrate Sceneform p.4 - Handle interaction private fun onSingleTap(tap: MotionEvent) { val frame = arSceneView.arFrame if (frame != null) tryPlaceModel(tap, frame) } private fun tryPlaceModel(tap: MotionEvent?, frame: Frame): Boolean { if (tap != null && frame.camera.trackingState === TrackingState.TRACKING) { for (hit in frame.hitTest(tap)) { val trackable = hit.trackable if (trackable is Plane && trackable.isPoseInPolygon(hit.hitPose)) { val anchorNode = AnchorNode(hit.createAnchor()) anchorNode.setParent(arSceneView.scene) createNode(0f,0f) return true } } } return false }
28. Integrate Sceneform p.5 - Create model private fun createNode(x: Float, y: Float): Node { val base = Node() val modelNode = Node() modelNode.setParent(base) modelNode.localPosition = Vector3(x, y, 0f) MaterialFactory.makeOpaqueWithColor(this, Color(android.graphics.Color.RED)) .thenAccept { material -> val sphere = ShapeFactory.makeSphere(0.1f, Vector3(0.0f, 0.15f, 0.0f), material) modelNode.renderable = sphere } return base }
29. Integrate Sceneform p.6 - Model transformation • Node#setLocalPosition(Vector3) • Node#setLocalRotation(Quaternion) • Node#setLocalScale(Vector3) • Node#setWorldPosition(Vector3) • Node#setWorldRotation(Quaternion) • Node#setWorldScale(Vector3)
30. Integrate Sceneform p.6 - Model transformation modelNode.setOnTapListener { hitTestResult, motionEvent -> // hitTestResult.node } modelNode.setOnTouchListener { hitTestResult, motionEvent -> // hitTestResult.node }
31. private fun deleteNode(node: Node) { node.setParent(null) }