Add Reality to your App with ARCore and Sceneform

AR is a buzzword we hear everywhere. But is it really possible to build enhanced AR experiences quickly? There were quite a few attempts to bring AR to Android, but they were not easy to implement or required special hardware. ARCore solves both of the problems. It is a fast and performant SDK providing great API to build AR focused apps using just phone camera. During the talk we will discuss ARCore main concepts, geometry detection, 3D object rendering with Sceneform. I will also share potential problems and lessons learnt when building e-commerce AR experience at Jet.com.

1. @YuliyaKaleda Add Reality to App with ARCore
2. Jet.com AR experience
3. Tango several cameras sensors high cost $ few devices (Lenovo Phab 2 Pro, Asus Zenfone) D E P R E C A T E D
4. (Pixel, Samsung, LG V30, Asus) more low ARCore several cameras sensors devices cost $
5. Augmented Reality geometry detection 3D rendering +
6. ARCore geometry detection 3D rendering
7. ARCore OpenGL geometry detection 3D rendering +
8. ARCore Sceneform geometry detection 3D rendering +
9. ARCore Concepts motion tracking environmental understanding light estimation
10. Motion Tracking feature points camera position and orientation sensors
11. Environmental understanding planes surfaces with no texture !!!
12. Light estimation average intensity
13. Support Pixel, Samsung Galaxy, OnePlus, LG V30, Asus API 24 installed, ~ 45MB
14. Building AR experience • ARCore compatibility and installation • Geometry detection • 3D rendering
15. AR Apps AR optional AR required AR Stickers Jet.com
16. <manifest xmlns:android="http://schemas.android.com/apk/res/android" android:targetSandboxVersion="2" package=“com.jet.baselib"> <uses-feature android:name="android.hardware.camera.ar" android:required="true"/> <application android:name=“JetApp"> . . . <activity android:name=“.activities.SplashScreenActivity" /> <meta-data android:name="com.google.ar.core" android:value=“required" /> </application> </manifest> AR Required
17. <manifest xmlns:android="http://schemas.android.com/apk/res/android" android:targetSandboxVersion="2" package="com.jet.baselib"> <application android:name=“JetApp"> . . . <activity android:name=“.activities.SplashScreenActivity" /> <meta-data android:name="com.google.ar.core" android:value="optional" /> </application> </manifest> AR Optional
18. private void showArBanner() { ArCoreApk.Availability availability = ArCoreApk.getInstance() .checkAvailability(arCard.getContext()); if (availability.isTransient()) { new Handler().postDelayed(() -> showArBanner(), 200); } HelperViews.setVisibleOrGone(arCard, availability.isSupported()); } AR Optional
19. private void showArBanner() { ArCoreApk.Availability availability = ArCoreApk.getInstance() .checkAvailability(arCard.getContext()); if (availability.isTransient()) { new Handler().postDelayed(() -> showArBanner(), 200); } HelperViews.setVisibleOrGone(arCard, availability.isSupported()); } AR Optional
20. private boolean requestedARCoreInstall = true; public void onResume() { try { switch (ArCoreApk.getInstance().requestInstall(activity, requestedARCoreInstall)) { case INSTALLED: //Success break; case INSTALL_REQUESTED: //Request installation requestedARCoreInstall = false; return; } } catch (UnavailableUserDeclinedInstallationException e) { message = "Please install or update ARCore"; } catch (Exception e) { message = "This device does not support AR"; } } ARCore installation
21. private boolean requestedARCoreInstall = true; public void onResume() { try { switch (ArCoreApk.getInstance().requestInstall(activity, requestedARCoreInstall)) { case INSTALLED: //Success break; case INSTALL_REQUESTED: //Request installation requestedARCoreInstall = false; return; } } catch (UnavailableUserDeclinedInstallationException e) { message = "Please install or update ARCore"; } catch (Exception e) { message = "This device does not support AR"; } } ARCore installation
22. AR installation
23. Building AR experience • ARCore compatibility and installation • Geometry detection • 3D rendering
24. ARCore building blocks session
25. public void onResume() { if (session == null) { try { switch (ArCoreApk.getInstance().requestInstall(activity, requestedARCoreInstall)) { case INSTALLED: session = new Session(activity); break; case INSTALL_REQUESTED: requestedARCoreInstall = false; return; } } catch (UnavailableUserDeclinedInstallationException e) { ... } Config config = new Config(session); session.configure(config); } session.resume(); } Session
26. ARCore building blocks session
27. Session Config light estimation mode (AMBIENT_INTENSITY, DISABLED) plane finding mode (DISABLED, HORIZONTAL, VERTICAL, HORIZONTAL_AND_VERTICAL) update mode (BLOCKING, LATEST_CAMERA_IMAGE) cloud anchor mode (DISABLED, ENABLED)
28. public void onResume() { if (session == null) { try { switch (ArCoreApk.getInstance().requestInstall(activity, requestedARCoreInstall)) { case INSTALLED: session = new Session(activity); break; case INSTALL_REQUESTED: requestedARCoreInstall = false; return; } } catch (UnavailableUserDeclinedInstallationException e) { ... } // Create default config Config config = new Config(session); session.configure(config); } session.resume(); } Session Config
29. ARCore building blocks session frames
30. @Override public void onResume() { if (session == null) { return; } Frame frame = session.update(); } Frame
31. ARCore building blocks session frames camera
32. @Override public void onResume() { if (session == null) { return; } Frame frame = session.update(); Camera camera = frame.getCamera(); } Frame
33. ARCore building blocks session frames camera light light
34. @Override public void onResume() { if (session == null) { return; } Frame frame = session.update(); Camera camera = frame.getCamera(); float lightIntensity = frame.getLightEstimate().getPixelIntensity(); } Frame
35. ARCore building blocks session frames camera light hitResult
36. HitResult feature points planes
37. @Override public void onResume() { if (session == null) { return; } Frame frame = session.update(); Camera camera = frame.getCamera(); float lightIntensity = frame.getLightEstimate().getPixelIntensity(); List<HitResult> hitResults = frame.hitTest(position.x, position.y); //List<HitResult> hitResults = frame.hitTest(motionEvent); } Frame
38. @Override public void onResume() { if (session == null) { return; } Frame frame = session.update(); Camera camera = frame.getCamera(); float lightIntensity = frame.getLightEstimate().getPixelIntensity(); //List<HitResult> hitResults = frame.hitTest(position.x, position.y); List<HitResult> hitResults = frame.hitTest(motionEvent); } Frame
39. @Override public void onResume() { . . . List<HitResult> hitResults = frame.hitTest(position.x, position.y); if (hitResults.size() > 0) { HitResult hit = getClosestHit(hitResults); } } private HitResult getClosestHit(List<HitResult> hitResults) { for (HitResult hitResult : hitResults) { if (hitResult.getTrackable() instanceof Plane) { return hitResult; } } return hitResults.get(0); } HitResult
40. ARCore building blocks session frames camera light HitResult anchors
41. @Override public void onResume() { . . . List<HitResult> hitResults = frame.hitTest(position.x, position.y); if (hitResults.size() > 0) { HitResult hit = getClosestHit(hitResults); Anchor anchor = hit.createAnchor(); } } Anchor
42. Building AR experience • ARCore compatibility and installation • Geometry detection • 3D rendering
43. <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:app="http://schemas.android.com/apk/res-auto" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" tools:context="com.jet.arlib.activities.ArActivity"> <com.google.ar.sceneform.ArSceneView android:id=“@+id/sceneview" android:layout_width="match_parent" android:layout_height="match_parent"/> </RelativeLayout> UI setup
44. public class ArActivity extends JetActivity { private ArSceneView arSceneView; @Override protected void onCreate(Bundle savedInstanceState) { arSceneView = findViewById(R.id.sceneview); } @Override protected void onResume() { if (session != null) { Config config = new Config(session); config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE); session.configure(config); arSceneView.setupSession(session); try { arSceneView.resume(); } catch (CameraNotAvailableException ex) { } } } } UI setup
45. UI setup <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:app="http://schemas.android.com/apk/res-auto" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" tools:context="com.jet.arlib.activities.ArActivity"> <fragment android:name="com.google.ar.sceneform.ux.ArFragment" android:id=“@+id/ar_fragment" android:layout_width="match_parent" android:layout_height=“match_parent"/> </RelativeLayout>
46. 3D rendering geometry texture + .obj | .glTF | .fbx .png | .webP
47. 3D rendering geometry texture o GoogleHome_GEO v 9.16292e-006 0.00498037 -0.0290344 v -0.00769833 0.00176132 -0.0292111 v -0.0148647 0.00176132 -0.0261927 v -0.0210186 0.00176132 -0.0213912 v -0.0257407 0.00176132 -0.0151337 v -0.0287091 0.00176132 -0.00784677 v -0.0297216 0.00176132 -2.6893e-005 v -0.0287091 0.00176132 0.00779299 v -0.0257407 0.00176132 0.01508 v -0.0210186 0.00176132 0.0213374 v -0.0148647 0.00176132 0.0261389 v -0.00769833 0.00176132 0.0291573 v -7.85162e-006 0.00176132 0.0301868 v 0.00768262 0.00176132 0.0291573 v 0.014849 0.00176132 0.0261389 v 0.0210029 0.00176132 0.0213374 v 0.025725 0.00176132 0.01508 v 0.0286934 0.00176132 0.00779299 … .obj .png
48. Sceneform Plugin
49. Sceneform Plugin
50. Sceneform Plugin
51. Sceneform Plugin
52. Sceneform Plugin
53. Sceneform Plugin
54. .sfa file { materials: [ { name: "diffuseTex", parameters: [ { baseColor: "googlehome", }, { baseColorTint: [ 0.47999999999999998, 0.47999999999999998, 0.47999999999999998, 1, ], }, { metallic: 1, }, { roughness: 0.088044999999999998, }, { opacity: null, }, ], source: "build/sceneform_sdk/default_materials/obj_material.sfm", }, ], model: { attributes: [ "Position", "TexCoord", "Orientation", ], file: "../app/sampledata/googlehome.obj", name: "googlehome", suggested_collision: { center: { x: 0, y: 0, z: 0, }, size: { x: 1, y: 1, z: 1, }, type: "Box", }, }, samplers: [ { file: "../app/sampledata/googlehome.png", name: "googlehome", pipeline_name: "googlehome.png", }, ], version: "0.51:1", }
55. Sceneform Plugin apply plugin: 'com.android.application' android { } dependencies { } apply plugin: 'com.google.ar.sceneform.plugin' sceneform.asset('../app/sampledata/googlehome.obj', 'default', '../app/sampledata/googlehome.sfa', 'src/main/assets/googlehome') :createAsset-src_47main_47assets_47googlehome => .sfa :compileAsset-src_47main_47assets_47googlehome => .sfb
56. private ModelRenderable googleHomeRenderable; public void drawProduct(MotionEvent event) { ModelRenderable.builder() .setSource(this, Uri.parse(“googlehome.sfb")) //R.raw.googlehome .build() .thenAccept(renderable -> googleHomeRenderable = renderable) .exceptionally( throwable -> { Log.e(TAG, "Unable to load Renderable.", throwable); return null; }); } Finally draw
57. private ModelRenderable googleHomeRenderable; public void drawProduct(MotionEvent event) { . . . Frame frame = arSceneView.getArFrame(); if (frame.getCamera().getTrackingState() != TrackingState.TRACKING) return; List<HitResult> hitResults = frame.hitTest(event); if (hitResults.size() > 0) { HitResult hit = getClosestHit(hitResults); Anchor anchor = hit.createAnchor(); } } Finally draw
58. private ModelRenderable googleHomeRenderable; public void drawProduct(MotionEvent event) { . . . Frame frame = arSceneView.getArFrame(); if (frame.getCamera().getTrackingState() != TrackingState.TRACKING) return; List<HitResult> hitResults = frame.hitTest(event); if (hitResults.size() > 0) { HitResult hit = getClosestHit(hitResults); Anchor anchor = hit.createAnchor(); AnchorNode anchorNode = new AnchorNode(anchor); anchorNode.setParent(arSceneView.getScene()); } } Finally draw
59. private ModelRenderable googleHomeRenderable; public void drawProduct(MotionEvent event) { . . . Frame frame = arSceneView.getArFrame(); if (frame.getCamera().getTrackingState() != TrackingState.TRACKING) return; List<HitResult> hitResults = frame.hitTest(event); if (hitResults.size() > 0) { HitResult hit = getClosestHit(hitResults); Anchor anchor = hit.createAnchor(); AnchorNode anchorNode = new AnchorNode(anchor); anchorNode.setParent(arSceneView.getScene()); Node googleHomeNode = new Node(); googleHomeNode.setRenderable(googleHomeRenderable); anchorNode.addChild(googleHomeNode); } } Finally draw
60. Attention!!!
61. UI guidance
62. Performance don’t rely on world coordinates, use anchors > 12 anchors don’t use vibration detect a plane if available, else closest feature point 30 fps or higher lock orientation
63. private void moveActiveObject(Session session, Frame frame, MotionEvent event) { List<HitResult> hitResults = frame.hitTest(event); Anchor newAnchor = null; if (hitResults.size() > 0) { HitResult hit = hitResults.get(0); newAnchor = hit.createAnchor(); } if (newAnchor != null) { Pose cameraPose = frame.getCamera().getPose(); Pose anchorPose = newAnchor.getPose(); float distance = getDistanceBetweenPoses(cameraPose, anchorPose); if (distance > OBJ_MAX_DISTANCE) return; activeArObject.getAnchor().detach(); activeArObject.setAnchor(newAnchor); } } Distance
64. if (anchor.getTrackingState() != TrackingState.TRACKING) { //Do not draw objects } if (camera.getTrackingState() == TrackingState.PAUSED) { return; } Tracking State TRACKING PAUSED STOPPED
65. Thank you! @YuliyaKaleda