Making things speak with ARKit and ARCore

Transcript

1. Making things speak with ARKit and ARCore Christof Roduner, PhD - VP Engineering and co-founder, Scandit Luca Torella, PhD - Senior iOS Software Engineer, Scandit MCE Conference, 6 June 2018, Warsaw
3. AR for barcode applications
5. 2017 - The Year of AR Jun 26, 2017 Aug 29, 2017
6. Immersive experiences for everyone Pokemon Go IKEA The Walking Dead
7. What do these apps have in common? Physical space is augmented with digital content Content is often highly visual or realistic looking Perspective on content changes when phone is moved Replace your imagination with a visualization
8. A slightly different angle Augmenting physical objects with digital content (vs. augmenting spaces) Things can speak to you with AR “IoT without embedded computing” Many applications to improve businesses processes Make content from databases accessible through AR
9. Inventory management
11. Allergy checker
13. Hololens-based order picking
15. Price label verification
17. Last mile delivery
19. augmenting physical objects = lots of applications in business
20. This Talk 1. Adding content to the physical space ◦ ARKit and ARCore basics 2. Identifying physical objects ◦ With a template image (“augmented images”) ◦ By its shape ◦ With ML 3. Challenges and limitations
21. Adding content to the physical space
22. 3D coordinate system of the world Goal: perspective of content changes as phone moves Need 3D coordinate system of the physical world Need tracking of the phone’s 3D coordinates Need 3D coordinates for content (anchor) Content position Phone position
23. 3D with a single phone camera? Generally, 3D requires two images from two different positions (human eye) Visual inertial odometry (VIO): • Capture image • Change position and measure movement with IMU • Capture image again and calculate 3D image Inertial measurement unit (IMU): • Multiple sensors to measure acceleration • Dead reckoning to “guess” how far you move • Very hard to calibrate • Hardware / software integration helps (Apple) Camera movement
24. ARCore device support Asus Zenfone AR, Zenfone ARES Google Nexus 5X, Nexus 6P, Pixel, Pixel XL, Pixel 2, Pixel 2XL Nokia Nokia 6.1, Nokia 8 Sirocco Huawei P20, P20 Pro, Mate RS Porsche Design LG G6, G7 ThinQ, V35 ThinQ Motorola Moto G5S Plus, Moto G6 Plus, Moto Z2 Force OnePlus OnePlus 3T, OnePlus 5 Samsung Galaxy A5 (2017), Galaxy A7 (2017), Galaxy A8, Galaxy A8+ (2018), Galaxy Note8, Galaxy S7, Galaxy S7 edge, Galaxy S8, Galaxy S8+, Galaxy S9, Galaxy S9+ Sony Xperia XZ Premium, Xperia XZ1, Xperia XZ1 Compact Xiaomi Mi Mix 2S Approx. 35 Android devices supported as of June 2018
25. Floating in space is great, but... vs. Vehicle flying in space Candle standing on a table Plane detection to the rescue!
26. Plane detection Today’s frameworks can detect planes Uses feature points Requires texture, uniform surfaces won’t work Only horizontal and vertical planes are supported
27. Hit testing Where on a plane should content be placed? Ask user to tap phone screen to select point Send ray into the scene and find 3D coordinates of intersection with plane
28. “Hello World”
31. Rendering (ARKit) SpriteKit SceneKit Metal
32. Rendering (ARCore) OpenGL Sceneform BETA
36. ARSession ARConfiguration ARFrame AVCaptureSession CMMotionManager ARFrame ARFrame ARKit basic concepts
37. ARConfiguration ARFrame ARFrame ARFrame ARSCNView ARSession ARSCNViewDelegate ARKit and SceneKit
38. Loading the initial scene // Set the view's delegate sceneView.delegate = self // Create a new scene let scene = SCNScene(named: "art.scnassets/spaceShip.scn")! // Set the scene to the view sceneView.scene = scene
39. Starting the session // Create a session configuration let configuration = ARWorldTrackingConfiguration() // Run the view's session sceneView.session.run(configuration)
40. Placing objects on a plane
42. Plane detection let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = [.horizontal, .vertical]
43. Anchors Represent a point or a plane in the real world and are used to add content List of anchors is maintained by ARKit throughout the session How anchors are added: • Programmatically by your code • Automatically when a new plane (or image, face, ...) is detected You can get notified whenever an anchor is added / removed / updated Tracking becomes more reliable in the area around an anchor
44. Automatically added anchors func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } // Show to the user that a new surface has been detected. // … }
45. Hit testing @objc private func handleTap(gestureRecognizer: UITapGestureRecognizer) { let point = gestureRecognizer.location(in: sceneView) addCandle(at: position) } let hitResults = sceneView.hitTest(point, types: [.existingPlaneUsingGeometry]) guard let firstHit = hitResults.first else { return } let position = SCNVector3(firstHit.worldTransform.columns.3.x, firstHit.worldTransform.columns.3.y, firstHit.worldTransform.columns.3.z)
46. Manually added anchors (with SceneKit) func addCandle(at position: SCNVector3) { let candle = SCNScene(named: "art.scnassets/candle.scn")! sceneView.scene.rootNode.addChildNode(candleNode) } let candleNode = candle.rootNode.childNode(withName: "candle", recursively: false) candleNode.position = position
47. Light estimation
49. Light estimation sceneView.automaticallyUpdatesLighting = true
50. Identifying objects
51. Approaches to identify objects With a template With ML “car” image 3D shape concept
52. Template images With a template With ML “car” image 3D shape concept
53. Template images Practical requirements: • Surface with real-world image needs to be flat, no curved objects • Need to know approximate size of real-world image • Width / height ratio needs to be within a certain range Availability by platform: • ARKit 1.5 - released in April (in beta since January) • ARCore 1.2 - released last month
55. Adding the template images
56. Configuration let images = ARReferenceImage.referenceImages(inGroupNamed: "AR Resources", bundle: nil)! let configuration = ARWorldTrackingConfiguration() configuration.detectionImages = images session.run(configuration)
57. Highlighting the image location func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let imageAnchor = anchor as? ARImageAnchor else { return } // Highlight the image found... }
58. 3D shapes With a template With ML “car” image 3D shape concept
59. Scanning a 3D template A R K it 2.0 • Costly in terms of processing power • Disable some ARKit features ARSession ARObjectScanningConfiguration ARReferenceObject .arobject
60. Identifying a 3D object A R K it 2.0 let referenceObjects = ARReferenceObject.referenceObjects(inGroupNamed: "Objects", bundle: nil) let configuration = ARWorldTrackingConfiguration() configuration.detectionObjects = referenceObjects sceneView.session.run(configuration) ARWorldTrackingConfiguration ARReferenceObject .arobject
61. Identifying a 3D object A R K it 2.0 func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let imageAnchor = anchor as? ARObjectAnchor else { return } // ... }
62. Machine learning With a template With ML “car” image 3D shape concept
63. Classifier Model (classifier) hot_dog: 0.89 dog: 0.08 cat: 0.05 apple: 0.01 boat: 0.27 labels and scores example image of an object training data
64. Mobile machine learning frameworks June 2017 May 2018
65. Two ways to obtain a model 1. Use a pre-trained model ◦ ImageNet: recognize object type from a set of 1000 categories (e.g., animals, foods, vehicles, trees, people, etc.) ◦ MIT Places: recognize scene from a set of 205 categories (e.g., airport, bedroom, park, coast, etc.) ◦ Age, gender, emotion, brand, ... 2. Use your own model ◦ Train own model with a framework like TensorFlow, Keras, Caffe, Turi Create, etc. ◦ Convert to mlmodel (Apple) or tflite (Google) Use model in Core ML (Apple) or ML Kit (Google) and run on the device
66. Core ML and ARKit ARSession ARFrame Output Vision
67. Core ML and ARKit let model = try VNCoreMLModel(for: Model().model) Model.mlmodel Model Model ModelInput ModelOutput let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer) try handler.perform([request]) request = VNCoreMLRequest(model: model) { (request, error) in // do something with the result... } Process frames on a separate thread Don’t process every frame
68. Comparing Core ML and ML Kit Similar API and feature set Core ML • Available for iOS only (since iOS 11) • Runs on the device ML Kit • Available for Android and iOS (beta released last month) • Runs on the device or in the cloud
69. Limitations of object recognition Problem: You can only detect which object the image contains But where is it? Localizing objects requires a more sophisticated model (“object detection”) YOLO (“you only look once”)
70. Thanks for listening! Btw, we are hiring in Warsaw...
71. Questions?