Measuring a room with ARKit: experiences and challenges

In October 2017, Terence, Jan and Adrian took 3 days to tackle the challenge of measuring a room with ARKit technology available in iOS 11 for capable devices. Here is a what they learned.

As presented at the App Builders Zurich meetup on 21st Feb 2018: https://www.meetup.com/App-Builders-Zurich/events/246680649

Transcript

5. Sponsor Scout 24 is hiring! http://scout24.ch/de/jobs-und-karriere/offene-stellen/
6. Organizers
7. Who • Adrian, Jan, Terence • iOS Engineers • Scout24
8. What • Experiments with Augmented reality (ARKit) • Create a working prototype • Have fun
9. Why • Solve a real user problem • Continuous education of ourselves • Learn a new paradigm
10. When & Where • 3 Days time (17th-19th October 2017) • Isolated from the rest of the world • Coworking space at the Effingerstrasse
13. ARKit • Augmented reality - Apple technology • Put virtual objects in the real space • Use the phone’s camera and sensors to recognise the surroundings
14. ARKit from WWDC - 2017 Apple
15. ARKit • Works on all modern devices running iOS 11 (from iPhone6s and from iPad Pros) • Already available on millions of devices • It’s quite precise in distances • Can recognise horizontal planes or individual points
16. from Youtube, by Mark Dawson: https://youtu.be/9ZnG9wrVxtM
17. Use case • Person will sell his house • He needs the floorplan for the house he’s selling • He needs the measures of his house
19. Solution • DEMO
20. Technical discussion • Main data structures in the SDK • Working with planes • Mathematical model
21. ARKit Ref: https://developer.apple.com/documentation/arkit ARKit combines device motion tracking, camera scene capture, to simplify the task of building an AR experience. advanced scene processing, and display conveniences
22. Content technologies • 3 possible content technologies (views): • SceneKit, for basic use cases • SpriteKit, for 2D images • Metal, for custom renderers (Unity/Unreal)
23. Scene and session class ViewController: UIViewController, ARSCNViewDelegate { @IBOutlet var sceneView: ARSCNView! override func viewDidLoad() { super.viewDidLoad() sceneView.delegate = self sceneView.session = ARSession() sceneView.scene = SCNScene() } override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.session.run(configuration) } // ... }
24. ARSCNViewDelegate
25. SCNNode and ARAnchor • SCNNode is a structural element in the hierarchical graph of our SCNScene • Akin to UIView in UIKit
 func addChildNode(_ child: SCNNode) • ARAnchor is a real-world position and orientation where we can place virtual objects in the scene • Rendering SCNNode’s over ARAnchor’s creates the AR experience
26. Plane detection: basic process 1. ARKit detects feature points in the world through the camera of the device 2. If plane detection is enabled, feature points are analyzed for planes 3.If a plane is recognized, an ARPlaneAnchor is reported in delegate method:
 renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) 4. We can place objects in the scene and they will appear as if they were in the plane defined in the real world
27. Plane detection: practical considerations • Plane detection is slow and depends on environmental factors. • The lag to detect the first plane poses a big challenge for the UX. 
 Some apps do a little teaser game to begin, like TweetReality by Oscar Falmer.
 Ref: https://itunes.apple.com/us/app/tweetreality/id1295207318? mt=8 • Some authors do their own heuristics based on cloud of feature points (ARPointCloud). Example: ARuler by duzexu
 Ref: https://github.com/duzexu/ARuler • Plane detection consumes resources, disable when you don’t need it! • New in ARKit 1.5: vertical planes
28. Detected Plane
29. Dragging Plane
30. Dragging Plane hitTest() Dragging Plane Detected Plane iPhone
31. Hit Test • Tap location = 1pt (center) • Far away point hard to Tap
32. Hit Test • Minigun Solution Finger Tap Area
33. Project a Point projectPoint() Dragging Plane Detected Plane iPhone Distance
34. Math • Position of our objects (SCNNode) —> SCNVector3 • X,y,z coordinates
35. Math
36. Math • Distance between points in the 3D space
37. Math
38. Math • Get the area of the polygon (3D) http://geomalgorithms.com/a01-_area.html
39. Unittests
40. Unittests Building time
41. Unittests ARKit find the plane
42. Unittests ARKit = NO BREAKPOINTS!
43. Unittests
44. Unittests
45. Unittests
46. Limitations • Speed of floor recognition —> slow • Lack of precision when you are far • Furniture • Not auto-find the walls, solved in ARKit 1.5
47. Technical aspects • Difficult to test • Not easy to think in 3D • A lot of math • User interface in 3D
48. Quality • The existing solutions have similar limitations • There is some “it depends”
49. Don’t use in train ;)
50. Questions? Terence Alberti: terence.alberti@gmail.com Jan Böhler: jan.boehler@scout24.ch Adrian Tineo: adrian.tineo@scout24.ch / @atineoSE