Augmented Reality Beyond ARKit

This talk describes in detail the limitations of iOS's ARKit technology, and why at Topology Eyewear we rely on our own machine learning & computer vision technology to provide our own augmented reality experience and to drive manufacturing of custom product.

Delivered in San Jose at AltConf 2018 simultaneously with WWDC.

Transcript

1. Augmented Reality Beyond ARKit @alexisgallagher alexis@TopologyEyewear.com
3. ARKit gives good face BlendShapeLocations: - measures facial expressions ARFaceGeometry.vertices - 1,220 vertex positions - stable w/r/t/ landmarks
4. What ARKit Does not give: EARS
5. What ARKit Does not give: EARS
6. Topics: - How we use AR - What ARKit does not give - AR without ARKit
7. How we use AR (at TOPOLOGY EYEWEAR)
8. We use AR: for shopping What will these look like on me? » Do these frames suit my face shape? » Does this material suit my complexion? » What does it look like from the side?
9. We use AR: for proof of fit But will these finally fit? » Will they be too big? » Will they slide down my nose? leave red marks? » Will they squeeze my ears?
10. We use AR: for manufacture How do we make these? » Temple length? » Splay angle? » Nosebridge contours?
11. We use AR: for manufacture How do we make these? » Temple length? » Splay angle? » Nosebridge contours?
13. We use AR: for manufacture How do we make these? » Temple length? » Splay angle? » Nosebridge contours?
14. We use AR: for manufacture How do we make these? » Temple length? » Splay angle? » Nosebridge contours?
15. What ARKit does not give
16. What ARKit Does not give: 1. EARS
17. What ARKit Does not give: 2. All the iPhones
18. What ARKit Does not give: 2. All the iPhones
19. What ARKit Does not give: 3. Accurate Scale
20. AR without ARKit
21. AR without ARKit Step 1. Selfie capture (iOS) Step 2. Computer vision (Linux) Step 3. TryOn Graphics (iOS)
22. AR without ARKit Step 1. Selfie capture (iOS) » CoreMotion for device orientation » AVFoundation for face detection, face pose detection, video capture, custom camera capture settings, video trimming, custom video metadata, camera intrinsics
23. AR without ARKit Step 2. Computer vision (Linux) » Multi-step pipeline, built on: » many open source libraries; ~80+ kLOC of our own Python and C++ code; proprietary refinements for the eyewear fitting problem; patented IP » CNNs (buzz!), ERTs, dedicated feature detectors, structure from motion, morphable face models, PLS regression, etc.. » Thousands of samples of manually labeled training data
24. AR without ARKit Step 3. TryOn Graphics » Trusty UIKit to composite an SCNView on top of a UImageView » Compute eyewear position, orientation, and occlusion based on data from server- side computer vision stack » Render glasses with custom Metal shaders in SceneKit
25. AR without ARKit Step 3. TryOn Graphics » Render glasses with custom Metal shaders in SceneKit
26. So ... is it worth it?
27. ARKit vs "TopoKit" ARKit TopoKit realtime Yes out-of-the-box Yes all the iPhones Yes ears Yes eyewear-accurate Yes
28. But ... is it necessary? A puzzling truth About TrueDepth
32. ARKit face-tracking is exclusive to iPhone X » Make sense: » only iPhone X has the "TrueDepth camera system" » so only iPhone X provides ARFaceAnchor » since the depth sensor enables face tracking...? » But then » if Face ID is so accurate, why is ARKit not?
34. Wait, ARKit doesn't use the depth sensor? No, ARKit doesn't use depth sensor!
35. So ... is it worth it? yes, since we need accuracy
37. Pupillary Distance
41. PD error value (mm) ARKit RMS average ~3.5 ARKit max ~7 TopoKit RMS average ~1.5 Optics requirements 2 Also: - consistent error with some subjects - high error variance, per trial, with facial error
42. Pupillary Distance Our solution: » Dedicated capture flow using back camera » Use card as reference object » Capture video to reduce error
43. Machine Learning Startup-style
49. Conclusions: » ARKit gives you magic! » That magic has some (arbitrary?) limits » But, with work, you can go beyond it » What are your real requirements? » Keep it simple. Sandwiches.
50. Augmented Reality Beyond ARKit @alexisgallagher alexis@topologyeyewear.com Interested? Please say hi!