Augmented Reality Beyond ARKit

This talk describes in detail the limitations of iOS's ARKit technology, and why at Topology Eyewear we rely on our own machine learning & computer vision technology to provide our own augmented reality experience and to drive manufacturing of custom product.

Delivered in San Jose at AltConf 2018 simultaneously with WWDC.

Transcript

1. Augmented Reality
   Beyond ARKit
   @alexisgallagher
   alexis@TopologyEyewear.com
   

   View Slide

2. View Slide

3. ARKit gives
   good face
   BlendShapeLocations:
   - measures facial expressions
   ARFaceGeometry.vertices
   - 1,220 vertex positions
   - stable w/r/t/ landmarks
   

   View Slide

4. What ARKit
   Does not give:
   EARS
   

   View Slide

5. What ARKit
   Does not give:
   EARS
   

   View Slide

6. Topics:
   - How we use AR
   - What ARKit does not give
   - AR without ARKit
   

   View Slide

7. How we use AR
   (at TOPOLOGY EYEWEAR)
   

   View Slide

8. We use AR: for shopping
   What will these look
   like on me?
   » Do these frames suit my
   face shape?
   » Does this material suit my
   complexion?
   » What does it look like from
   the side?
   

   View Slide

9. We use AR: for proof of fit
   But will these
   finally fit?
   » Will they be too big?
   » Will they slide down my
   nose? leave red marks?
   » Will they squeeze my ears?
   

   View Slide

10. We use AR: for manufacture
    How do we make
    these?
    » Temple length?
    » Splay angle?
    » Nosebridge contours?
    

    View Slide

11. We use AR: for manufacture
    How do we make
    these?
    » Temple length?
    » Splay angle?
    » Nosebridge contours?
    

    View Slide

12. View Slide

13. We use AR: for manufacture
    How do we make
    these?
    » Temple length?
    » Splay angle?
    » Nosebridge contours?
    

    View Slide

14. We use AR: for manufacture
    How do we make
    these?
    » Temple length?
    » Splay angle?
    » Nosebridge contours?
    

    View Slide

15. What ARKit
    does not give
    

    View Slide

16. What ARKit
    Does not give:
    1. EARS
    

    View Slide

17. What ARKit
    Does not give:
    2. All the
    iPhones
    

    View Slide

18. What ARKit
    Does not give:
    2. All the
    iPhones
    

    View Slide

19. What ARKit
    Does not give:
    3. Accurate
    Scale
    

    View Slide

20. AR without ARKit
    

    View Slide

21. AR without ARKit
    Step 1. Selfie capture (iOS)
    Step 2. Computer vision (Linux)
    Step 3. TryOn Graphics (iOS)
    

    View Slide

22. AR without ARKit
    Step 1. Selfie capture
    (iOS)
    » CoreMotion for device
    orientation
    » AVFoundation for face
    detection, face pose
    detection, video
    capture, custom camera
    capture settings, video
    trimming, custom
    video metadata, camera
    intrinsics
    

    View Slide

23. AR without ARKit
    Step 2. Computer vision (Linux)
    » Multi-step pipeline, built on:
    » many open source libraries; ~80+ kLOC of our own
    Python and C++ code; proprietary refinements for
    the eyewear fitting problem; patented IP
    » CNNs (buzz!), ERTs, dedicated feature detectors,
    structure from motion, morphable face models,
    PLS regression, etc..
    » Thousands of samples of manually labeled
    training data
    

    View Slide

24. AR without ARKit
    Step 3. TryOn
    Graphics
    » Trusty UIKit to composite
    an SCNView on top of a
    UImageView
    » Compute eyewear position,
    orientation, and occlusion
    based on data from server-
    side computer vision stack
    » Render glasses with custom
    Metal shaders in SceneKit
    

    View Slide

25. AR without ARKit
    Step 3. TryOn
    Graphics
    » Render glasses with custom
    Metal shaders in SceneKit
    

    View Slide

26. So ... is it
    worth it?
    

    View Slide

27. ARKit vs "TopoKit"
    ARKit TopoKit
    realtime Yes
    out-of-the-box Yes
    all the iPhones Yes
    ears Yes
    eyewear-accurate Yes
    

    View Slide

28. But ... is it necessary?
    A puzzling truth
    About TrueDepth
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. ARKit face-tracking is exclusive to iPhone X
    » Make sense:
    » only iPhone X has the "TrueDepth camera system"
    » so only iPhone X provides ARFaceAnchor
    » since the depth sensor enables face tracking...?
    » But then
    » if Face ID is so accurate, why is ARKit not?
    

    View Slide

33. View Slide

34. Wait, ARKit doesn't use the depth sensor?
    No, ARKit doesn't use depth sensor!
    

    View Slide

35. So ... is it worth it?
    yes, since we need
    accuracy
    

    View Slide

36. View Slide

37. Pupillary Distance
    

    View Slide

38. View Slide

39. View Slide

40. View Slide

41. PD error value (mm)
    ARKit RMS average ~3.5
    ARKit max ~7
    TopoKit RMS average ~1.5
    Optics requirements 2
    Also:
    - consistent error with some subjects
    - high error variance, per trial, with facial error
    

    View Slide

42. Pupillary Distance
    Our solution:
    » Dedicated capture flow
    using back camera
    » Use card as reference
    object
    » Capture video to reduce
    error
    

    View Slide

43. Machine Learning
    Startup-style
    

    View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. Conclusions:
    » ARKit gives you magic!
    » That magic has some (arbitrary?) limits
    » But, with work, you can go beyond it
    » What are your real requirements?
    » Keep it simple. Sandwiches.
    

    View Slide

50. Augmented Reality
    Beyond ARKit
    @alexisgallagher
    alexis@topologyeyewear.com
    Interested? Please say hi!
    

    View Slide