Augmented Reality Beyond ARKit

Augmented Reality Beyond ARKit

This talk describes in detail the limitations of iOS's ARKit technology, and why at Topology Eyewear we rely on our own machine learning & computer vision technology to provide our own augmented reality experience and to drive manufacturing of custom product.

Delivered in San Jose at AltConf 2018 simultaneously with WWDC.

4b71cdf53bc76edd1a2ab66c446954b5?s=128

Alexis Gallagher

June 05, 2018
Tweet

Transcript

  1. 2.
  2. 6.

    Topics: - How we use AR - What ARKit does

    not give - AR without ARKit
  3. 8.

    We use AR: for shopping What will these look like

    on me? » Do these frames suit my face shape? » Does this material suit my complexion? » What does it look like from the side?
  4. 9.

    We use AR: for proof of fit But will these

    finally fit? » Will they be too big? » Will they slide down my nose? leave red marks? » Will they squeeze my ears?
  5. 10.

    We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  6. 11.

    We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  7. 12.
  8. 13.

    We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  9. 14.

    We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  10. 21.

    AR without ARKit Step 1. Selfie capture (iOS) Step 2.

    Computer vision (Linux) Step 3. TryOn Graphics (iOS)
  11. 22.

    AR without ARKit Step 1. Selfie capture (iOS) » CoreMotion

    for device orientation » AVFoundation for face detection, face pose detection, video capture, custom camera capture settings, video trimming, custom video metadata, camera intrinsics
  12. 23.

    AR without ARKit Step 2. Computer vision (Linux) » Multi-step

    pipeline, built on: » many open source libraries; ~80+ kLOC of our own Python and C++ code; proprietary refinements for the eyewear fitting problem; patented IP » CNNs (buzz!), ERTs, dedicated feature detectors, structure from motion, morphable face models, PLS regression, etc.. » Thousands of samples of manually labeled training data
  13. 24.

    AR without ARKit Step 3. TryOn Graphics » Trusty UIKit

    to composite an SCNView on top of a UImageView » Compute eyewear position, orientation, and occlusion based on data from server- side computer vision stack » Render glasses with custom Metal shaders in SceneKit
  14. 25.

    AR without ARKit Step 3. TryOn Graphics » Render glasses

    with custom Metal shaders in SceneKit
  15. 27.

    ARKit vs "TopoKit" ARKit TopoKit realtime Yes out-of-the-box Yes all

    the iPhones Yes ears Yes eyewear-accurate Yes
  16. 29.
  17. 30.
  18. 31.
  19. 32.

    ARKit face-tracking is exclusive to iPhone X » Make sense:

    » only iPhone X has the "TrueDepth camera system" » so only iPhone X provides ARFaceAnchor » since the depth sensor enables face tracking...? » But then » if Face ID is so accurate, why is ARKit not?
  20. 33.
  21. 36.
  22. 38.
  23. 39.
  24. 40.
  25. 41.

    PD error value (mm) ARKit RMS average ~3.5 ARKit max

    ~7 TopoKit RMS average ~1.5 Optics requirements 2 Also: - consistent error with some subjects - high error variance, per trial, with facial error
  26. 42.

    Pupillary Distance Our solution: » Dedicated capture flow using back

    camera » Use card as reference object » Capture video to reduce error
  27. 44.
  28. 45.
  29. 46.
  30. 47.
  31. 48.
  32. 49.

    Conclusions: » ARKit gives you magic! » That magic has

    some (arbitrary?) limits » But, with work, you can go beyond it » What are your real requirements? » Keep it simple. Sandwiches.