Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Augmented Reality Beyond ARKit

Augmented Reality Beyond ARKit

This talk describes in detail the limitations of iOS's ARKit technology, and why at Topology Eyewear we rely on our own machine learning & computer vision technology to provide our own augmented reality experience and to drive manufacturing of custom product.

Delivered in San Jose at AltConf 2018 simultaneously with WWDC.

4b71cdf53bc76edd1a2ab66c446954b5?s=128

Alexis Gallagher

June 05, 2018
Tweet

Transcript

  1. Augmented Reality Beyond ARKit @alexisgallagher alexis@TopologyEyewear.com

  2. None
  3. ARKit gives good face BlendShapeLocations: - measures facial expressions ARFaceGeometry.vertices

    - 1,220 vertex positions - stable w/r/t/ landmarks
  4. What ARKit Does not give: EARS

  5. What ARKit Does not give: EARS

  6. Topics: - How we use AR - What ARKit does

    not give - AR without ARKit
  7. How we use AR (at TOPOLOGY EYEWEAR)

  8. We use AR: for shopping What will these look like

    on me? » Do these frames suit my face shape? » Does this material suit my complexion? » What does it look like from the side?
  9. We use AR: for proof of fit But will these

    finally fit? » Will they be too big? » Will they slide down my nose? leave red marks? » Will they squeeze my ears?
  10. We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  11. We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  12. None
  13. We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  14. We use AR: for manufacture How do we make these?

    » Temple length? » Splay angle? » Nosebridge contours?
  15. What ARKit does not give

  16. What ARKit Does not give: 1. EARS

  17. What ARKit Does not give: 2. All the iPhones

  18. What ARKit Does not give: 2. All the iPhones

  19. What ARKit Does not give: 3. Accurate Scale

  20. AR without ARKit

  21. AR without ARKit Step 1. Selfie capture (iOS) Step 2.

    Computer vision (Linux) Step 3. TryOn Graphics (iOS)
  22. AR without ARKit Step 1. Selfie capture (iOS) » CoreMotion

    for device orientation » AVFoundation for face detection, face pose detection, video capture, custom camera capture settings, video trimming, custom video metadata, camera intrinsics
  23. AR without ARKit Step 2. Computer vision (Linux) » Multi-step

    pipeline, built on: » many open source libraries; ~80+ kLOC of our own Python and C++ code; proprietary refinements for the eyewear fitting problem; patented IP » CNNs (buzz!), ERTs, dedicated feature detectors, structure from motion, morphable face models, PLS regression, etc.. » Thousands of samples of manually labeled training data
  24. AR without ARKit Step 3. TryOn Graphics » Trusty UIKit

    to composite an SCNView on top of a UImageView » Compute eyewear position, orientation, and occlusion based on data from server- side computer vision stack » Render glasses with custom Metal shaders in SceneKit
  25. AR without ARKit Step 3. TryOn Graphics » Render glasses

    with custom Metal shaders in SceneKit
  26. So ... is it worth it?

  27. ARKit vs "TopoKit" ARKit TopoKit realtime Yes out-of-the-box Yes all

    the iPhones Yes ears Yes eyewear-accurate Yes
  28. But ... is it necessary? A puzzling truth About TrueDepth

  29. None
  30. None
  31. None
  32. ARKit face-tracking is exclusive to iPhone X » Make sense:

    » only iPhone X has the "TrueDepth camera system" » so only iPhone X provides ARFaceAnchor » since the depth sensor enables face tracking...? » But then » if Face ID is so accurate, why is ARKit not?
  33. None
  34. Wait, ARKit doesn't use the depth sensor? No, ARKit doesn't

    use depth sensor!
  35. So ... is it worth it? yes, since we need

    accuracy
  36. None
  37. Pupillary Distance

  38. None
  39. None
  40. None
  41. PD error value (mm) ARKit RMS average ~3.5 ARKit max

    ~7 TopoKit RMS average ~1.5 Optics requirements 2 Also: - consistent error with some subjects - high error variance, per trial, with facial error
  42. Pupillary Distance Our solution: » Dedicated capture flow using back

    camera » Use card as reference object » Capture video to reduce error
  43. Machine Learning Startup-style

  44. None
  45. None
  46. None
  47. None
  48. None
  49. Conclusions: » ARKit gives you magic! » That magic has

    some (arbitrary?) limits » But, with work, you can go beyond it » What are your real requirements? » Keep it simple. Sandwiches.
  50. Augmented Reality Beyond ARKit @alexisgallagher alexis@topologyeyewear.com Interested? Please say hi!