Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AR without SceneKit

2bfb16e53b154afc9f141f831b4aaaf2?s=47 naru-jpn
February 07, 2018

AR without SceneKit

Create AR application using Metal without SceneKit

2bfb16e53b154afc9f141f831b4aaaf2?s=128

naru-jpn

February 07, 2018
Tweet

Transcript

  1. AR without SceneKit #mobilearmeetup @Timers 2018/2/7

  2. Relation of ARKit and SceneKit

  3. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView

  4. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light

    Estimation Plane Detection Manage 3D Objects
  5. Remove SceneKit and Use Metal

  6. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light

    Estimation Plane Detection Manage 3D Objects
  7. ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection

  8. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering
  9. Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation

    Plane Detection Take Photo
  10. MetalAR https://github.com/naru-jpn/MetalAR

  11. None
  12. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo
  13. // Create ARSession session = ARSession() // ARSessionDelegate session.delegate =

    self
  14. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo
  15. // Defined in ARSessionDelegate // Called when ARFrame is updated

    func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
  16. A running AR session continuously captures video frames from the

    device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
  17. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo
  18. Pipeline of off-screen rendering Draw video frame Filter Draw Instances

    ARFrame MTLTexture
  19. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw

    video frame Commands sent to GPU
  20. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel

    declared in Metal Performance Shaders Draw video frame Commands sent to GPU
  21. Performance on iPhone7

  22. Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms

    Rendering
  23. Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~

    10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
  24. Challenges

  25. Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)

    Physics engine Skeletal Animation …
  26. References

  27. Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading

    Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
  28. AR Template Project