AR without SceneKit

Transcript

1. AR without SceneKit #mobilearmeetup @Timers 2018/2/7

2. Relation of ARKit and SceneKit

3. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView

4. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light

   Estimation Plane Detection Manage 3D Objects

5. Remove SceneKit and Use Metal

6. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light

   Estimation Plane Detection Manage 3D Objects

7. ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection

8. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

   Detection Rendering

9. Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation

   Plane Detection Take Photo

10. MetalAR https://github.com/naru-jpn/MetalAR

11. None

12. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo

13. // Create ARSession session = ARSession() // ARSessionDelegate session.delegate =

    self

14. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo

15. // Defined in ARSessionDelegate // Called when ARFrame is updated

    func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }

16. A running AR session continuously captures video frames from the

    device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe

17. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane

    Detection Rendering Take Photo

18. Pipeline of off-screen rendering Draw video frame Filter Draw Instances

    ARFrame MTLTexture

19. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw

    video frame Commands sent to GPU

20. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel

    declared in Metal Performance Shaders Draw video frame Commands sent to GPU

21. Performance on iPhone7

22. Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms

    Rendering

23. Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~

    10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread

24. Challenges

25. Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)

    Physics engine Skeletal Animation …

26. References

27. Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading

    Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf

28. AR Template Project