ARKit 2.0

Transcript

1. ARKit 2.0
   @TachibanaKaoru
   2018/6/11 WWDC GoriGori Catch-Up
   

   View Slide

2. About Me
   @TachibanaKaoru
   Freelance iOS Engineer
   ϘϧμϦϯά޷͖
   ࠓ೥ͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ
   όϦౡɺαϯτϦʔχౡɺόϯίΫ
   

   View Slide

3. ARKit
   

   View Slide

4. What is ARKit?
   iOS11~
   Device with A9 chip~
   SceneKit or SpriteKit or Metal
   Unity ARKit Plugin
   

   View Slide

5. ARKit capable devices
   Phone X
   iPhone 8, iPhone 8 Plus
   iPhone 7, iPhone 7 Plus
   iPhone 6s, iPhone 6s Plus
   iPhone SE
   iPad Pro (all models)
   iPad (5th generation)
   

   View Slide

6. ARKit
   1.0 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌
   1.5 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌
   2.0
   ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ
   Object SacnରԠɺࢹઢɾઉݕ஌
   

   View Slide

7. AR Configuration Mode
   Camera iOS Device
   AROrientationTracking Back Camera iOS 11.0+
   ARKit Capable
   Devices
   ARWorldTracking Back Camera iOS 11.0+
   ARKit Capable
   Devices
   ARFaceTracking Front Camera iOS 11.0+ Only iPhone X
   ARImageTracking Back Camera iOS 12.0+
   ARKit Capable
   Devices
   ARObjectScanning Back Camera iOS 12.0+
   ARKit Capable
   Devices
   

   View Slide

8. USDZ
   

   View Slide

9. USDZ
   USD : Universal Scene Description
   3D Model Format from Pixar
   PDF in 3D world!
   From iOS12 ~
   

   View Slide

10. USDZ on iOS12
    Available on Safari, memo, Mail and so on.
    

    View Slide

11. USDZ on iOS12
    Check USDZ files on “AR Quick Look Gallery”
    https://developer.apple.com/arkit/gallery/
    

    View Slide

12. USDZ on your App
    Available with QLPreviewController, WKWebView, SFSafariViewController
    

    View Slide

13. USDZ with QuickLook
    import QuickLook
    class ViewController: UIViewController {
    override func viewDidLoad() {
    super.viewDidLoad()
    }
    // Call Here to show QuickLook Preview Controller!!
    @IBAction func showQuickLook(_ sender: Any) {
    let quickViewCon = QLPreviewController()
    quickViewCon.dataSource = self
    self.present(quickViewCon, animated: true, completion: nil)
    }
    }
    

    View Slide

14. USDZ with QuickLook
    extension ViewController: QLPreviewControllerDataSource {
    func numberOfPreviewItems(in controller: QLPreviewController) -> Int{
    return 1
    }
    func previewController(_ controller: QLPreviewController, previewItemAt index: Int) -> QLPreviewItem{
    return MyARItem()
    }
    }
    class MyARItem: NSObject, QLPreviewItem{
    var previewItemURL: URL? {
    return Bundle.main.url(forResource: "coffee", withExtension: "usdz")
    }
    }
    

    View Slide

15. How does ARKit works?
    

    View Slide

16. AR Configuration Mode
    Camera iOS Device
    AROrientationTracking Back Camera iOS 11.0+
    ARKit Capable
    Devices
    ARWorldTracking Back Camera iOS 11.0+
    ARKit Capable
    Devices
    ARFaceTracking Front Camera iOS 11.0+ Only iPhone X
    ARImageTracking Back Camera iOS 12.0+
    ARKit Capable
    Devices
    ARObjectScanning Back Camera iOS 12.0+
    ARKit Capable
    Devices
    

    View Slide

17. AR Configuration Mode
    Camera iOS Device
    AROrientationTracking Back Camera iOS 11.0+
    ARKit Capable
    Devices
    ARWorldTracking Back Camera iOS 11.0+
    ARKit Capable
    Devices
    ARFaceTracking Front Camera iOS 11.0+ Only iPhone X
    ARImageTracking Back Camera iOS 12.0+
    ARKit Capable
    Devices
    ARObjectScanning Back Camera iOS 12.0+
    ARKit Capable
    Devices
    

    View Slide

18. run(_ configuration)
    Basics of ARKit
    Recap
    ARConfiguration
    ARFrame
    ARFrame
    ARFrame
    CMMotionManager
    AVCaptureSession
    currentFrame
    ARSession
    

    View Slide

19. Behind the Scenes—Inertial Odometry
    Motion sensor
    Time
    Image
    ARFrame
    Motion
    +
    Motion
    +
    

    View Slide

20. Behind the Scenes—Inertial Odometry
    Motion sensor
    Image
    ARFrame
    Motion
    +
    Motion
    + Time
    Image
    ARFrame
    Motion
    +
    Image
    ARFrame
    Motion
    +
    Image
    ARFrame
    Motion
    +
    Motion
    +
    Image
    ARFrame
    Motion
    +
    Motion
    +
    

    View Slide

21. Orientation Tracking API
    AROrientationTrackingConfiguration
    open class ARCamera : NSObject, NSCopying {
    open var transform: simd_float4x4 { get }
    open var eulerAngles: simd_float3 { get }
    …
    }
    AROrientationTrackingConfiguration
    

    View Slide

22. Behind the Scenes—Visual Inertial Odometry
    Motion data and computer vision
    Image
    ARFrame
    Motion
    +
    Image
    ARFrame
    Motion
    +
    Image
    ARFrame
    Motion
    +
    Image
    ARFrame
    Motion
    CV
    Image
    ARFrame
    Motion
    CV Time
    1 m
    

    View Slide

23. Behind the Scenes—Visual Inertial Odometry
    Extracting robust features
    

    View Slide

24. Behind the Scenes—Visual Inertial Odometry
    Triangulation creates World Map
    

    View Slide

25. Behind the Scenes—Visual Inertial Odometry
    Initialization
    C
    W
    

    View Slide

26. AROrientationTracking
    ARWorldTracking
    

    View Slide

27. Share AR!
    

    View Slide

28. What is “share AR?”
    “share AR” means share World tracking information and 3D models
    Use ARWorldMap class
    

    View Slide

29. ARWorldMap
    ARWorlMap͕อ࣋͢Δ৘ใ
    3Dͷಛ௃఺ͳͲͷτϥοΩϯά৘
    ใͱɺ഑ஔ͞ΕͨΞϯΧʔ৘ใɻ
    ARWorldTrackingConfigurationͰͷΈ
    औಘ͢Δ͜ͱ͕Ͱ͖Δɻ
    Internal tracking data
    • Map of 3D feature points
    • Local appearance
    List of named anchors
    World Map Data
    A
    

    View Slide

30. Share the World Map
    // Retrieve world map from ARSession
    session.getCurrentWorldMap { worldMap, error in
    guard let worldMap = worldMap else {
    showAlert(error)
    return
    }
    // Serialize
    let data = try NSKeyedArchiver.archivedData(withRootObject: worldMap,
    requiringSecureCoding: true)
    }
    

    View Slide

31. Relocalize to World Map
    Setup configuration
    // Receive / Load world map
    let worldMap : ARWorldMap = ...
    // Create a session configuration
    let configuration = ARWorldTrackingConfiguration()
    configuration.initialWorldMap = worldMap
    // Run the session
    session.run(configuration)
    W
    

    View Slide

32. SwiftShot
    ARWorldMapΛMultipeerConnectivity্Ͱڞ༗͢ΔήʔϜ
    SwiftShot
    https://developer.apple.com/documentation/arkit/
    swiftshot_creating_a_game_for_augmented_reality
    Inside SwiftShot: Creating an AR Game
    https://developer.apple.com/videos/play/wwdc2018/605
    

    View Slide

33. 3D Object Scan
    

    View Slide

34. 3D Object Scan
    ARKit Ͱ 3D ObjectΛScanͯ͠ɺ
    ARReferenceObjectΛ࡞Δ
    ARReferenceObjectΛ࢖ͬͯɺͦͷ෺ମ͕
    Ͳ͜ʹଘࡏ͢Δ͔ݕ஌Ͱ͖Δ
    

    View Slide

35. 3D Object Scan
    ARKitͰscanͯ͠3DϞσϧ͕࡞੒Ͱ͖
    Δʁ
    NO!
    ARKitͷ scanning͸ɺ෺ମ͔Βಛ௃఺ͷ
    ू߹Λอଘ͢Δ͚ͩɻ
    ͦͷ෺ମΛݕ஌͢Δ͜ͱ͸Ͱ͖ͯ΋ɺ
    ෺ମͷ3DϞσϧ͸࡞Εͳ͍ɻ
    How to Acquire an Object?
    Similar representation as a world map
    

    View Slide

36. How to get ARReferenceObject?
    AppleͷαϯϓϧίʔυɺScanning and Detecting 3D ObjectsΛ࢖͏
    https://developer.apple.com/documentation/arkit/
    scanning_and_detecting_3d_objects
    

    View Slide

37. How to use ARReferenceObject?
    Scanning and Detecting 3D Objects αϯϓϧΛ࢖ͬͯɺ
    ARReferenceObjectΛεΩϟϯɻΤΫεϙʔτͯ͠ɺ*.arobjectΛऔಘɻ
    XcodeͰɺARͷ৽نϓϩδΣΫτΛ࡞੒͠ɺAsset CatalogʹAR
    Resource GroupΛ࡞Γɺ͜ͷϑΝΠϧΛDrag&Dropɻ
    

    View Slide

38. Prepare for Scan
    override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    let configuration = ARWorldTrackingConfiguration()
    configuration.detectionObjects =
    ARReferenceObject.referenceObjects(inGroupNamed: "apple", bundle: nil)!
    sceneView.session.run(configuration)
    }
    }
    

    View Slide

39. Detection of Object
    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
    if let objectAnchor = anchor as? ARObjectAnchor{
    let foundObject = objectAnchor.referenceObject
    // check name"
    let objectname = foundObject.name
    // do something to the object!
    objectAnchor.addTitleNode(on: node)
    }
    }
    

    View Slide

40. To Scan Efficiently
    ໛༷͕͋Δ΋ͷ
    ςʔϒϧͷ্ʹͷΔ͘Β͍ͷେ͖͞
    ॊΒ͔͍΋ͷͰ͸ͳ͘ɺݻ͍෺ମ
    ҆ఆͨࣨ͠಺র໌
    Use iPhoneX !
    

    View Slide

41. Personal Recommendation
    Nintendo Amiibo
    

    View Slide

42. ARObjectScanningConfiguration
    ARObjectScanningConfiguration is for use only in development scenarios.
    High-fidelity spatial mapping has a high performance and energy cost, and
    disables ARKit features not necessary for reference object scanning. To
    produce AR experiences for end users, use ARWorldTrackingConfiguration.
    Appleͱͯ͠͸ɺObject ScanϞʔυ͸ɺ։ൃϑΣʔζͰ࢖͏΋ͷͱ૝ఆ͠
    ͍ͯΔΑ͏Ͱ͢ɻ
    ։ൃऀ͕ಛఆͷΦϒδΣΫτΛScan͠ɺͦ͜Ͱநग़ͨ͠
    ARReferenceObjectΛΞϓϦʹ͍ΕͯϦϦʔεɻ
    

    View Slide

43. ·ͱΊ
    ARKit 2.0Ͱ͸͓΋͠Ζ͍ػೳ͕͍Ζ͍Ζ௥Ճ͞Ε͍ͯ·͢ɻ
    AR Share
    Object Scan
    USDZ ϑΝΠϧΛ࢖͑͹࣮૷͕΄ͱΜͲ͍Γ·ͤΜɻ
    Ͱ͖Δָ͚ͩΛ͠·͠ΐ͏ʂ
    

    View Slide