ARKit 2.0

61a68b2172503ecdf7a7f7757df56071?s=47 Tachibana Kaoru
June 11, 2018
370

ARKit 2.0

61a68b2172503ecdf7a7f7757df56071?s=128

Tachibana Kaoru

June 11, 2018
Tweet

Transcript

  1. ARKit 2.0 @TachibanaKaoru 2018/6/11 WWDC GoriGori Catch-Up

  2. About Me @TachibanaKaoru Freelance iOS Engineer ϘϧμϦϯά޷͖ ࠓ೥ͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫ

  3. ARKit

  4. What is ARKit? iOS11~ Device with A9 chip~ SceneKit or

    SpriteKit or Metal Unity ARKit Plugin
  5. ARKit capable devices Phone X iPhone 8, iPhone 8 Plus

    iPhone 7, iPhone 7 Plus iPhone 6s, iPhone 6s Plus iPhone SE iPad Pro (all models) iPad (5th generation)
  6. ARKit 1.0 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌ 1.5 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌ 2.0 ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ Object SacnରԠɺࢹઢɾઉݕ஌

  7. AR Configuration Mode Camera iOS Device AROrientationTracking Back Camera iOS

    11.0+ ARKit Capable Devices ARWorldTracking Back Camera iOS 11.0+ ARKit Capable Devices ARFaceTracking Front Camera iOS 11.0+ Only iPhone X ARImageTracking Back Camera iOS 12.0+ ARKit Capable Devices ARObjectScanning Back Camera iOS 12.0+ ARKit Capable Devices
  8. USDZ

  9. USDZ USD : Universal Scene Description 3D Model Format from

    Pixar PDF in 3D world! From iOS12 ~
  10. USDZ on iOS12 Available on Safari, memo, Mail and so

    on.
  11. USDZ on iOS12 Check USDZ files on “AR Quick Look

    Gallery” https://developer.apple.com/arkit/gallery/
  12. USDZ on your App Available with QLPreviewController, WKWebView, SFSafariViewController

  13. USDZ with QuickLook import QuickLook class ViewController: UIViewController { override

    func viewDidLoad() { super.viewDidLoad() } // Call Here to show QuickLook Preview Controller!! @IBAction func showQuickLook(_ sender: Any) { let quickViewCon = QLPreviewController() quickViewCon.dataSource = self self.present(quickViewCon, animated: true, completion: nil) } }
  14. USDZ with QuickLook extension ViewController: QLPreviewControllerDataSource { func numberOfPreviewItems(in controller:

    QLPreviewController) -> Int{ return 1 } func previewController(_ controller: QLPreviewController, previewItemAt index: Int) -> QLPreviewItem{ return MyARItem() } } class MyARItem: NSObject, QLPreviewItem{ var previewItemURL: URL? { return Bundle.main.url(forResource: "coffee", withExtension: "usdz") } }
  15. How does ARKit works?

  16. AR Configuration Mode Camera iOS Device AROrientationTracking Back Camera iOS

    11.0+ ARKit Capable Devices ARWorldTracking Back Camera iOS 11.0+ ARKit Capable Devices ARFaceTracking Front Camera iOS 11.0+ Only iPhone X ARImageTracking Back Camera iOS 12.0+ ARKit Capable Devices ARObjectScanning Back Camera iOS 12.0+ ARKit Capable Devices
  17. AR Configuration Mode Camera iOS Device AROrientationTracking Back Camera iOS

    11.0+ ARKit Capable Devices ARWorldTracking Back Camera iOS 11.0+ ARKit Capable Devices ARFaceTracking Front Camera iOS 11.0+ Only iPhone X ARImageTracking Back Camera iOS 12.0+ ARKit Capable Devices ARObjectScanning Back Camera iOS 12.0+ ARKit Capable Devices
  18. run(_ configuration) Basics of ARKit Recap ARConfiguration ARFrame ARFrame ARFrame

    CMMotionManager AVCaptureSession currentFrame ARSession
  19. Behind the Scenes—Inertial Odometry Motion sensor Time Image ARFrame Motion

    + Motion +
  20. Behind the Scenes—Inertial Odometry Motion sensor Image ARFrame Motion +

    Motion + Time Image ARFrame Motion + Image ARFrame Motion + Image ARFrame Motion + Motion + Image ARFrame Motion + Motion +
  21. Orientation Tracking API AROrientationTrackingConfiguration open class ARCamera : NSObject, NSCopying

    { open var transform: simd_float4x4 { get } open var eulerAngles: simd_float3 { get } … } AROrientationTrackingConfiguration
  22. Behind the Scenes—Visual Inertial Odometry Motion data and computer vision

    Image ARFrame Motion + Image ARFrame Motion + Image ARFrame Motion + Image ARFrame Motion CV Image ARFrame Motion CV Time 1 m
  23. Behind the Scenes—Visual Inertial Odometry Extracting robust features

  24. Behind the Scenes—Visual Inertial Odometry Triangulation creates World Map

  25. Behind the Scenes—Visual Inertial Odometry Initialization C W

  26. AROrientationTracking ARWorldTracking

  27. Share AR!

  28. What is “share AR?” “share AR” means share World tracking

    information and 3D models Use ARWorldMap class
  29. ARWorldMap ARWorlMap͕อ࣋͢Δ৘ใ 3Dͷಛ௃఺ͳͲͷτϥοΩϯά৘ ใͱɺ഑ஔ͞ΕͨΞϯΧʔ৘ใɻ ARWorldTrackingConfigurationͰͷΈ औಘ͢Δ͜ͱ͕Ͱ͖Δɻ Internal tracking data •

    Map of 3D feature points • Local appearance List of named anchors World Map Data A
  30. Share the World Map // Retrieve world map from ARSession

    session.getCurrentWorldMap { worldMap, error in guard let worldMap = worldMap else { showAlert(error) return } // Serialize let data = try NSKeyedArchiver.archivedData(withRootObject: worldMap, requiringSecureCoding: true) }
  31. Relocalize to World Map Setup configuration // Receive / Load

    world map let worldMap : ARWorldMap = ... // Create a session configuration let configuration = ARWorldTrackingConfiguration() configuration.initialWorldMap = worldMap // Run the session session.run(configuration) W
  32. SwiftShot ARWorldMapΛMultipeerConnectivity্Ͱڞ༗͢ΔήʔϜ SwiftShot https://developer.apple.com/documentation/arkit/ swiftshot_creating_a_game_for_augmented_reality Inside SwiftShot: Creating an AR

    Game https://developer.apple.com/videos/play/wwdc2018/605
  33. 3D Object Scan

  34. 3D Object Scan ARKit Ͱ 3D ObjectΛScanͯ͠ɺ ARReferenceObjectΛ࡞Δ ARReferenceObjectΛ࢖ͬͯɺͦͷ෺ମ͕ Ͳ͜ʹଘࡏ͢Δ͔ݕ஌Ͱ͖Δ

  35. 3D Object Scan ARKitͰscanͯ͠3DϞσϧ͕࡞੒Ͱ͖ Δʁ NO! ARKitͷ scanning͸ɺ෺ମ͔Βಛ௃఺ͷ ू߹Λอଘ͢Δ͚ͩɻ ͦͷ෺ମΛݕ஌͢Δ͜ͱ͸Ͱ͖ͯ΋ɺ

    ෺ମͷ3DϞσϧ͸࡞Εͳ͍ɻ How to Acquire an Object? Similar representation as a world map
  36. How to get ARReferenceObject? AppleͷαϯϓϧίʔυɺScanning and Detecting 3D ObjectsΛ࢖͏ https://developer.apple.com/documentation/arkit/

    scanning_and_detecting_3d_objects
  37. How to use ARReferenceObject? Scanning and Detecting 3D Objects αϯϓϧΛ࢖ͬͯɺ

    ARReferenceObjectΛεΩϟϯɻΤΫεϙʔτͯ͠ɺ*.arobjectΛऔಘɻ XcodeͰɺARͷ৽نϓϩδΣΫτΛ࡞੒͠ɺAsset CatalogʹAR Resource GroupΛ࡞Γɺ͜ͷϑΝΠϧΛDrag&Dropɻ
  38. Prepare for Scan override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated)

    let configuration = ARWorldTrackingConfiguration() configuration.detectionObjects = ARReferenceObject.referenceObjects(inGroupNamed: "apple", bundle: nil)! sceneView.session.run(configuration) } }
  39. Detection of Object func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode,

    for anchor: ARAnchor) { if let objectAnchor = anchor as? ARObjectAnchor{ let foundObject = objectAnchor.referenceObject // check name" let objectname = foundObject.name // do something to the object! objectAnchor.addTitleNode(on: node) } }
  40. To Scan Efficiently ໛༷͕͋Δ΋ͷ ςʔϒϧͷ্ʹͷΔ͘Β͍ͷେ͖͞ ॊΒ͔͍΋ͷͰ͸ͳ͘ɺݻ͍෺ମ ҆ఆͨࣨ͠಺র໌ Use iPhoneX !

  41. Personal Recommendation Nintendo Amiibo

  42. ARObjectScanningConfiguration ARObjectScanningConfiguration is for use only in development scenarios. High-fidelity

    spatial mapping has a high performance and energy cost, and disables ARKit features not necessary for reference object scanning. To produce AR experiences for end users, use ARWorldTrackingConfiguration. Appleͱͯ͠͸ɺObject ScanϞʔυ͸ɺ։ൃϑΣʔζͰ࢖͏΋ͷͱ૝ఆ͠ ͍ͯΔΑ͏Ͱ͢ɻ ։ൃऀ͕ಛఆͷΦϒδΣΫτΛScan͠ɺͦ͜Ͱநग़ͨ͠ ARReferenceObjectΛΞϓϦʹ͍ΕͯϦϦʔεɻ
  43. ·ͱΊ ARKit 2.0Ͱ͸͓΋͠Ζ͍ػೳ͕͍Ζ͍Ζ௥Ճ͞Ε͍ͯ·͢ɻ AR Share Object Scan USDZ ϑΝΠϧΛ࢖͑͹࣮૷͕΄ͱΜͲ͍Γ·ͤΜɻ Ͱ͖Δָ͚ͩΛ͠·͠ΐ͏ʂ