Depth in Depth #iOSDC

iOSDC Japan 2018での発表「Depth in Depth」の資料です。

## 概要

原始のiPhoneからカメラは搭載されていましたが、深度センサが搭載されたのは比較的最近のことです。カメラやGPSが、デジタルの世界と我々が生きる現実世界を繋ぐ重要な役割を担い、アプリ開発者に多くの創造性を与えてくれたのと同様に、「奥行き」がわかるようになったというのはアプリ開発の次元がひとつ増えたようなものです。本トークではiOSでの深度の扱いについて、取得方法だけではなく、細かいパラメータの意味やMetalでの処理まで詳解します。

## アジェンダ

- 深度の種類とセンサのしくみ
- 深度データを取得する
- 深度データを使ってみる
- できること・できないこと事例集

Transcript

1. Depth in Depth
   Shuichi Tsutsumi @shu223
   iOSDC 2018
   

   View Slide

2. ࣗݾ঺հ
   • Fyusion Inc.ࣾһ@αϯϑϥϯγείʢH-1BϏβʣ
   • Senior Principal Software Engineer
   • ࠓ೥͔ΒύʔτλΠϜܖ໿ʹ → ॊೈʹಇ͖ͭͭϏβ͸ҡ
   ࣋
   • ϑϦʔϥϯε@౦ژ
   • ݱࡏΞΫςΟϒʹ͓ख఻͍͍ͯ͠Δͷ͸8ࣾ΄ͲʢIoTܥɺ
   MetalɺARKitɺNFCɺυϩʔϯɺٕज़ސ໰ɺւ֎ɺetc...ʣ
   

   View Slide

3. • GitHub: shu223
   • ϒϩά: ͸ͯͳɺQiitaɺNote
   • ॻ੶:
   

   View Slide

4. Depth in Depth
   σϓεʢਂ౓ʣৄղ
   

   View Slide

5. ਂ౓ͱ͸ʁ
   

   View Slide

6. Ԟߦ͖Λࣔ͢৘ใ
   1νϟϯωϧͷը૾σʔλʢਂ౓Ϛοϓͱݺ͹ΕΔʣͰද͞ΕΔ
   ʢࠨɿRGBը૾ɹʗɹӈɿਂ౓Ϛοϓʣ
   

   View Slide

7. ʮ͋ʔ͸͍͸͍ɻࣸਅΛϘέͤ͞Δ΍ͭͰ͠ΐʯ
   

   View Slide

8. ✋
   

   View Slide

9. ਂ౓ͷ༻్: എܠΛඈ͹͢
   • ྫɿϑϦϚܥΞϓϦͰग़඼࣌ʹഎܠΛҰൃͰඈ͹ͤΔػೳ
   

   View Slide

10. ʢσϞʣ
    

    View Slide

11. ਂ౓ͷ༻్: 3DεΩϟϯ
    • 3DϞσϧΛ࡞੒
    • ෦԰ͷݟऔΓਤΛ࡞੒
    

    View Slide

12. ਂ౓ͷ༻్: ͦͷଞ
    ԕִࡱӨ࣌ͷ
    Ի੠ʹΑΔΠϯετϥΫγϣϯ
    ʮ30ηϯνޙΖʹԼ͕͍ͬͯͩ͘͞ʯ
    ※࣮ࡍʹਂ౓͸࢖ͬͯͳ͍͔΋͠Ε·ͤΜ͕ɺਂ౓ͷ༻్ͷՄೳੑ
    ͱͯ͠ྫ͍ࣔͯ͠·͢
    

    View Slide

13. ਂ౓͸ൃ૝࣍ୈͰ
    ৭ʑͳ࢖͍ํ͕Ͱ͖Δʂ
    

    View Slide

14. Ͳͷ͋ͨΓΛʮৄղʯ͢Δͷ͔ʁ
    • ʮiOSͷAPIͷ࢖͍ํʯ͸࠷খݶʹ
    • εϥΠυʹ͸ࡌ͓͖ͤͯ·͢
    • ຊ࣭తɾ൚༻తͳʮਂ౓σʔλͷऔΓѻ͍ํ๏ʯΛཧղ͢Δ
    

    View Slide

15. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू
    

    View Slide

16. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू
    

    View Slide

17. Disparityʢࢹࠩʣ
    iPhone 7/8 Plus, Xͷഎ໘Χϝϥ
    

    View Slide

18. ϐϯϗʔϧΧϝϥ
    

    View Slide

19. ̎୆ͷಉ͡Focal LengthΛ࣋ͭΧϝϥ
    

    View Slide

20. Disparityʢࢹࠩʣ
    

    View Slide

21. DepthͱDisparityͷؔ܎
    

    View Slide

22. Depth͸Disparity͔ΒܭࢉͰ͖Δ
    • Depth(z) ͷٯ਺ = ਖ਼نԽ͞Εͨ Disparity(d)
    

    View Slide

23. Time of Flight (ToF)
    • DepthΛ௚઀ଌΔํࣜ
    • ର৅ʹরࣹͨ͠ޫ͕൓ࣹͯ͠໭ͬͯ͘Δ·Ͱͷԟ෮ͷ࣌ؒ
    ͔Βڑ཭Λܭଌ
    

    View Slide

24. TrueDepth
    iPhone Xͷલ໘Χϝϥ
    

    View Slide

25. • Proximity sensorʢۙ઀ηϯαʣɾɾɾToFํࣜͰإ·Ͱͷڑ
    ཭Λܭଌ
    

    View Slide

26. • Dot projector (υοτϓϩδΣΫλ)ɾɾɾ3ສҎ্ͷݟ͑ͳ͍
    υοτΛإͷ্ʹরࣹ
    

    View Slide

27. • Infrated camera (੺֎ઢΧϝϥ)ɾɾɾυοτͷύλʔϯΛಡ
    ΈऔΔ → إͷϚοϓը૾
    

    View Slide

28. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू
    

    View Slide

29. AVDepthData
    • ਂ౓Λද͢Ϋϥε
    • Ͳͷํ๏Ͱऔಘͯ͠΋࠷ऴతʹ͸͜Ε͕ಘΒΕΔ
    • @available(iOS 11.0, *)
    

    View Slide

30. AVDepthData
    var depthDataMap: CVPixelBuffer { get }
    • ਂ౓ϚοϓͷϐΫηϧσʔλ
    • ैདྷ͔Β͋ΔCVPixelBufferͳͷͰɺCore ImageͰ΋
    MetalͰ΋ɺͦͷޙͷॲཧํ๏͸ࣗࡏ
    

    View Slide

31. iOSͰਂ౓σʔλΛऔಘ͢Δํ๏
    ༻్ผʹ෼͚Δͱ̏छྨ
    • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • ํ๏3: ARKit͔Βऔಘ
    

    View Slide

32. ਂ౓औಘํ๏1:
    ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ImageIOΛར༻
    • ʢΧϝϥϩʔϧ͔ΒಡΈࠐΜͰΞηοτͷURLΛಛఆ͢Δͱ
    ͜Ζ·Ͱ͸Photosʣ
    

    View Slide

33. ΞηοτͷURL͔ΒCGImageSourceΛ࡞੒
    let source = CGImageSourceCreateWithURL(url as CFURL, nil)
    

    View Slide

34. CGImageSource͔Βਂ౓σʔλͷDictionaryΛऔಘ
    let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
    source, 0,
    kCGImageAuxiliaryDataTypeDisparity // ଞͷλΠϓ΋͋Γ
    ) as? [String : AnyObject]
    // infoʹ͸ਂ౓Ϛοϓͷ CFDataRef ΍
    // ϝλσʔλͷCGImageMetadataRef͕ೖ͍ͬͯΔ
    

    View Slide

35. औಘͨ͠Dictionary͔ΒAVDepthDataΛॳظԽ
    let depthData =
    try! AVDepthData(fromDictionaryRepresentation: info)
    

    View Slide

36. ਂ౓औಘํ๏2:
    Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • AVFoundationΛར༻
    • جຊతʹ͸ैདྷͷΧϝϥࡱӨͷ࡞๏௨Γ
    

    View Slide

37. ਂ౓͕औΕΔσόΠελΠϓͷ AVCaptureDevice Λ࢖͏
    .builtInDualCamera, .builtInTrueDepthCamera
    let device = AVCaptureDevice.DiscoverySession(
    deviceTypes: [.builtInDualCamera],
    mediaType: .video,
    position: .back).devices.first
    

    View Slide

38. ਂ౓͕औΕΔΩϟϓνϟʔϑΥʔϚοτΛࢦఆ͢Δ
    var supportedDepthDataFormats: [AVCaptureDevice.Format]
    • AVCaptureDevice.Format ͷϓϩύςΟ
    • ॴ๬ͷ΋ͷΛબͼɺAVCaptureDevice ͷ
    activeDepthDataFormat ʹηοτ͢Δ
    

    View Slide

39. ηογϣϯͷग़ྗʹ AVCaptureDepthDataOutput Λ௥Ճ
    let depthOutput = AVCaptureDepthDataOutput()
    captureSession.addOutput(depthOutput)
    depthOutput.setDelegate(
    self, callbackQueue: dataOutputQueue)
    • AVCaptureVideoDataOutputʢϏσΦग़ྗʣ΍
    AVCaptureAudioʙʢΦʔσΟΦग़ྗʣΛ௥Ճ͢Δͷͱಉ౳
    

    View Slide

40. AVCaptureDepthDataOutputDelegate ͷ didOutput ϝιου
    ͔Β AVDepthData ͕ಘΒΕΔ
    func depthDataOutput(
    _ output: AVCaptureDepthDataOutput,
    didOutput depthData: AVDepthData,
    timestamp: CMTime,
    connection: AVCaptureConnection)
    

    View Slide

41. ਂ౓औಘํ๏3:
    ARKit͔Βऔಘ
    ARFrameʹcapturedDepthDataͱ͍͏ϓϩύςΟ͕͋Δ
    var capturedDepthData: AVDepthData? { get }
    

    View Slide

42. ⚠
    ํ๏ʹ༏ྼ͸ͳ͍ɻ༻్ʹΑͬͯબͿɻ
    • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • ํ๏3: ARKit͔Βऔಘ
    

    View Slide

43. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू
    

    View Slide

44. ਂ౓σʔλΛ࢖ͬͯΈΔ
    ਂ౓σʔλͰഎܠ߹੒
    

    View Slide

45. View Slide

46. എܠ߹੒ͷख๏
    ϚεΫը૾Λج४ʹɺΦϦδφϧը૾ͱഎܠը૾ΛϒϨϯυ͢
    Δ
    • ΫϩϚΩʔ߹੒͸ɺϚεΫͷ୅ΘΓʹchromaʢ࠼౓ʣΛར
    ༻
    

    View Slide

47. CIBlendWithMask
    Parameters
    inputImage ೖྗը૾
    inputBackgroundImage എܠը૾
    inputMaskImage ϚεΫը૾
    • ೖྗը૾͕ἧ͍ͬͯ͑͞Ε͹͜ΕΛ࢖͏͚ͩʂ
    • CIFilter͸MetalύΠϓϥΠϯͰ΋ॲཧͰ͖Δ
    

    View Slide

48. ϚεΫͱͯ͠ਂ౓Λ༻͍Ε͹OK!?
    

    View Slide

49. ࣮૷
    • ϑϩϯτΧϝϥ͔ΒϦΞϧλΠϜʹਂ౓Λऔಘ
    • CIFilterͷCIBlendWithMaskͰϒϨϯυ
    • MetalͰϨϯμϦϯάɺMTKViewʹඳը
    

    View Slide

50. ݁ՌʢσϞʣ
    

    View Slide

51. View Slide

52. Կ͕ى͖͍ͯΔͷ͔ʁ
    

    View Slide

53. ਂ౓ΛՄࢹԽͯ͠ΈΔ
    

    View Slide

54. ໰୊఺ͦͷ̍ɿ
    നͱࠇ͕൓ର
    • ϚεΫ͸ɺΦϦδφϧ෦෼͕നͰ͋
    Δ΂͖
    

    View Slide

55. ըૉ஋Λ൓సʢന˱ࠇʣͤ͞Ε͹Α͍ʁ
    

    View Slide

56. ໰୊఺ͦͷ̎ɿ
    إ͕άϨʔͷ··
    • ׬શʹനͰͳ͍ͱɺഎܠ͕ϒϨϯυ
    ͞Εͯ͠·͏
    

    View Slide

57. ରࡦɿ
    إͷਂ౓Λ͖͍͠஋ʹɺਂ౓
    ϚοϓΛ2஋Խ͢Δ
    • AVCaptureMetadataOutputͰإΛೝ
    ࣝ
    • إͷத৺ͷਂ౓+offsetΛ͖͍͠஋
    ʢcutOffʣͱ͢Δ
    • ਂ౓Ϛοϓʹ͓͍ͯɺ
    • ਂ౓͕cutOffҎ্Ͱ͋Ε͹ࠇ
    • ਂ౓͕cutOffҎԼͰ͋Ε͹ന
    

    View Slide

58. ݁ՌʢσϞʣ
    

    View Slide

59. View Slide

60. ͦͷଞͷ໰୊఺
    • ڥք͕ΪβΪβ
    • ϝΨω͕എܠʹɾɾɾ"Holes"
    

    View Slide

61. ରࡦɿ
    • ΪβΪβɾɾɾεϜʔδϯά͢Δ
    • HolesɾɾɾRGBΛར༻ͯ͠ຒΊΔ
    

    View Slide

62. isFilteringEnabled
    • AVCaptureDepthDataOutputͷϓϩύςΟ
    • ۭؒํ޲͚ͩͰͳ͘ɺ࣌ܥྻํ޲ʹ΋εϜʔδϯά
    • "Holes"΋ຒΊͯ͘ΕΔʢRGBΛར༻ʣ
    

    View Slide

63. View Slide

64. ݁ՌʢσϞʣ
    

    View Slide

65. ׬੒ʂ
    

    View Slide

66. ඍົʁ
    • ൅ͷໟͷ·ΘΓ
    • إͷਂ౓Λج४ʹ͖͍͠஋Λઃఆ͠
    ͍ͯΔɾɾɾ൚༻ੑʹ͚ܽΔ
    

    View Slide

67. Portrait Matte
    

    View Slide

68. Portrait Matteͱ͸ʁ
    • എܠͱલܠͷ෼཭ʢηάϝϯςʔγϣϯʣʹಛԽͨ͠ϑΥʔ
    Ϛοτ
    • iOS 12͔ΒऔಘՄೳʹͳͬͨ
    

    View Slide

69. ैདྷͷਂ౓ϚοϓʢࠨʣͱPortrait Matteʢӈʣ
    

    View Slide

70. ൅ͷໟ΋͖Ε͍ʹ෼཭
    

    View Slide

71. औಘํ๏
    • લड़ͷCGImageSourceΛར༻͢Δํ๏ͱ΄΅ಉ͡
    • λΠϓͱͯ͠ ʙPortraitEffectsMatte Λࢦఆ͢Δ
    • AVDapthData ͷ୅ΘΓʹ AVPortraitEffectsMatte
    

    View Slide

72. guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
    self, 0,
    kCGImageAuxiliaryDataTypePortraitEffectsMatte
    ) as? [String : AnyObject] else { return }
    let matte =
    AVPortraitEffectsMatte(fromDictionaryRepresentation: info)
    

    View Slide

73. View Slide

74. Portrait Matteͷ੍໿
    • ੩ࢭըͷΈ
    • ਓ͕ө͍ͬͯΔ৔߹ͷΈ
    • iOS 12Ҏ্
    

    View Slide

75. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू
    

    View Slide

76. Ͳ͏͍͏έʔεͰɺԿ͕Ͱ͖ͯԿ͕Ͱ͖ͳ͍ͷ͔
    ೺Ѳͮ͠Β͍
    • ࡱӨࡁΈࣸਅ͔Β / ϦΞϧλΠϜ / ARKit
    • Dual / TrueDepth / ͦΕҎ֎
    • ϑϩϯτ / ϦΞ
    • iOS 12 / 11 / ͦΕҎԼ
    • ର৅͕إ / શ਎ / ෺ମ
    

    View Slide

77. ۩ମతͳࣄྫͰઆ໌ͯ͠Έ·͢
    

    View Slide

78. ʮ͜ΕͰ΋͏੾Γൈ͖࡞ۀ͸Photoshop͍Βͣ
    ͩʂʯ
    • ର৅ը૾ʹਂ౓σʔλ͕ೖ͍ͬͯΔඞཁ͕͋Δ
    • Portrait Matteͷ඼࣭Λظ଴͢ΔͳΒɺਓ͕ؒࣸͬͯͳ͍ͱ͍
    ͚ͳ͍
    

    View Slide

79. ʮ෦԰͕ࢄΒ͔ͬͯͯ΋ϏσΦձٞͰ͖Δʂʯ
    • Portrait Matte͸ϦΞϧλΠϜͰ͸ར༻Ͱ͖ͳ͍
    • Ͱ΋ϑϩϯτΧϝϥͰਓؒͷإΛத৺ʹࡱӨ͢ΔͷͰɺී௨
    ͷਂ౓ϕʔεͰ΋ͦΕͳΓʹ͍͚ͦ͏
    

    View Slide

80. ʮϑϦϚΞϓϦͰग़඼࣌ʹࣗಈͰഎܠΛඈ͹ͤΔ
    ػೳΛೖΕΑ͏ʯ
    • ਓҎ֎ͷʮϞϊʯ͸Portrait Matte͸औΕͳ͍
    • Disprityϕʔεͷਂ౓͸ͱΕΔ
    • ͦͷʮϞϊʯͷਂ౓ͷ͖͍͠஋͸Ͳ͏΍ܾͬͯΊΔʁ
    • ϑΝογϣϯ౳ʹݶఆ͢Ε͹ྑͦ͞͏
    

    View Slide

81. ʮARKitͷΦΫϧʔδϣϯ͕͜ΕͰͰ
    ͖Δʂʯ
    • ϑϩϯτɾɾɾOK
    • ϦΞɾɾɾNG
    • ARFaceTrackingConfiguration࢖༻࣌ͷΈ༗ޮ
    • എ໘ΧϝϥΛ࢖͏ίϯϑΟΪϡϨʔγϣϯʢྫɿ
    ARWorldTrackingʙʣͰ͸ capturedDepth ͕ nil
    

    View Slide

82. ·ͱΊ
    • ਂ౓͸ʢࣸਅͷϘέҎ֎ʹ΋ʣൃ૝࣍ୈͰ͍Ζ͍Ζͱ࢖͑Δ
    • ਂ౓͸Ͳ͏΍ͬͯܭଌ͢Δͷ͔ʁ
    • ਂ౓σʔλɺͲ͏΍ͬͯ࢖͏ͷ͔ʁ
    • Կ͕Ͱ͖ͯɺԿ͕Ͱ͖ͳ͍ͷ͔ʁ
    

    View Slide

83. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    • Twitter: https://twitter.com/shu223
    • GitHub: https://github.com/shu223
    • ϒϩά: http://shu223.hatenablog.com/
    

    View Slide

84. ิ଍༻ࢿྉ
    

    View Slide

85. AVCaptureDataOutputSynchronizer
    

    View Slide

86. // ϏσΦग़ྗͱਂ౓ग़ྗΛಉظ
    synchronizer = AVCaptureDataOutputSynchronizer(
    dataOutputs: [videoOutput, depthOutput])
    

    View Slide

87. ಉظͨ͠ग़ྗΛड͚औΔ
    AVCaptureDataOutputSynchronizerDelegate
    didOutputσϦήʔτϝιου͔Β
    AVCaptureSynchronizedDataCollectionΦϒδΣΫτ͕ಘΒΕ
    Δ
    • → AVCaptureSynchronizedDepthData
    • → AVDepthData
    

    View Slide

88. // ਂ౓σʔλ͕AVCaptureSynchronizedDepthDataܕͰऔΓग़ͤΔ
    if let syncedDepthData =
    synchronizedDataCollection.synchronizedData(for: depthOutput)
    as? AVCaptureSynchronizedDepthData,
    !syncedDepthData.depthDataWasDropped
    {
    // AVDepthDataΦϒδΣΫτΛऔಘ
    let depthData = syncedDepthData.depthData
    ...
    }
    

    View Slide

89. ਂ౓σʔλΛ࢖ͬͯΈΔ
    MetalͰࣸਅΛ3DϚοϐ
    ϯά
    

    View Slide

90. Processed by Focos
    

    View Slide

91. • ௖఺γΣʔμ
    • ਂ౓σʔλΛz࠲ඪʹ
    • ϑϥάϝϯτγΣʔμ
    • RGBΧϥʔΛద༻
    

    View Slide