$30 off During Our Annual Pro Sale. View Details »

Depth in Depth #iOSDC

shu223
September 01, 2018

Depth in Depth #iOSDC

iOSDC Japan 2018での発表「Depth in Depth」の資料です。

## 概要

原始のiPhoneからカメラは搭載されていましたが、深度センサが搭載されたのは比較的最近のことです。カメラやGPSが、デジタルの世界と我々が生きる現実世界を繋ぐ重要な役割を担い、アプリ開発者に多くの創造性を与えてくれたのと同様に、「奥行き」がわかるようになったというのはアプリ開発の次元がひとつ増えたようなものです。本トークではiOSでの深度の扱いについて、取得方法だけではなく、細かいパラメータの意味やMetalでの処理まで詳解します。

## アジェンダ

- 深度の種類とセンサのしくみ
- 深度データを取得する
- 深度データを使ってみる
- できること・できないこと事例集

shu223

September 01, 2018
Tweet

More Decks by shu223

Other Decks in Programming

Transcript

  1. Depth in Depth
    Shuichi Tsutsumi @shu223
    iOSDC 2018

    View Slide

  2. ࣗݾ঺հ
    • Fyusion Inc.ࣾһ@αϯϑϥϯγείʢH-1BϏβʣ
    • Senior Principal Software Engineer
    • ࠓ೥͔ΒύʔτλΠϜܖ໿ʹ → ॊೈʹಇ͖ͭͭϏβ͸ҡ

    • ϑϦʔϥϯε@౦ژ
    • ݱࡏΞΫςΟϒʹ͓ख఻͍͍ͯ͠Δͷ͸8ࣾ΄ͲʢIoTܥɺ
    MetalɺARKitɺNFCɺυϩʔϯɺٕज़ސ໰ɺւ֎ɺetc...ʣ

    View Slide

  3. • GitHub: shu223
    • ϒϩά: ͸ͯͳɺQiitaɺNote
    • ॻ੶:

    View Slide

  4. Depth in Depth
    σϓεʢਂ౓ʣৄղ

    View Slide

  5. ਂ౓ͱ͸ʁ

    View Slide

  6. Ԟߦ͖Λࣔ͢৘ใ
    1νϟϯωϧͷը૾σʔλʢਂ౓Ϛοϓͱݺ͹ΕΔʣͰද͞ΕΔ
    ʢࠨɿRGBը૾ɹʗɹӈɿਂ౓Ϛοϓʣ

    View Slide

  7. ʮ͋ʔ͸͍͸͍ɻࣸਅΛϘέͤ͞Δ΍ͭͰ͠ΐʯ

    View Slide


  8. View Slide

  9. ਂ౓ͷ༻్: എܠΛඈ͹͢
    • ྫɿϑϦϚܥΞϓϦͰग़඼࣌ʹഎܠΛҰൃͰඈ͹ͤΔػೳ

    View Slide

  10. ʢσϞʣ

    View Slide

  11. ਂ౓ͷ༻్: 3DεΩϟϯ
    • 3DϞσϧΛ࡞੒
    • ෦԰ͷݟऔΓਤΛ࡞੒

    View Slide

  12. ਂ౓ͷ༻్: ͦͷଞ
    ԕִࡱӨ࣌ͷ
    Ի੠ʹΑΔΠϯετϥΫγϣϯ
    ʮ30ηϯνޙΖʹԼ͕͍ͬͯͩ͘͞ʯ
    ※࣮ࡍʹਂ౓͸࢖ͬͯͳ͍͔΋͠Ε·ͤΜ͕ɺਂ౓ͷ༻్ͷՄೳੑ
    ͱͯ͠ྫ͍ࣔͯ͠·͢

    View Slide

  13. ਂ౓͸ൃ૝࣍ୈͰ
    ৭ʑͳ࢖͍ํ͕Ͱ͖Δʂ

    View Slide

  14. Ͳͷ͋ͨΓΛʮৄղʯ͢Δͷ͔ʁ
    • ʮiOSͷAPIͷ࢖͍ํʯ͸࠷খݶʹ
    • εϥΠυʹ͸ࡌ͓͖ͤͯ·͢
    • ຊ࣭తɾ൚༻తͳʮਂ౓σʔλͷऔΓѻ͍ํ๏ʯΛཧղ͢Δ

    View Slide

  15. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

    View Slide

  16. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

    View Slide

  17. Disparityʢࢹࠩʣ
    iPhone 7/8 Plus, Xͷഎ໘Χϝϥ

    View Slide

  18. ϐϯϗʔϧΧϝϥ

    View Slide

  19. ̎୆ͷಉ͡Focal LengthΛ࣋ͭΧϝϥ

    View Slide

  20. Disparityʢࢹࠩʣ

    View Slide

  21. DepthͱDisparityͷؔ܎

    View Slide

  22. Depth͸Disparity͔ΒܭࢉͰ͖Δ
    • Depth(z) ͷٯ਺ = ਖ਼نԽ͞Εͨ Disparity(d)

    View Slide

  23. Time of Flight (ToF)
    • DepthΛ௚઀ଌΔํࣜ
    • ର৅ʹরࣹͨ͠ޫ͕൓ࣹͯ͠໭ͬͯ͘Δ·Ͱͷԟ෮ͷ࣌ؒ
    ͔Βڑ཭Λܭଌ

    View Slide

  24. TrueDepth
    iPhone Xͷલ໘Χϝϥ

    View Slide

  25. • Proximity sensorʢۙ઀ηϯαʣɾɾɾToFํࣜͰإ·Ͱͷڑ
    ཭Λܭଌ

    View Slide

  26. • Dot projector (υοτϓϩδΣΫλ)ɾɾɾ3ສҎ্ͷݟ͑ͳ͍
    υοτΛإͷ্ʹরࣹ

    View Slide

  27. • Infrated camera (੺֎ઢΧϝϥ)ɾɾɾυοτͷύλʔϯΛಡ
    ΈऔΔ → إͷϚοϓը૾

    View Slide

  28. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

    View Slide

  29. AVDepthData
    • ਂ౓Λද͢Ϋϥε
    • Ͳͷํ๏Ͱऔಘͯ͠΋࠷ऴతʹ͸͜Ε͕ಘΒΕΔ
    • @available(iOS 11.0, *)

    View Slide

  30. AVDepthData
    var depthDataMap: CVPixelBuffer { get }
    • ਂ౓ϚοϓͷϐΫηϧσʔλ
    • ैདྷ͔Β͋ΔCVPixelBufferͳͷͰɺCore ImageͰ΋
    MetalͰ΋ɺͦͷޙͷॲཧํ๏͸ࣗࡏ

    View Slide

  31. iOSͰਂ౓σʔλΛऔಘ͢Δํ๏
    ༻్ผʹ෼͚Δͱ̏छྨ
    • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • ํ๏3: ARKit͔Βऔಘ

    View Slide

  32. ਂ౓औಘํ๏1:
    ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ImageIOΛར༻
    • ʢΧϝϥϩʔϧ͔ΒಡΈࠐΜͰΞηοτͷURLΛಛఆ͢Δͱ
    ͜Ζ·Ͱ͸Photosʣ

    View Slide

  33. ΞηοτͷURL͔ΒCGImageSourceΛ࡞੒
    let source = CGImageSourceCreateWithURL(url as CFURL, nil)

    View Slide

  34. CGImageSource͔Βਂ౓σʔλͷDictionaryΛऔಘ
    let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
    source, 0,
    kCGImageAuxiliaryDataTypeDisparity // ଞͷλΠϓ΋͋Γ
    ) as? [String : AnyObject]
    // infoʹ͸ਂ౓Ϛοϓͷ CFDataRef ΍
    // ϝλσʔλͷCGImageMetadataRef͕ೖ͍ͬͯΔ

    View Slide

  35. औಘͨ͠Dictionary͔ΒAVDepthDataΛॳظԽ
    let depthData =
    try! AVDepthData(fromDictionaryRepresentation: info)

    View Slide

  36. ਂ౓औಘํ๏2:
    Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • AVFoundationΛར༻
    • جຊతʹ͸ैདྷͷΧϝϥࡱӨͷ࡞๏௨Γ

    View Slide

  37. ਂ౓͕औΕΔσόΠελΠϓͷ AVCaptureDevice Λ࢖͏
    .builtInDualCamera, .builtInTrueDepthCamera
    let device = AVCaptureDevice.DiscoverySession(
    deviceTypes: [.builtInDualCamera],
    mediaType: .video,
    position: .back).devices.first

    View Slide

  38. ਂ౓͕औΕΔΩϟϓνϟʔϑΥʔϚοτΛࢦఆ͢Δ
    var supportedDepthDataFormats: [AVCaptureDevice.Format]
    • AVCaptureDevice.Format ͷϓϩύςΟ
    • ॴ๬ͷ΋ͷΛબͼɺAVCaptureDevice ͷ
    activeDepthDataFormat ʹηοτ͢Δ

    View Slide

  39. ηογϣϯͷग़ྗʹ AVCaptureDepthDataOutput Λ௥Ճ
    let depthOutput = AVCaptureDepthDataOutput()
    captureSession.addOutput(depthOutput)
    depthOutput.setDelegate(
    self, callbackQueue: dataOutputQueue)
    • AVCaptureVideoDataOutputʢϏσΦग़ྗʣ΍
    AVCaptureAudioʙʢΦʔσΟΦग़ྗʣΛ௥Ճ͢Δͷͱಉ౳

    View Slide

  40. AVCaptureDepthDataOutputDelegate ͷ didOutput ϝιου
    ͔Β AVDepthData ͕ಘΒΕΔ
    func depthDataOutput(
    _ output: AVCaptureDepthDataOutput,
    didOutput depthData: AVDepthData,
    timestamp: CMTime,
    connection: AVCaptureConnection)

    View Slide

  41. ਂ౓औಘํ๏3:
    ARKit͔Βऔಘ
    ARFrameʹcapturedDepthDataͱ͍͏ϓϩύςΟ͕͋Δ
    var capturedDepthData: AVDepthData? { get }

    View Slide


  42. ํ๏ʹ༏ྼ͸ͳ͍ɻ༻్ʹΑͬͯબͿɻ
    • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ
    • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ
    • ํ๏3: ARKit͔Βऔಘ

    View Slide

  43. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

    View Slide

  44. ਂ౓σʔλΛ࢖ͬͯΈΔ
    ਂ౓σʔλͰഎܠ߹੒

    View Slide

  45. View Slide

  46. എܠ߹੒ͷख๏
    ϚεΫը૾Λج४ʹɺΦϦδφϧը૾ͱഎܠը૾ΛϒϨϯυ͢
    Δ
    • ΫϩϚΩʔ߹੒͸ɺϚεΫͷ୅ΘΓʹchromaʢ࠼౓ʣΛར

    View Slide

  47. CIBlendWithMask
    Parameters
    inputImage ೖྗը૾
    inputBackgroundImage എܠը૾
    inputMaskImage ϚεΫը૾
    • ೖྗը૾͕ἧ͍ͬͯ͑͞Ε͹͜ΕΛ࢖͏͚ͩʂ
    • CIFilter͸MetalύΠϓϥΠϯͰ΋ॲཧͰ͖Δ

    View Slide

  48. ϚεΫͱͯ͠ਂ౓Λ༻͍Ε͹OK!?

    View Slide

  49. ࣮૷
    • ϑϩϯτΧϝϥ͔ΒϦΞϧλΠϜʹਂ౓Λऔಘ
    • CIFilterͷCIBlendWithMaskͰϒϨϯυ
    • MetalͰϨϯμϦϯάɺMTKViewʹඳը

    View Slide

  50. ݁ՌʢσϞʣ

    View Slide

  51. View Slide

  52. Կ͕ى͖͍ͯΔͷ͔ʁ

    View Slide

  53. ਂ౓ΛՄࢹԽͯ͠ΈΔ

    View Slide

  54. ໰୊఺ͦͷ̍ɿ
    നͱࠇ͕൓ର
    • ϚεΫ͸ɺΦϦδφϧ෦෼͕നͰ͋
    Δ΂͖

    View Slide

  55. ըૉ஋Λ൓సʢന˱ࠇʣͤ͞Ε͹Α͍ʁ

    View Slide

  56. ໰୊఺ͦͷ̎ɿ
    إ͕άϨʔͷ··
    • ׬શʹനͰͳ͍ͱɺഎܠ͕ϒϨϯυ
    ͞Εͯ͠·͏

    View Slide

  57. ରࡦɿ
    إͷਂ౓Λ͖͍͠஋ʹɺਂ౓
    ϚοϓΛ2஋Խ͢Δ
    • AVCaptureMetadataOutputͰإΛೝ

    • إͷத৺ͷਂ౓+offsetΛ͖͍͠஋
    ʢcutOffʣͱ͢Δ
    • ਂ౓Ϛοϓʹ͓͍ͯɺ
    • ਂ౓͕cutOffҎ্Ͱ͋Ε͹ࠇ
    • ਂ౓͕cutOffҎԼͰ͋Ε͹ന

    View Slide

  58. ݁ՌʢσϞʣ

    View Slide

  59. View Slide

  60. ͦͷଞͷ໰୊఺
    • ڥք͕ΪβΪβ
    • ϝΨω͕എܠʹɾɾɾ"Holes"

    View Slide

  61. ରࡦɿ
    • ΪβΪβɾɾɾεϜʔδϯά͢Δ
    • HolesɾɾɾRGBΛར༻ͯ͠ຒΊΔ

    View Slide

  62. isFilteringEnabled
    • AVCaptureDepthDataOutputͷϓϩύςΟ
    • ۭؒํ޲͚ͩͰͳ͘ɺ࣌ܥྻํ޲ʹ΋εϜʔδϯά
    • "Holes"΋ຒΊͯ͘ΕΔʢRGBΛར༻ʣ

    View Slide

  63. View Slide

  64. ݁ՌʢσϞʣ

    View Slide

  65. ׬੒ʂ

    View Slide

  66. ඍົʁ
    • ൅ͷໟͷ·ΘΓ
    • إͷਂ౓Λج४ʹ͖͍͠஋Λઃఆ͠
    ͍ͯΔɾɾɾ൚༻ੑʹ͚ܽΔ

    View Slide

  67. Portrait Matte

    View Slide

  68. Portrait Matteͱ͸ʁ
    • എܠͱલܠͷ෼཭ʢηάϝϯςʔγϣϯʣʹಛԽͨ͠ϑΥʔ
    Ϛοτ
    • iOS 12͔ΒऔಘՄೳʹͳͬͨ

    View Slide

  69. ैདྷͷਂ౓ϚοϓʢࠨʣͱPortrait Matteʢӈʣ

    View Slide

  70. ൅ͷໟ΋͖Ε͍ʹ෼཭

    View Slide

  71. औಘํ๏
    • લड़ͷCGImageSourceΛར༻͢Δํ๏ͱ΄΅ಉ͡
    • λΠϓͱͯ͠ ʙPortraitEffectsMatte Λࢦఆ͢Δ
    • AVDapthData ͷ୅ΘΓʹ AVPortraitEffectsMatte

    View Slide

  72. guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
    self, 0,
    kCGImageAuxiliaryDataTypePortraitEffectsMatte
    ) as? [String : AnyObject] else { return }
    let matte =
    AVPortraitEffectsMatte(fromDictionaryRepresentation: info)

    View Slide

  73. View Slide

  74. Portrait Matteͷ੍໿
    • ੩ࢭըͷΈ
    • ਓ͕ө͍ͬͯΔ৔߹ͷΈ
    • iOS 12Ҏ্

    View Slide

  75. ΞδΣϯμ
    • ਂ౓ͷछྨͱηϯαͷ͘͠Έ
    • ਂ౓σʔλΛऔಘ͢Δ
    • ਂ౓σʔλΛ࢖ͬͯΈΔ
    • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

    View Slide

  76. Ͳ͏͍͏έʔεͰɺԿ͕Ͱ͖ͯԿ͕Ͱ͖ͳ͍ͷ͔
    ೺Ѳͮ͠Β͍
    • ࡱӨࡁΈࣸਅ͔Β / ϦΞϧλΠϜ / ARKit
    • Dual / TrueDepth / ͦΕҎ֎
    • ϑϩϯτ / ϦΞ
    • iOS 12 / 11 / ͦΕҎԼ
    • ର৅͕إ / શ਎ / ෺ମ

    View Slide

  77. ۩ମతͳࣄྫͰઆ໌ͯ͠Έ·͢

    View Slide

  78. ʮ͜ΕͰ΋͏੾Γൈ͖࡞ۀ͸Photoshop͍Βͣ
    ͩʂʯ
    • ର৅ը૾ʹਂ౓σʔλ͕ೖ͍ͬͯΔඞཁ͕͋Δ
    • Portrait Matteͷ඼࣭Λظ଴͢ΔͳΒɺਓ͕ؒࣸͬͯͳ͍ͱ͍
    ͚ͳ͍

    View Slide

  79. ʮ෦԰͕ࢄΒ͔ͬͯͯ΋ϏσΦձٞͰ͖Δʂʯ
    • Portrait Matte͸ϦΞϧλΠϜͰ͸ར༻Ͱ͖ͳ͍
    • Ͱ΋ϑϩϯτΧϝϥͰਓؒͷإΛத৺ʹࡱӨ͢ΔͷͰɺී௨
    ͷਂ౓ϕʔεͰ΋ͦΕͳΓʹ͍͚ͦ͏

    View Slide

  80. ʮϑϦϚΞϓϦͰग़඼࣌ʹࣗಈͰഎܠΛඈ͹ͤΔ
    ػೳΛೖΕΑ͏ʯ
    • ਓҎ֎ͷʮϞϊʯ͸Portrait Matte͸औΕͳ͍
    • Disprityϕʔεͷਂ౓͸ͱΕΔ
    • ͦͷʮϞϊʯͷਂ౓ͷ͖͍͠஋͸Ͳ͏΍ܾͬͯΊΔʁ
    • ϑΝογϣϯ౳ʹݶఆ͢Ε͹ྑͦ͞͏

    View Slide

  81. ʮARKitͷΦΫϧʔδϣϯ͕͜ΕͰͰ
    ͖Δʂʯ
    • ϑϩϯτɾɾɾOK
    • ϦΞɾɾɾNG
    • ARFaceTrackingConfiguration࢖༻࣌ͷΈ༗ޮ
    • എ໘ΧϝϥΛ࢖͏ίϯϑΟΪϡϨʔγϣϯʢྫɿ
    ARWorldTrackingʙʣͰ͸ capturedDepth ͕ nil

    View Slide

  82. ·ͱΊ
    • ਂ౓͸ʢࣸਅͷϘέҎ֎ʹ΋ʣൃ૝࣍ୈͰ͍Ζ͍Ζͱ࢖͑Δ
    • ਂ౓͸Ͳ͏΍ͬͯܭଌ͢Δͷ͔ʁ
    • ਂ౓σʔλɺͲ͏΍ͬͯ࢖͏ͷ͔ʁ
    • Կ͕Ͱ͖ͯɺԿ͕Ͱ͖ͳ͍ͷ͔ʁ

    View Slide

  83. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    • Twitter: https://twitter.com/shu223
    • GitHub: https://github.com/shu223
    • ϒϩά: http://shu223.hatenablog.com/

    View Slide

  84. ิ଍༻ࢿྉ

    View Slide

  85. AVCaptureDataOutputSynchronizer

    View Slide

  86. // ϏσΦग़ྗͱਂ౓ग़ྗΛಉظ
    synchronizer = AVCaptureDataOutputSynchronizer(
    dataOutputs: [videoOutput, depthOutput])

    View Slide

  87. ಉظͨ͠ग़ྗΛड͚औΔ
    AVCaptureDataOutputSynchronizerDelegate
    didOutputσϦήʔτϝιου͔Β
    AVCaptureSynchronizedDataCollectionΦϒδΣΫτ͕ಘΒΕ
    Δ
    • → AVCaptureSynchronizedDepthData
    • → AVDepthData

    View Slide

  88. // ਂ౓σʔλ͕AVCaptureSynchronizedDepthDataܕͰऔΓग़ͤΔ
    if let syncedDepthData =
    synchronizedDataCollection.synchronizedData(for: depthOutput)
    as? AVCaptureSynchronizedDepthData,
    !syncedDepthData.depthDataWasDropped
    {
    // AVDepthDataΦϒδΣΫτΛऔಘ
    let depthData = syncedDepthData.depthData
    ...
    }

    View Slide

  89. ਂ౓σʔλΛ࢖ͬͯΈΔ
    MetalͰࣸਅΛ3DϚοϐ
    ϯά

    View Slide

  90. Processed by Focos

    View Slide

  91. • ௖఺γΣʔμ
    • ਂ౓σʔλΛz࠲ඪʹ
    • ϑϥάϝϯτγΣʔμ
    • RGBΧϥʔΛద༻

    View Slide