Depth in Depth #iOSDC - Speaker Deck

Depth in Depth #iOSDC

by shu223

Slide 1

Slide 1 text

Depth in Depth Shuichi Tsutsumi @shu223 iOSDC 2018

Slide 2

Slide 2 text

ࣗݾ঺հ • Fyusion Inc.ࣾһ@αϯϑϥϯγείʢH-1BϏβʣ • Senior Principal Software Engineer • ࠓ೥͔ΒύʔτλΠϜܖ໿ʹ → ॊೈʹಇ͖ͭͭϏβ͸ҡ ࣋ • ϑϦʔϥϯε@౦ژ • ݱࡏΞΫςΟϒʹ͓ख఻͍͍ͯ͠Δͷ͸8ࣾ΄ͲʢIoTܥɺ MetalɺARKitɺNFCɺυϩʔϯɺٕज़ސ໰ɺւ֎ɺetc...ʣ

Slide 3

Slide 3 text

• GitHub: shu223 • ϒϩά: ͸ͯͳɺQiitaɺNote • ॻ੶:

Slide 4

Slide 4 text

Depth in Depth σϓεʢਂ౓ʣৄղ

Slide 5

Slide 5 text

ਂ౓ͱ͸ʁ

Slide 6

Slide 6 text

Ԟߦ͖Λࣔ͢৘ใ 1νϟϯωϧͷը૾σʔλʢਂ౓Ϛοϓͱݺ͹ΕΔʣͰද͞ΕΔ ʢࠨɿRGBը૾ɹʗɹӈɿਂ౓Ϛοϓʣ

Slide 7

Slide 7 text

ʮ͋ʔ͸͍͸͍ɻࣸਅΛϘέͤ͞Δ΍ͭͰ͠ΐʯ

Slide 8

Slide 8 text

✋

Slide 9

Slide 9 text

ਂ౓ͷ༻్: എܠΛඈ͹͢ • ྫɿϑϦϚܥΞϓϦͰग़඼࣌ʹഎܠΛҰൃͰඈ͹ͤΔػೳ

Slide 10

Slide 10 text

ʢσϞʣ

Slide 11

Slide 11 text

ਂ౓ͷ༻్: 3DεΩϟϯ • 3DϞσϧΛ࡞੒ • ෦԰ͷݟऔΓਤΛ࡞੒

Slide 12

Slide 12 text

ਂ౓ͷ༻్: ͦͷଞ ԕִࡱӨ࣌ͷ Ի੠ʹΑΔΠϯετϥΫγϣϯ ʮ30ηϯνޙΖʹԼ͕͍ͬͯͩ͘͞ʯ ※࣮ࡍʹਂ౓͸࢖ͬͯͳ͍͔΋͠Ε·ͤΜ͕ɺਂ౓ͷ༻్ͷՄೳੑ ͱͯ͠ྫ͍ࣔͯ͠·͢

Slide 13

Slide 13 text

ਂ౓͸ൃ૝࣍ୈͰ ৭ʑͳ࢖͍ํ͕Ͱ͖Δʂ

Slide 14

Slide 14 text

Ͳͷ͋ͨΓΛʮৄղʯ͢Δͷ͔ʁ • ʮiOSͷAPIͷ࢖͍ํʯ͸࠷খݶʹ • εϥΠυʹ͸ࡌ͓͖ͤͯ·͢ • ຊ࣭తɾ൚༻తͳʮਂ౓σʔλͷऔΓѻ͍ํ๏ʯΛཧղ͢Δ

Slide 15

Slide 15 text

ΞδΣϯμ • ਂ౓ͷछྨͱηϯαͷ͘͠Έ • ਂ౓σʔλΛऔಘ͢Δ • ਂ౓σʔλΛ࢖ͬͯΈΔ • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

Slide 16

Slide 16 text

ΞδΣϯμ • ਂ౓ͷछྨͱηϯαͷ͘͠Έ • ਂ౓σʔλΛऔಘ͢Δ • ਂ౓σʔλΛ࢖ͬͯΈΔ • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

Slide 17

Slide 17 text

Disparityʢࢹࠩʣ iPhone 7/8 Plus, Xͷഎ໘Χϝϥ

Slide 18

Slide 18 text

ϐϯϗʔϧΧϝϥ

Slide 19

Slide 19 text

̎୆ͷಉ͡Focal LengthΛ࣋ͭΧϝϥ

Slide 20

Slide 20 text

Disparityʢࢹࠩʣ

Slide 21

Slide 21 text

DepthͱDisparityͷؔ܎

Slide 22

Slide 22 text

Depth͸Disparity͔ΒܭࢉͰ͖Δ • Depth(z) ͷٯ਺ = ਖ਼نԽ͞Εͨ Disparity(d)

Slide 23

Slide 23 text

Time of Flight (ToF) • DepthΛ௚઀ଌΔํࣜ • ର৅ʹরࣹͨ͠ޫ͕൓ࣹͯ͠໭ͬͯ͘Δ·Ͱͷԟ෮ͷ࣌ؒ ͔Βڑ཭Λܭଌ

Slide 24

Slide 24 text

TrueDepth iPhone Xͷલ໘Χϝϥ

Slide 25

Slide 25 text

• Proximity sensorʢۙ઀ηϯαʣɾɾɾToFํࣜͰإ·Ͱͷڑ ཭Λܭଌ

Slide 26

Slide 26 text

• Dot projector (υοτϓϩδΣΫλ)ɾɾɾ3ສҎ্ͷݟ͑ͳ͍ υοτΛإͷ্ʹরࣹ

Slide 27

Slide 27 text

• Infrated camera (੺֎ઢΧϝϥ)ɾɾɾυοτͷύλʔϯΛಡ ΈऔΔ → إͷϚοϓը૾

Slide 28

Slide 28 text

ΞδΣϯμ • ਂ౓ͷछྨͱηϯαͷ͘͠Έ • ਂ౓σʔλΛऔಘ͢Δ • ਂ౓σʔλΛ࢖ͬͯΈΔ • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

Slide 29

Slide 29 text

AVDepthData • ਂ౓Λද͢Ϋϥε • Ͳͷํ๏Ͱऔಘͯ͠΋࠷ऴతʹ͸͜Ε͕ಘΒΕΔ • @available(iOS 11.0, *)

Slide 30

Slide 30 text

AVDepthData var depthDataMap: CVPixelBuffer { get } • ਂ౓ϚοϓͷϐΫηϧσʔλ • ैདྷ͔Β͋ΔCVPixelBufferͳͷͰɺCore ImageͰ΋ MetalͰ΋ɺͦͷޙͷॲཧํ๏͸ࣗࡏ

Slide 31

Slide 31 text

iOSͰਂ౓σʔλΛऔಘ͢Δํ๏ ༻్ผʹ෼͚Δͱ̏छྨ • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ • ํ๏3: ARKit͔Βऔಘ

Slide 32

Slide 32 text

ਂ౓औಘํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ • ImageIOΛར༻ • ʢΧϝϥϩʔϧ͔ΒಡΈࠐΜͰΞηοτͷURLΛಛఆ͢Δͱ ͜Ζ·Ͱ͸Photosʣ

Slide 33

Slide 33 text

ΞηοτͷURL͔ΒCGImageSourceΛ࡞੒ let source = CGImageSourceCreateWithURL(url as CFURL, nil)

Slide 34

Slide 34 text

CGImageSource͔Βਂ౓σʔλͷDictionaryΛऔಘ let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( source, 0, kCGImageAuxiliaryDataTypeDisparity // ଞͷλΠϓ΋͋Γ ) as? [String : AnyObject] // infoʹ͸ਂ౓Ϛοϓͷ CFDataRef ΍ // ϝλσʔλͷCGImageMetadataRef͕ೖ͍ͬͯΔ

Slide 35

Slide 35 text

औಘͨ͠Dictionary͔ΒAVDepthDataΛॳظԽ let depthData = try! AVDepthData(fromDictionaryRepresentation: info)

Slide 36

Slide 36 text

ਂ౓औಘํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ • AVFoundationΛར༻ • جຊతʹ͸ैདྷͷΧϝϥࡱӨͷ࡞๏௨Γ

Slide 37

Slide 37 text

ਂ౓͕औΕΔσόΠελΠϓͷ AVCaptureDevice Λ࢖͏ .builtInDualCamera, .builtInTrueDepthCamera let device = AVCaptureDevice.DiscoverySession( deviceTypes: [.builtInDualCamera], mediaType: .video, position: .back).devices.first

Slide 38

Slide 38 text

ਂ౓͕औΕΔΩϟϓνϟʔϑΥʔϚοτΛࢦఆ͢Δ var supportedDepthDataFormats: [AVCaptureDevice.Format] • AVCaptureDevice.Format ͷϓϩύςΟ • ॴ๬ͷ΋ͷΛબͼɺAVCaptureDevice ͷ activeDepthDataFormat ʹηοτ͢Δ

Slide 39

Slide 39 text

ηογϣϯͷग़ྗʹ AVCaptureDepthDataOutput Λ௥Ճ let depthOutput = AVCaptureDepthDataOutput() captureSession.addOutput(depthOutput) depthOutput.setDelegate( self, callbackQueue: dataOutputQueue) • AVCaptureVideoDataOutputʢϏσΦग़ྗʣ΍ AVCaptureAudioʙʢΦʔσΟΦग़ྗʣΛ௥Ճ͢Δͷͱಉ౳

Slide 40

Slide 40 text

AVCaptureDepthDataOutputDelegate ͷ didOutput ϝιου ͔Β AVDepthData ͕ಘΒΕΔ func depthDataOutput( _ output: AVCaptureDepthDataOutput, didOutput depthData: AVDepthData, timestamp: CMTime, connection: AVCaptureConnection)

Slide 41

Slide 41 text

ਂ౓औಘํ๏3: ARKit͔Βऔಘ ARFrameʹcapturedDepthDataͱ͍͏ϓϩύςΟ͕͋Δ var capturedDepthData: AVDepthData? { get }

Slide 42

Slide 42 text

⚠ ํ๏ʹ༏ྼ͸ͳ͍ɻ༻్ʹΑͬͯબͿɻ • ํ๏1: ࡱӨࡁΈࣸਅ͔Βऔಘ • ํ๏2: Χϝϥ͔ΒϦΞϧλΠϜʹऔಘ • ํ๏3: ARKit͔Βऔಘ

Slide 43

Slide 43 text

ΞδΣϯμ • ਂ౓ͷछྨͱηϯαͷ͘͠Έ • ਂ౓σʔλΛऔಘ͢Δ • ਂ౓σʔλΛ࢖ͬͯΈΔ • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

Slide 44

Slide 44 text

ਂ౓σʔλΛ࢖ͬͯΈΔ ਂ౓σʔλͰഎܠ߹੒

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

എܠ߹੒ͷख๏ ϚεΫը૾Λج४ʹɺΦϦδφϧը૾ͱഎܠը૾ΛϒϨϯυ͢ Δ • ΫϩϚΩʔ߹੒͸ɺϚεΫͷ୅ΘΓʹchromaʢ࠼౓ʣΛར ༻

Slide 47

Slide 47 text

CIBlendWithMask Parameters inputImage ೖྗը૾ inputBackgroundImage എܠը૾ inputMaskImage ϚεΫը૾ • ೖྗը૾͕ἧ͍ͬͯ͑͞Ε͹͜ΕΛ࢖͏͚ͩʂ • CIFilter͸MetalύΠϓϥΠϯͰ΋ॲཧͰ͖Δ

Slide 48

Slide 48 text

ϚεΫͱͯ͠ਂ౓Λ༻͍Ε͹OK!?

Slide 49

Slide 49 text

࣮૷ • ϑϩϯτΧϝϥ͔ΒϦΞϧλΠϜʹਂ౓Λऔಘ • CIFilterͷCIBlendWithMaskͰϒϨϯυ • MetalͰϨϯμϦϯάɺMTKViewʹඳը

Slide 50

Slide 50 text

݁ՌʢσϞʣ

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Կ͕ى͖͍ͯΔͷ͔ʁ

Slide 53

Slide 53 text

ਂ౓ΛՄࢹԽͯ͠ΈΔ

Slide 54

Slide 54 text

໰୊఺ͦͷ̍ɿ നͱࠇ͕൓ର • ϚεΫ͸ɺΦϦδφϧ෦෼͕നͰ͋ Δ΂͖

Slide 55

Slide 55 text

ըૉ஋Λ൓సʢന˱ࠇʣͤ͞Ε͹Α͍ʁ

Slide 56

Slide 56 text

໰୊఺ͦͷ̎ɿ إ͕άϨʔͷ·· • ׬શʹനͰͳ͍ͱɺഎܠ͕ϒϨϯυ ͞Εͯ͠·͏

Slide 57

Slide 57 text

ରࡦɿ إͷਂ౓Λ͖͍͠஋ʹɺਂ౓ ϚοϓΛ2஋Խ͢Δ • AVCaptureMetadataOutputͰإΛೝ ࣝ • إͷத৺ͷਂ౓+offsetΛ͖͍͠஋ ʢcutOffʣͱ͢Δ • ਂ౓Ϛοϓʹ͓͍ͯɺ • ਂ౓͕cutOffҎ্Ͱ͋Ε͹ࠇ • ਂ౓͕cutOffҎԼͰ͋Ε͹ന

Slide 58

Slide 58 text

݁ՌʢσϞʣ

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

ͦͷଞͷ໰୊఺ • ڥք͕ΪβΪβ • ϝΨω͕എܠʹɾɾɾ"Holes"

Slide 61

Slide 61 text

ରࡦɿ • ΪβΪβɾɾɾεϜʔδϯά͢Δ • HolesɾɾɾRGBΛར༻ͯ͠ຒΊΔ

Slide 62

Slide 62 text

isFilteringEnabled • AVCaptureDepthDataOutputͷϓϩύςΟ • ۭؒํ޲͚ͩͰͳ͘ɺ࣌ܥྻํ޲ʹ΋εϜʔδϯά • "Holes"΋ຒΊͯ͘ΕΔʢRGBΛར༻ʣ

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

݁ՌʢσϞʣ

Slide 65

Slide 65 text

׬੒ʂ

Slide 66

Slide 66 text

ඍົʁ • ൅ͷໟͷ·ΘΓ • إͷਂ౓Λج४ʹ͖͍͠஋Λઃఆ͠ ͍ͯΔɾɾɾ൚༻ੑʹ͚ܽΔ

Slide 67

Slide 67 text

Portrait Matte

Slide 68

Slide 68 text

Portrait Matteͱ͸ʁ • എܠͱલܠͷ෼཭ʢηάϝϯςʔγϣϯʣʹಛԽͨ͠ϑΥʔ Ϛοτ • iOS 12͔ΒऔಘՄೳʹͳͬͨ

Slide 69

Slide 69 text

ैདྷͷਂ౓ϚοϓʢࠨʣͱPortrait Matteʢӈʣ

Slide 70

Slide 70 text

൅ͷໟ΋͖Ε͍ʹ෼཭

Slide 71

Slide 71 text

औಘํ๏ • લड़ͷCGImageSourceΛར༻͢Δํ๏ͱ΄΅ಉ͡ • λΠϓͱͯ͠ ʙPortraitEffectsMatte Λࢦఆ͢Δ • AVDapthData ͷ୅ΘΓʹ AVPortraitEffectsMatte

Slide 72

Slide 72 text

guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( self, 0, kCGImageAuxiliaryDataTypePortraitEffectsMatte ) as? [String : AnyObject] else { return } let matte = AVPortraitEffectsMatte(fromDictionaryRepresentation: info)

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Portrait Matteͷ੍໿ • ੩ࢭըͷΈ • ਓ͕ө͍ͬͯΔ৔߹ͷΈ • iOS 12Ҏ্

Slide 75

Slide 75 text

ΞδΣϯμ • ਂ౓ͷछྨͱηϯαͷ͘͠Έ • ਂ౓σʔλΛऔಘ͢Δ • ਂ౓σʔλΛ࢖ͬͯΈΔ • Ͱ͖Δ͜ͱɾͰ͖ͳ͍͜ͱࣄྫू

Slide 76

Slide 76 text

Ͳ͏͍͏έʔεͰɺԿ͕Ͱ͖ͯԿ͕Ͱ͖ͳ͍ͷ͔ ೺Ѳͮ͠Β͍ • ࡱӨࡁΈࣸਅ͔Β / ϦΞϧλΠϜ / ARKit • Dual / TrueDepth / ͦΕҎ֎ • ϑϩϯτ / ϦΞ • iOS 12 / 11 / ͦΕҎԼ • ର৅͕إ / શ਎ / ෺ମ

Slide 77

Slide 77 text

۩ମతͳࣄྫͰઆ໌ͯ͠Έ·͢

Slide 78

Slide 78 text

ʮ͜ΕͰ΋͏੾Γൈ͖࡞ۀ͸Photoshop͍Βͣ ͩʂʯ • ର৅ը૾ʹਂ౓σʔλ͕ೖ͍ͬͯΔඞཁ͕͋Δ • Portrait Matteͷ඼࣭Λظ଴͢ΔͳΒɺਓ͕ؒࣸͬͯͳ͍ͱ͍ ͚ͳ͍

Slide 79

Slide 79 text

ʮ෦԰͕ࢄΒ͔ͬͯͯ΋ϏσΦձٞͰ͖Δʂʯ • Portrait Matte͸ϦΞϧλΠϜͰ͸ར༻Ͱ͖ͳ͍ • Ͱ΋ϑϩϯτΧϝϥͰਓؒͷإΛத৺ʹࡱӨ͢ΔͷͰɺී௨ ͷਂ౓ϕʔεͰ΋ͦΕͳΓʹ͍͚ͦ͏

Slide 80

Slide 80 text

ʮϑϦϚΞϓϦͰग़඼࣌ʹࣗಈͰഎܠΛඈ͹ͤΔ ػೳΛೖΕΑ͏ʯ • ਓҎ֎ͷʮϞϊʯ͸Portrait Matte͸औΕͳ͍ • Disprityϕʔεͷਂ౓͸ͱΕΔ • ͦͷʮϞϊʯͷਂ౓ͷ͖͍͠஋͸Ͳ͏΍ܾͬͯΊΔʁ • ϑΝογϣϯ౳ʹݶఆ͢Ε͹ྑͦ͞͏

Slide 81

Slide 81 text

ʮARKitͷΦΫϧʔδϣϯ͕͜ΕͰͰ ͖Δʂʯ • ϑϩϯτɾɾɾOK • ϦΞɾɾɾNG • ARFaceTrackingConfiguration࢖༻࣌ͷΈ༗ޮ • എ໘ΧϝϥΛ࢖͏ίϯϑΟΪϡϨʔγϣϯʢྫɿ ARWorldTrackingʙʣͰ͸ capturedDepth ͕ nil

Slide 82

Slide 82 text

·ͱΊ • ਂ౓͸ʢࣸਅͷϘέҎ֎ʹ΋ʣൃ૝࣍ୈͰ͍Ζ͍Ζͱ࢖͑Δ • ਂ౓͸Ͳ͏΍ͬͯܭଌ͢Δͷ͔ʁ • ਂ౓σʔλɺͲ͏΍ͬͯ࢖͏ͷ͔ʁ • Կ͕Ͱ͖ͯɺԿ͕Ͱ͖ͳ͍ͷ͔ʁ

Slide 83

Slide 83 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ • Twitter: https://twitter.com/shu223 • GitHub: https://github.com/shu223 • ϒϩά: http://shu223.hatenablog.com/

Slide 84

Slide 84 text

ิ଍༻ࢿྉ

Slide 85

Slide 85 text

AVCaptureDataOutputSynchronizer

Slide 86

Slide 86 text

// ϏσΦग़ྗͱਂ౓ग़ྗΛಉظ synchronizer = AVCaptureDataOutputSynchronizer( dataOutputs: [videoOutput, depthOutput])

Slide 87

Slide 87 text

ಉظͨ͠ग़ྗΛड͚औΔ AVCaptureDataOutputSynchronizerDelegate didOutputσϦήʔτϝιου͔Β AVCaptureSynchronizedDataCollectionΦϒδΣΫτ͕ಘΒΕ Δ • → AVCaptureSynchronizedDepthData • → AVDepthData

Slide 88

Slide 88 text

// ਂ౓σʔλ͕AVCaptureSynchronizedDepthDataܕͰऔΓग़ͤΔ if let syncedDepthData = synchronizedDataCollection.synchronizedData(for: depthOutput) as? AVCaptureSynchronizedDepthData, !syncedDepthData.depthDataWasDropped { // AVDepthDataΦϒδΣΫτΛऔಘ let depthData = syncedDepthData.depthData ... }

Slide 89

Slide 89 text

ਂ౓σʔλΛ࢖ͬͯΈΔ MetalͰࣸਅΛ3DϚοϐ ϯά

Slide 90

Slide 90 text

Processed by Focos

Slide 91

Slide 91 text

• ௖఺γΣʔμ • ਂ౓σʔλΛz࠲ඪʹ • ϑϥάϝϯτγΣʔμ • RGBΧϥʔΛద༻