ライブ配信アプリのアイテム再生をMetalで実装する事になった話

by noppefoxwolf

Slide 1

Slide 1 text

ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ .FUBMͰ࣮૷͢Δࣄʹͳͬͨ࿩ J04%$+BQBO5SBDL# JPTEDC TIBSF

Slide 2

Slide 2 text

OPQQF ! גࣜձࣾσΟʔɾΤψɾΤʔ " ͖ͭͶ͔Θ͍͍ # ΞϓϦ։ൃ̎̌̍̌ʙ JPTEDC TIBSF

Slide 3

Slide 3 text

ࠓ೔ͷ࿩ ɾ1PDPDIBͷ঺հ ɾΞΠςϜΤϑΣΫτͷ࠶ੜख๏ ɾ࠶ੜख๏ͷ࣮૷ൺֱ ɾ·ͱΊ JPTEDC TIBSF

Slide 4

Slide 4 text

ϥΠϒ഑৴ΞϓϦ1PDPDIBͷ঺հ JPTEDC TIBSF

Slide 5

Slide 5 text

1PDPDIB w ϥΠϒ഑৴ΞϓϦ w ʙ w ߴੑೳͳඒഽɾඒإϑΟϧλʔ w ΞΠςϜʹΑΔίϛϡχέʔ γϣϯ JPTEDC TIBSF

Slide 6

Slide 6 text

ΞΠςϜͷΤϑΣΫτͷ࠶ੜख๏ JPTEDC TIBSF

Slide 7

Slide 7 text

JPTEDC TIBSF

Slide 8

Slide 8 text

ΞΠςϜͷΤϑΣΫτ w º w GQT w ಁա৭͋Γ w ඵ JPTEDC TIBSF

Slide 9

Slide 9 text

ΞΠςϜͷΤϑΣΫτΛJ04্Ͱ࠶ੜ͢Δʹ͸ JPTEDC TIBSF

Slide 10

Slide 10 text

ᶃ"71MBZFSͰ࠶ੜ͢Δ let url = Bundle.main.url(forResource: "item_effect", withExtension: "mov")! let asset = AVURLAsset(url: url) let item = AVPlayerItem(asset: asset) let player = AVPlayer(playerItem: item) let playerLayer = AVPlayerLayer(player: player) playerLayer.backgroundColor = UIColor.clear.cgColor view.layer.addSubLayer(playerLayer) player.play() JPTEDC TIBSF

Slide 11

Slide 11 text

"71MBZFS-BZFSͰ͸ɺಁա৘ใͷ ͋ΔಈըΛѻ͏ࣄ͕Ͱ͖ͳ͍ɻ *NQPSUBOU 5IFWBMVFPGBQMBZFSMBZFS`T JOIFSJUFEDPOUFOUTQSPQFSUZJT PQBRVFBOETIPVMEOPUCF VTFE "71-BZFS-BZFSυΩϡϝ ϯτΑΓ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO BWGPVOEBUJPOBWQMBZFSMBZFS JPTEDC TIBSF

Slide 12

Slide 12 text

ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ let imageView = UIImageView() let images = (0...300).compactMap({ UIImage(named: "\($0).png") }) imageView.animationImages = images imageView.startAnimating() JPTEDC TIBSF

Slide 13

Slide 13 text

ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ $ ffmpeg -i input.mov images/output_%04d.png JPTEDC TIBSF

Slide 14

Slide 14 text

ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ $ ffmpeg -i input.mov images/output_%04d.png $ du -sh ./images 302M ./images ☺ JPTEDC TIBSF

Slide 15

Slide 15 text

ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ JPTEDC TIBSF

Slide 16

Slide 16 text

ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ Message from debugger: Terminated due to memory issue ͰΫϥογϡ JPTEDC TIBSF

Slide 17

Slide 17 text

ݕূ͔Β෼͔ͬͨ̎ͭͷ՝୊ײ w ಈըࣗମͷ༰ྔ w ࠶ੜύϑΥʔϚϯε JPTEDC TIBSF

Slide 18

Slide 18 text

ಈըࣗମͷ༰ྔ ඇѹॖNPWϑΝΠϧ .# ࿈൪1/(ϑΝΠϧ .# ѹॖ͢Δඞཁ͕͋Δ JPTEDC TIBSF

Slide 19

Slide 19 text

ϑΝΠϧͷѹॖͰඞਢͳཁ݅ w ಁա৘ใΛ͍࣋ͬͯΔ w GQTΛམͱ͞ͳ͍ w ॎԣͷαΠζΛมߋ͠ͳ͍ JPTEDC TIBSF

Slide 20

Slide 20 text

ϑΥʔϚοτ ಁա ѹॖ཰ σίʔυෛՙ BQOH ˓ .# º XFCQ ˓ .# º TG XFCN ˓ ,# º NQ º .# ˓ I "1/(͸༰ྔͷ࣌఺Ͱෆ࠾༻ͩͬͨͷͰະଌఆɻ8.͸σίʔμ͕ແͦ͞͏ͳͷͰະଌఆ WQXJUIBMQIB JPTEDC TIBSF

Slide 21

Slide 21 text

NQΛͳΜͱ͔ಁաʹରԠग़དྷͳ͍͔ ͜͜ͰɺੲήʔϜΛ࡞͍ͬͯͨ࣌ͷ͜ͱΛࢥ͍ग़͢ JPTEDC TIBSF

Slide 22

Slide 22 text

JPTEDC TIBSF

Slide 23

Slide 23 text

JPTEDC TIBSF

Slide 24

Slide 24 text

ಁա߹੒ͱඳը ͔͜͜Β͸ɺಁաಈըϓϨʔϠʔΛ࣮૷͍ͯ͘͠࿩ w $PSF*NBHF w 0QFO(-&4 w .FUBM ΛͦΕͧΕ࢖ͬͯ࡞ͬͨ݁ՌΛൺֱ͠·͢ɻ JPTEDC TIBSF

Slide 25

Slide 25 text

ಁա߹੒ͱඳը w º w GQT w ಁա৭ͳ͠ w ඵ w NQ JPTEDC TIBSF

Slide 26

Slide 26 text

$*'JMUFSΛ࢖ͬͨ߹੒ $*'JMUFS J04 Ҏ্ͷϓϦηοτϑΟϧλ $16͔(16Ͱॲཧ JPTEDC TIBSF

Slide 27

Slide 27 text

$*'JMUFSΛ࢖ͬͨ߹੒ $*'JMUFS J04 Ҏ্ͷϓϦηοτϑΟϧλ $16͔(16Ͱॲཧ $*#MFOE8JUI.BTL ΞϧϑΝ஋ͷը૾Λ࢖ͬͯಁա IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO DPSFJNBHFDJpMUFS JPTEDC TIBSF

Slide 28

Slide 28 text

$*'JMUFSΛ࢖ͬͨ߹੒ let filter = CIFilter(name: "CIBlendWithMask")! filter.setValue(baseCIImage, forKey: kCIInputImageKey) filter.setValue(alphaCIImage, forKey: kCIInputMaskImageKey) let filteredImage = filter.outputImage! let image = UIImage(ciImage: filteredImage) imageView.image = image JPTEDC TIBSF

Slide 29

Slide 29 text

J1IPOFDGQT JPTEDC TIBSF

Slide 30

Slide 30 text

0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը $"&"(--BZFS J04r (16Ͱॲཧ JPTEDC TIBSF

Slide 31

Slide 31 text

0QFO(-&45*14 JPTEDC TIBSF

Slide 32

Slide 32 text

0QFO(-&45*14 &"(-$POUFYU ςΫενϟ΍Τϥʔঢ়ଶͳͲΛ؅ཧ JPTEDC TIBSF

Slide 33

Slide 33 text

0QFO(-&45*14 // ͲͷίϯςΩετͰಈ࡞͢Δ͔ෆఆ glDisable(GLenum(GL_DEPTH_TEST)) // ར༻͢ΔcontextΛઃఆ EAGLContext.setCurrent(context) // ↑Ͱࢦఆͨ͠contextͰ࣮ߦ͞ΕΔ glDisable(GLenum(GL_DEPTH_TEST)) JPTEDC TIBSF

Slide 34

Slide 34 text

0QFO(-&45*14 glGetError() ྫɿGL_INVALID_ENUM ແޮͳFOVN஋Λࢦఆ͍ͯ͠Δɻ JPTEDC TIBSF

Slide 35

Slide 35 text

0QFO(-&45*14 9DPEFͰ͸ϒϨʔΫϙΠϯτΛ࢓ࠐΉ͜ͱ΋Մ JPTEDC TIBSF

Slide 36

Slide 36 text

0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը $PSF*NBHF 0QFO(-&4 $**NBHF ςΫενϟ $*'JMUFS ϑϥάϝϯτγΣʔμ 6**NBHF7JFX Ϩϯμϥ $"&"(--BZFS JPTEDC TIBSF

Slide 37

Slide 37 text

0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը varying lowp vec2 textureCoordinate; uniform sampler2D videoFrame; void main() { // ϕʔεͷ৭ͷx lowp float x = textureCoordinate.x / 2.0; // ϕʔε৭ͷऔಘ lowp vec2 baseCoord = vec2(x, textureCoordinate.y); lowp vec4 baseColor = texture2D(videoFrame, baseCoord); // alpha஋ͷऔಘ lowp vec2 alphaCoord = vec2(x + 0.5, textureCoordinate.y); lowp vec4 alphaColor = texture2D(videoFrame, alphaCoord); // RGB͸ϕʔε৭ɺA͸alpha஋Λར༻͢Δ gl_FragColor = vec4(baseColor.r, baseColor.g, baseColor.b, alphaColor.r); } JPTEDC TIBSF

Slide 38

Slide 38 text

HM@'SBH$PMPSWFD CBTF$PMPSS CBTF$PMPSH CBTF$PMPSC BMQIB$PMPSS JPTEDC TIBSF

Slide 39

Slide 39 text

0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը J1IPOFD㲈GQT JPTEDC TIBSF

Slide 40

Slide 40 text

͔͠͠ʜ JPTEDC TIBSF

Slide 41

Slide 41 text

JPTEDC TIBSF

Slide 42

Slide 42 text

88%$Ͱಥવͷ%FQSFDBUFEɻ ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ.FUBMͰ࣮૷͢Δ ࣄʹͳͬͨ࿩ ແࣄλΠτϧճऩ ☺ JPTEDC TIBSF

Slide 43

Slide 43 text

.FUBMͰॻ͍ͯΈ·ͨ͠ JPTEDC TIBSF

Slide 44

Slide 44 text

.FUBMͷར఺ w γΣʔμʔͷࣄલίϯύΠϧ w (16$16ͷڞ༗ϝϞϦۭؒ w (16΁ͷ໋ྩΛ·ͱΊͯૹΕΔ FUD %ΛΰϦΰϦʹ࢖͏ਓ͘Β͍͔͠ت͹ͳ͍ʜ JPTEDC TIBSF

Slide 45

Slide 45 text

.FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ (16ϓϩάϥϛϯάॳ৺ऀʹ༏͍͠.FUBM w (16Λ৮Δ্Ͱॻ͖΍͍͢*' w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ JPTEDC TIBSF

Slide 46

Slide 46 text

.FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ let desc = MTLRenderPipelineDescriptor() desc.vertexFunction = library.makeFunction(name: vertexFunctionName) desc.fragmentFunction = library.makeFunction(name: fragmentFunctionName) desc.colorAttachments[0].pixelFormat = pixelFormat 4XJGU0CK$Ͱ΋ॻ͖΍͍͢Ϟμϯͳ*' ίϯύΠϧ࣌ʹ໰୊͕ൃ֮͠΍͍͢ɻ JPTEDC TIBSF

Slide 47

Slide 47 text

.FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ γΣʔμʔͷΤϥʔ΋ίϯύΠϧ࣌ʹڭ͑ͯ͘ΕΔ JPTEDC TIBSF

Slide 48

Slide 48 text

.FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ w (16Λ৮Δ্Ͱॻ͖΍͍͢*' w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ σόοΨʔʹ͍ͭͯ͸͜ͷޙ঺հ͠·͢ɻ JPTEDC TIBSF

Slide 49

Slide 49 text

.FUBMͰॻ͍ͯΈ·ͨ͠ ͜ΕΒͷར఺Λ౿·͑ͯɺ.FUBMͰ ॻ͍ͯΈ·ͨ͠ɻ అ͞Μͷ.FUBMೖ໳͕ಡΈ΍͘͢ ͓ͯ͢͢ΊͰ͢ɻ JPTEDC TIBSF

Slide 50

Slide 50 text

J1IPOFT㲈GQT JPTEDC TIBSF

Slide 51

Slide 51 text

.FUBMͰॻ͍ͯΈ·ͨ͠ (16'SBNF$BQUVSF w ΧϝϥϚʔΫͷϘλϯ JPTEDC TIBSF

Slide 52

Slide 52 text

JPTEDC TIBSF

Slide 53

Slide 53 text

֤ςΫενϟΛQOH΍UJ⒎Ͱॻ͖ग़͢ࣄ΋Ͱ͖·͢ɻ JPTEDC TIBSF

Slide 54

Slide 54 text

JPTEDC TIBSF

Slide 55

Slide 55 text

0QFO(-&4൛ͷ(16'SBNF$BQUSVF΍*OTUSVNFOUT ͸ɺࣗ෼ͷ؀ڥͰ͸ಈ͔ͳ͘ͳ͍ͬͯ·ͨ͠ JPTEDC TIBSF

Slide 56

Slide 56 text

.FUBMͷ*OTUSVNFOUT͸౰વར༻Ͱ͖·͢ɻ JPTEDC TIBSF

Slide 57

Slide 57 text

·ͱΊ JPTEDC TIBSF

Slide 58

Slide 58 text

ΞϓϦΤϯδχΞ΋.FUBM͸৮Δ΂͖͔ w ࢥͬͨΑΓ΋೉͘͠ແ͘ɺཧղ͠΍͍͢ੈք w "3,JU΍$PSF.-ʹܨ͕Δٕज़ JPTEDC TIBSF

Slide 59

Slide 59 text

.FUBMͰ࣮૷͢Δඞཁ͸͋Δͷ͔ w ը૾ฤूͳΒ͹$*'JMUFSΛ·ͣ͸ݕ౼ w .FUBMͷσϝϦοτΛཧղ্ͨ͠Ͱ࠾༻͢Δ΂͖ JPTEDC TIBSF

Slide 60

Slide 60 text

.FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺ w J04ະຬͷσόΠεʹ͸.FUBMඇରԠͷ୺຤͕͋Γ ·͢ w γϛϡϨʔλ͸.FUBMඇରԠͰ͢ MTLCreateSystemDefaultDevice() != nil JPTEDC TIBSF

Slide 61

Slide 61 text

.FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺ γϛϡϨʔλͰϏϧυͰ͖ͳ͍ࣄ΋͋Γ·͢ Ұ෦ͷϔϧύʔ͸γϛϡϨʔλ4%,ʹؚ·Ε·ͤΜ w $".FUBM%SBXBCMF w $".FUBM-BZFS IUUQTHJTUHJUIVCDPNOPQQFGPYXPMGGBBBDCCEECG JPTEDC TIBSF

Slide 62

Slide 62 text

1PDPDIBͰ͸ .FUBMͰ࣮૷͠·͕ͨ͠ʜ w ࠓ͸0QFO(-&4൛Λ࢖͍ͬͯ·͢ w J04΋αϙʔτ͢ΔͨΊ w γϛϡϨʔλ࢖͍ͬͯΔ 0QFO(-&4͕ഇࢭ͞ΕͨΒͲ͏͢Δͷʁ ˠͦͷࠒʹ͸J04΋ରԠ͠ͳ͍ͩΖ͏͠ɺγϛϡϨʔλͰ΋.FUBM͕ಈ͘Α͏ʹ ͳ͍ͬͯ·͢ʂʂʢଟ෼Ͷʣ JPTEDC TIBSF

Slide 63

Slide 63 text

4PVSDF$PEF ಁաॲཧΛࢪͯ͠࠶ੜ͢ΔϓϨʔϠʔϥΠϒϥϦΛ࡞͍ͬͯ·͢ JPTEDC TIBSF

Slide 64

Slide 64 text

4PVSDF$PEF w OPQQFGPYXPMG,JUTVOFCJ w OPQQFGPYXPMG,[email protected] ! " ! JPTEDC TIBSF

Slide 65

Slide 65 text

! ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ "TLUIFTQFBLFS %F/"4QFBLFS-PVOHF JPTEDC TIBSF