ライブ配信アプリのアイテム再生をMetalで実装する事になった話

Transcript

1. ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ .FUBMͰ࣮૷͢Δࣄʹͳͬͨ࿩ J04%$
   +BQBO
   5SBDL
   #
    JPTED
   C
   
   
   TIBSF 
2. OPQQF !
   גࣜձࣾσΟʔɾΤψɾΤʔ "
   ͖ͭͶ͔Θ͍͍ #
   ΞϓϦ։ൃ̎̌̍̌ʙ JPTED
   C
   
   
   TIBSF 
3. ࠓ೔ͷ࿩ ɾ1PDPDIBͷ঺հ ɾΞΠςϜΤϑΣΫτͷ࠶ੜख๏ ɾ࠶ੜख๏ͷ࣮૷ൺֱ ɾ·ͱΊ JPTED
   C
   
   
   TIBSF 
4. ϥΠϒ഑৴ΞϓϦ1PDPDIBͷ঺հ JPTED
   C
   
   
   TIBSF 
5. 1PDPDIB w ϥΠϒ഑৴ΞϓϦ w 
   ʙ w ߴੑೳͳඒഽɾඒإϑΟϧλʔ w ΞΠςϜʹΑΔίϛϡχέʔ γϣϯ JPTED
   C
   
   
   TIBSF 
6. ΞΠςϜͷΤϑΣΫτͷ࠶ੜख๏ JPTED
   C
   
   
   TIBSF 
7. JPTED
   C
   
   
   TIBSF 
8. ΞΠςϜͷΤϑΣΫτ w  º  w GQT
   w ಁա৭͋Γ w ඵ JPTED
   C
   
   
   TIBSF 
9. ΞΠςϜͷΤϑΣΫτΛJ04্Ͱ࠶ੜ͢Δʹ͸ JPTED
   C
   
   
   TIBSF 
10. ᶃ
    "71MBZFSͰ࠶ੜ͢Δ let url = Bundle.main.url(forResource: "item_effect", withExtension: "mov")! let asset = AVURLAsset(url: url) let item = AVPlayerItem(asset: asset) let player = AVPlayer(playerItem: item) let playerLayer = AVPlayerLayer(player: player) playerLayer.backgroundColor = UIColor.clear.cgColor view.layer.addSubLayer(playerLayer) player.play() JPTED
    C
    
    
    TIBSF 
11. "71MBZFS-BZFSͰ͸ɺಁա৘ใͷ ͋ΔಈըΛѻ͏ࣄ͕Ͱ͖ͳ͍ɻ *NQPSUBOU 5IF
    WBMVF
    PG
    B
    QMBZFS
    MBZFS`T
    JOIFSJUFE
    DPOUFOUT
    QSPQFSUZ
    JT
    PQBRVF
    BOE
    TIPVME
    OPU
    CF
    VTFE
    "71-BZFS-BZFSυΩϡϝ ϯτΑΓ 
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO BWGPVOEBUJPOBWQMBZFSMBZFS JPTED
    C
    
    
    TIBSF 
12. ᶄ
    6**NBHF7JFXͰ࠶ੜ͢Δ let imageView = UIImageView() let images = (0...300).compactMap({ UIImage(named: "\($0).png") }) imageView.animationImages = images imageView.startAnimating() JPTED
    C
    
    
    TIBSF 
13. ᶄ
    6**NBHF7JFXͰ࠶ੜ͢Δ $ ffmpeg -i input.mov images/output_%04d.png JPTED
    C
    
    
    TIBSF 
14. ᶄ
    6**NBHF7JFXͰ࠶ੜ͢Δ $ ffmpeg -i input.mov images/output_%04d.png $ du -sh ./images 302M ./images ☺ JPTED
    C
    
    
    TIBSF 
15. ᶄ
    6**NBHF7JFXͰ࠶ੜ͢Δ JPTED
    C
    
    
    TIBSF 
16. ᶄ
    6**NBHF7JFXͰ࠶ੜ͢Δ Message from debugger: Terminated due to memory issue ͰΫϥογϡ JPTED
    C
    
    
    TIBSF 
17. ݕূ͔Β෼͔ͬͨ̎ͭͷ՝୊ײ w ಈըࣗମͷ༰ྔ w ࠶ੜύϑΥʔϚϯε JPTED
    C
    
    
    TIBSF 
18. ಈըࣗମͷ༰ྔ ඇѹॖNPWϑΝΠϧ .# ࿈൪1/(ϑΝΠϧ .# ѹॖ͢Δඞཁ͕͋Δ JPTED
    C
    
    
    TIBSF 
19. ϑΝΠϧͷѹॖͰඞਢͳཁ݅ w ಁա৘ใΛ͍࣋ͬͯΔ w GQTΛམͱ͞ͳ͍ w ॎԣͷαΠζΛมߋ͠ͳ͍ JPTED
    C
    
    
    TIBSF 
20. ϑΥʔϚοτ ಁա ѹॖ཰ σίʔυෛՙ BQOH ˓ .# º XFCQ ˓ .# º TG XFCN ˓ ,# º NQ º .# ˓ I 
    "1/(͸༰ྔͷ࣌఺Ͱෆ࠾༻ͩͬͨͷͰະଌఆɻ8&#.͸σίʔμ͕ແͦ͞͏ͳͷͰະଌఆ 
    WQ
    XJUI
    BMQIB JPTED
    C
    
    
    TIBSF 
21. NQΛͳΜͱ͔ಁաʹରԠग़དྷͳ͍͔ ͜͜ͰɺੲήʔϜΛ࡞͍ͬͯͨ࣌ͷ͜ͱΛࢥ͍ग़͢ JPTED
    C
    
    
    TIBSF 
22. JPTED
    C
    
    
    TIBSF 
23. JPTED
    C
    
    
    TIBSF 
24. ಁա߹੒ͱඳը ͔͜͜Β͸ɺಁաಈըϓϨʔϠʔΛ࣮૷͍ͯ͘͠࿩ w $PSF*NBHF w 0QFO(-&4 w .FUBM ΛͦΕͧΕ࢖ͬͯ࡞ͬͨ݁ՌΛൺֱ͠·͢ɻ JPTED
    C
    
    
    TIBSF 
25. ಁա߹੒ͱඳը w  º  w GQT
    w ಁա৭ͳ͠ w ඵ w NQ JPTED
    C
    
    
    TIBSF 
26. $*'JMUFSΛ࢖ͬͨ߹੒ $*'JMUFS
    
    J04
    
    
    Ҏ্ͷϓϦηοτϑΟϧλ
    
    $16͔(16Ͱॲཧ JPTED
    C
    
    
    TIBSF 
27. $*'JMUFSΛ࢖ͬͨ߹੒ $*'JMUFS
    
    J04
    
    
    Ҏ্ͷϓϦηοτϑΟϧλ
    
    $16͔(16Ͱॲཧ $*#MFOE8JUI.BTL
    
    ΞϧϑΝ஋ͷը૾Λ࢖ͬͯಁա 
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO DPSFJNBHFDJpMUFS JPTED
    C
    
    
    TIBSF 
28. $*'JMUFSΛ࢖ͬͨ߹੒ let filter = CIFilter(name: "CIBlendWithMask")! filter.setValue(baseCIImage, forKey: kCIInputImageKey) filter.setValue(alphaCIImage, forKey: kCIInputMaskImageKey) let filteredImage = filter.outputImage! let image = UIImage(ciImage: filteredImage) imageView.image = image JPTED
    C
    
    
    TIBSF 
29. J1IPOFD
    
    GQT JPTED
    C
    
    
    TIBSF 
30. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը $"&"(--BZFS
    
    J04
    r
    
    (16Ͱॲཧ JPTED
    C
    
    
    TIBSF 
31. 0QFO(-&4
    5*14 JPTED
    C
    
    
    TIBSF 
32. 0QFO(-&4
    5*14 &"(-$POUFYU
    
    ςΫενϟ΍Τϥʔঢ়ଶͳͲΛ؅ཧ JPTED
    C
    
    
    TIBSF 
33. 0QFO(-&4
    5*14 // ͲͷίϯςΩετͰಈ࡞͢Δ͔ෆఆ glDisable(GLenum(GL_DEPTH_TEST)) // ར༻͢ΔcontextΛઃఆ EAGLContext.setCurrent(context) // ↑Ͱࢦఆͨ͠contextͰ࣮ߦ͞ΕΔ glDisable(GLenum(GL_DEPTH_TEST)) JPTED
    C
    
    
    TIBSF 
34. 0QFO(-&4
    5*14 glGetError() ྫɿGL_INVALID_ENUM
    
    ແޮͳFOVN஋Λࢦఆ͍ͯ͠Δɻ JPTED
    C
    
    
    TIBSF 
35. 0QFO(-&4
    5*14 9DPEFͰ͸ϒϨʔΫϙΠϯτΛ࢓ࠐΉ͜ͱ΋Մ JPTED
    C
    
    
    TIBSF 
36. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը $PSF*NBHF 0QFO(-&4 $**NBHF ςΫενϟ $*'JMUFS ϑϥάϝϯτγΣʔμ 6**NBHF7JFX Ϩϯμϥ $"&"(--BZFS JPTED
    C
    
    
    TIBSF 
37. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը varying lowp vec2 textureCoordinate; uniform sampler2D videoFrame; void main() { // ϕʔεͷ৭ͷx lowp float x = textureCoordinate.x / 2.0; // ϕʔε৭ͷऔಘ lowp vec2 baseCoord = vec2(x, textureCoordinate.y); lowp vec4 baseColor = texture2D(videoFrame, baseCoord); // alpha஋ͷऔಘ lowp vec2 alphaCoord = vec2(x + 0.5, textureCoordinate.y); lowp vec4 alphaColor = texture2D(videoFrame, alphaCoord); // RGB͸ϕʔε৭ɺA͸alpha஋Λར༻͢Δ gl_FragColor = vec4(baseColor.r, baseColor.g, baseColor.b, alphaColor.r); } JPTED
    C
    
    
    TIBSF 
38. HM@'SBH$PMPS
    
    WFD CBTF$PMPSS
    CBTF$PMPSH
    CBTF$PMPSC
    BMQIB$PMPSS  JPTED
    C
    
    
    TIBSF 
39. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը J1IPOFD
    
    㲈GQT JPTED
    C
    
    
    TIBSF 
40. ͔͠͠ʜ JPTED
    C
    
    
    TIBSF 
41. JPTED
    C
    
    
    TIBSF 
42. 88%$Ͱಥવͷ%FQSFDBUFEɻ ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ.FUBMͰ࣮૷͢Δ ࣄʹͳͬͨ࿩ ແࣄλΠτϧճऩ ☺ JPTED
    C
    
    
    TIBSF 
43. .FUBMͰॻ͍ͯΈ·ͨ͠ JPTED
    C
    
    
    TIBSF 
44. .FUBMͷར఺ w γΣʔμʔͷࣄલίϯύΠϧ w (16$16ͷڞ༗ϝϞϦۭؒ w (16΁ͷ໋ྩΛ·ͱΊͯૹΕΔ FUD %ΛΰϦΰϦʹ࢖͏ਓ͘Β͍͔͠ت͹ͳ͍ʜ JPTED
    C
    
    
    TIBSF 
45. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ (16ϓϩάϥϛϯάॳ৺ऀʹ༏͍͠.FUBM w (16Λ৮Δ্Ͱॻ͖΍͍͢*' w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ JPTED
    C
    
    
    TIBSF 
46. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ let desc = MTLRenderPipelineDescriptor() desc.vertexFunction = library.makeFunction(name: vertexFunctionName) desc.fragmentFunction = library.makeFunction(name: fragmentFunctionName) desc.colorAttachments[0].pixelFormat = pixelFormat 4XJGU0CK$Ͱ΋ॻ͖΍͍͢Ϟμϯͳ*' ίϯύΠϧ࣌ʹ໰୊͕ൃ֮͠΍͍͢ɻ JPTED
    C
    
    
    TIBSF 
47. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ γΣʔμʔͷΤϥʔ΋ίϯύΠϧ࣌ʹڭ͑ͯ͘ΕΔ JPTED
    C
    
    
    TIBSF 
48. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺ w (16Λ৮Δ্Ͱॻ͖΍͍͢*' w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ σόοΨʔʹ͍ͭͯ͸͜ͷޙ঺հ͠·͢ɻ JPTED
    C
    
    
    TIBSF 
49. .FUBMͰॻ͍ͯΈ·ͨ͠ ͜ΕΒͷར఺Λ౿·͑ͯɺ.FUBMͰ ॻ͍ͯΈ·ͨ͠ɻ అ͞Μͷ.FUBMೖ໳͕ಡΈ΍͘͢ ͓ͯ͢͢ΊͰ͢ɻ JPTED
    C
    
    
    TIBSF 
50. J1IPOFT
    
    㲈GQT JPTED
    C
    
    
    TIBSF 
51. .FUBMͰॻ͍ͯΈ·ͨ͠ (16
    'SBNF
    $BQUVSF w ΧϝϥϚʔΫͷϘλϯ JPTED
    C
    
    
    TIBSF 
52. JPTED
    C
    
    
    TIBSF 
53. ֤ςΫενϟΛQOH΍UJ⒎Ͱॻ͖ग़͢ࣄ΋Ͱ͖·͢ɻ JPTED
    C
    
    
    TIBSF 
54. JPTED
    C
    
    
    TIBSF 
55. 0QFO(-&4൛ͷ(16
    'SBNF
    $BQUSVF΍*OTUSVNFOUT ͸ɺࣗ෼ͷ؀ڥͰ͸ಈ͔ͳ͘ͳ͍ͬͯ·ͨ͠ JPTED
    C
    
    
    TIBSF 
56. .FUBMͷ*OTUSVNFOUT͸౰વར༻Ͱ͖·͢ɻ JPTED
    C
    
    
    TIBSF 
57. ·ͱΊ JPTED
    C
    
    
    TIBSF 
58. ΞϓϦΤϯδχΞ΋.FUBM͸৮Δ΂͖͔ w ࢥͬͨΑΓ΋೉͘͠ແ͘ɺཧղ͠΍͍͢ੈք w "3,JU΍$PSF.-ʹܨ͕Δٕज़ JPTED
    C
    
    
    TIBSF 
59. .FUBMͰ࣮૷͢Δඞཁ͸͋Δͷ͔ w ը૾ฤूͳΒ͹$*'JMUFSΛ·ͣ͸ݕ౼ w .FUBMͷσϝϦοτΛཧղ্ͨ͠Ͱ࠾༻͢Δ΂͖ JPTED
    C
    
    
    TIBSF 
60. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺ w J04ະຬͷσόΠεʹ͸.FUBMඇରԠͷ୺຤͕͋Γ ·͢ w γϛϡϨʔλ͸.FUBMඇରԠͰ͢ MTLCreateSystemDefaultDevice() != nil JPTED
    C
    
    
    TIBSF 
61. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺ γϛϡϨʔλͰϏϧυͰ͖ͳ͍ࣄ΋͋Γ·͢ Ұ෦ͷϔϧύʔ͸γϛϡϨʔλ4%,ʹؚ·Ε·ͤΜ w $".FUBM%SBXBCMF w $".FUBM-BZFS 
    IUUQTHJTUHJUIVCDPNOPQQFGPYXPMGGBBBDCCEECG JPTED
    C
    
    
    TIBSF 
62. 1PDPDIBͰ͸ .FUBMͰ࣮૷͠·͕ͨ͠ʜ w ࠓ͸0QFO(-&4൛Λ࢖͍ͬͯ·͢ w J04΋αϙʔτ͢ΔͨΊ w γϛϡϨʔλ࢖͍ͬͯΔ 0QFO(-&4͕ഇࢭ͞ΕͨΒͲ͏͢Δͷʁ ˠͦͷࠒʹ͸J04΋ରԠ͠ͳ͍ͩΖ͏͠ɺγϛϡϨʔλͰ΋.FUBM͕ಈ͘Α͏ʹ ͳ͍ͬͯ·͢ʂʂʢଟ෼Ͷʣ JPTED
    C
    
    
    TIBSF 
63. 4PVSDF$PEF ಁաॲཧΛࢪͯ͠࠶ੜ͢ΔϓϨʔϠʔϥΠϒϥϦΛ࡞͍ͬͯ·͢ JPTED
    C
    
    
    TIBSF 
64. 4PVSDF$PEF w OPQQFGPYXPMG,JUTVOFCJ w OPQQFGPYXPMG,JUTVOFCJ@.FUBM !
    "
    ! JPTED
    C
    
    
    TIBSF 
65. !
    
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ "TL
    UIF
    TQFBLFS 
    
     %F/"
    4QFBLFS
    -PVOHF 
    
     JPTED
    C
    
    
    TIBSF