ライブ配信アプリのアイテム再生をMetalで実装する事になった話

Transcript

1. ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ
   .FUBMͰ࣮૷͢Δࣄʹͳͬͨ࿩
   J04%$+BQBO5SBDL#
   JPTEDC TIBSF
   
   

   View Slide

2. OPQQF
   !
   גࣜձࣾσΟʔɾΤψɾΤʔ
   "
   ͖ͭͶ͔Θ͍͍
   #
   ΞϓϦ։ൃ̎̌̍̌ʙ
   JPTEDC TIBSF
   
   

   View Slide

3. ࠓ೔ͷ࿩
   ɾ1PDPDIBͷ঺հ
   ɾΞΠςϜΤϑΣΫτͷ࠶ੜख๏
   ɾ࠶ੜख๏ͷ࣮૷ൺֱ
   ɾ·ͱΊ
   JPTEDC TIBSF
   
   

   View Slide

4. ϥΠϒ഑৴ΞϓϦ1PDPDIBͷ঺հ
   JPTEDC TIBSF
   
   

   View Slide

5. 1PDPDIB
   w ϥΠϒ഑৴ΞϓϦ
   w ʙ
   w ߴੑೳͳඒഽɾඒإϑΟϧλʔ
   w ΞΠςϜʹΑΔίϛϡχέʔ
   γϣϯ
   JPTEDC TIBSF
   
   

   View Slide

6. ΞΠςϜͷΤϑΣΫτͷ࠶ੜख๏
   JPTEDC TIBSF
   
   

   View Slide

7. JPTEDC TIBSF
   
   

   View Slide

8. ΞΠςϜͷΤϑΣΫτ
   w  º 
   w GQT
   w ಁա৭͋Γ
   w ඵ
   JPTEDC TIBSF
   
   

   View Slide

9. ΞΠςϜͷΤϑΣΫτΛJ04্Ͱ࠶ੜ͢Δʹ͸
   JPTEDC TIBSF
   
   

   View Slide

10. ᶃ"71MBZFSͰ࠶ੜ͢Δ
    let url = Bundle.main.url(forResource: "item_effect",
    withExtension: "mov")!
    let asset = AVURLAsset(url: url)
    let item = AVPlayerItem(asset: asset)
    let player = AVPlayer(playerItem: item)
    let playerLayer = AVPlayerLayer(player: player)
    playerLayer.backgroundColor = UIColor.clear.cgColor
    view.layer.addSubLayer(playerLayer)
    player.play()
    JPTEDC TIBSF
    
    

    View Slide

11. "71MBZFS-BZFSͰ͸ɺಁա৘ใͷ
    ͋ΔಈըΛѻ͏ࣄ͕Ͱ͖ͳ͍ɻ
    *NQPSUBOU
    5IFWBMVFPGBQMBZFSMBZFS`T
    JOIFSJUFEDPOUFOUTQSPQFSUZJT
    PQBRVFBOETIPVMEOPUCF
    VTFE "71-BZFS-BZFSυΩϡϝ
    ϯτΑΓ
    
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO
    BWGPVOEBUJPOBWQMBZFSMBZFS
    JPTEDC TIBSF
    
    

    View Slide

12. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    let imageView = UIImageView()
    let images = (0...300).compactMap({
    UIImage(named: "\($0).png")
    })
    imageView.animationImages = images
    imageView.startAnimating()
    JPTEDC TIBSF
    
    

    View Slide

13. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    $ ffmpeg -i input.mov images/output_%04d.png
    JPTEDC TIBSF
    
    

    View Slide

14. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    $ ffmpeg -i input.mov images/output_%04d.png
    $ du -sh ./images
    302M ./images
    ☺
    JPTEDC TIBSF
    
    

    View Slide

15. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    JPTEDC TIBSF
    
    

    View Slide

16. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    Message from debugger: Terminated due to memory
    issue
    ͰΫϥογϡ
    JPTEDC TIBSF
    
    

    View Slide

17. ݕূ͔Β෼͔ͬͨ̎ͭͷ՝୊ײ
    w ಈըࣗମͷ༰ྔ
    w ࠶ੜύϑΥʔϚϯε
    JPTEDC TIBSF
    
    

    View Slide

18. ಈըࣗମͷ༰ྔ
    ඇѹॖNPWϑΝΠϧ .#
    ࿈൪1/(ϑΝΠϧ .#
    ѹॖ͢Δඞཁ͕͋Δ
    JPTEDC TIBSF
    
    

    View Slide

19. ϑΝΠϧͷѹॖͰඞਢͳཁ݅
    w ಁա৘ใΛ͍࣋ͬͯΔ
    w GQTΛམͱ͞ͳ͍
    w ॎԣͷαΠζΛมߋ͠ͳ͍
    JPTEDC TIBSF
    
    

    View Slide

20. ϑΥʔϚοτ ಁա ѹॖ཰ σίʔυෛՙ
    BQOH ˓ .# º
    XFCQ ˓ .# º TG
    
    XFCN ˓ ,# º
    NQ º .# ˓ I
    
    "1/(͸༰ྔͷ࣌఺Ͱෆ࠾༻ͩͬͨͷͰະଌఆɻ8.͸σίʔμ͕ແͦ͞͏ͳͷͰະଌఆ
    WQXJUIBMQIB
    JPTEDC TIBSF
    
    

    View Slide

21. NQΛͳΜͱ͔ಁաʹରԠग़དྷͳ͍͔
    ͜͜ͰɺੲήʔϜΛ࡞͍ͬͯͨ࣌ͷ͜ͱΛࢥ͍ग़͢
    JPTEDC TIBSF
    
    

    View Slide

22. JPTEDC TIBSF
    
    

    View Slide

23. JPTEDC TIBSF
    
    

    View Slide

24. ಁա߹੒ͱඳը
    ͔͜͜Β͸ɺಁաಈըϓϨʔϠʔΛ࣮૷͍ͯ͘͠࿩
    w $PSF*NBHF
    w 0QFO(-&4
    w .FUBM
    ΛͦΕͧΕ࢖ͬͯ࡞ͬͨ݁ՌΛൺֱ͠·͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

25. ಁա߹੒ͱඳը
    w  º 
    w GQT
    w ಁա৭ͳ͠
    w ඵ
    w NQ
    JPTEDC TIBSF
    
    

    View Slide

26. $*'JMUFSΛ࢖ͬͨ߹੒
    $*'JMUFS
    J04
    Ҏ্ͷϓϦηοτϑΟϧλ
    $16͔(16Ͱॲཧ
    JPTEDC TIBSF
    
    

    View Slide

27. $*'JMUFSΛ࢖ͬͨ߹੒
    $*'JMUFS
    J04
    Ҏ্ͷϓϦηοτϑΟϧλ
    $16͔(16Ͱॲཧ
    $*#MFOE8JUI.BTL
    ΞϧϑΝ஋ͷը૾Λ࢖ͬͯಁա
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO
    DPSFJNBHFDJpMUFS
    JPTEDC TIBSF
    
    

    View Slide

28. $*'JMUFSΛ࢖ͬͨ߹੒
    let filter = CIFilter(name: "CIBlendWithMask")!
    filter.setValue(baseCIImage,
    forKey: kCIInputImageKey)
    filter.setValue(alphaCIImage,
    forKey: kCIInputMaskImageKey)
    let filteredImage = filter.outputImage!
    let image = UIImage(ciImage: filteredImage)
    imageView.image = image
    JPTEDC TIBSF
    
    

    View Slide

29. J1IPOFDGQT
    JPTEDC TIBSF
    
    

    View Slide

30. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    $"&"(--BZFS
    J04r
    (16Ͱॲཧ
    JPTEDC TIBSF
    
    

    View Slide

31. 0QFO(-&45*14
    JPTEDC TIBSF
    
    

    View Slide

32. 0QFO(-&45*14
    &"(-$POUFYU
    ςΫενϟ΍Τϥʔঢ়ଶͳͲΛ؅ཧ
    JPTEDC TIBSF
    
    

    View Slide

33. 0QFO(-&45*14
    // ͲͷίϯςΩετͰಈ࡞͢Δ͔ෆఆ
    glDisable(GLenum(GL_DEPTH_TEST))
    // ར༻͢ΔcontextΛઃఆ
    EAGLContext.setCurrent(context)
    // ↑Ͱࢦఆͨ͠contextͰ࣮ߦ͞ΕΔ
    glDisable(GLenum(GL_DEPTH_TEST))
    JPTEDC TIBSF
    
    

    View Slide

34. 0QFO(-&45*14
    glGetError()
    ྫɿGL_INVALID_ENUM
    ແޮͳFOVN஋Λࢦఆ͍ͯ͠Δɻ
    JPTEDC TIBSF
    
    

    View Slide

35. 0QFO(-&45*14
    9DPEFͰ͸ϒϨʔΫϙΠϯτΛ࢓ࠐΉ͜ͱ΋Մ
    JPTEDC TIBSF
    
    

    View Slide

36. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    $PSF*NBHF 0QFO(-&4
    $**NBHF ςΫενϟ
    $*'JMUFS ϑϥάϝϯτγΣʔμ
    6**NBHF7JFX Ϩϯμϥ $"&"(--BZFS
    
    JPTEDC TIBSF
    
    

    View Slide

37. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    varying lowp vec2 textureCoordinate;
    uniform sampler2D videoFrame;
    void main() {
    // ϕʔεͷ৭ͷx
    lowp float x = textureCoordinate.x / 2.0;
    // ϕʔε৭ͷऔಘ
    lowp vec2 baseCoord = vec2(x, textureCoordinate.y);
    lowp vec4 baseColor = texture2D(videoFrame, baseCoord);
    // alpha஋ͷऔಘ
    lowp vec2 alphaCoord = vec2(x + 0.5, textureCoordinate.y);
    lowp vec4 alphaColor = texture2D(videoFrame, alphaCoord);
    // RGB͸ϕʔε৭ɺA͸alpha஋Λར༻͢Δ
    gl_FragColor = vec4(baseColor.r, baseColor.g, baseColor.b, alphaColor.r);
    }
    JPTEDC TIBSF
    
    

    View Slide

38. HM@'SBH$PMPSWFD CBTF$PMPSS CBTF$PMPSH CBTF$PMPSC BMQIB$PMPSS
    
    JPTEDC TIBSF
    
    

    View Slide

39. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    J1IPOFD㲈GQT
    JPTEDC TIBSF
    
    

    View Slide

40. ͔͠͠ʜ
    JPTEDC TIBSF
    
    

    View Slide

41. JPTEDC TIBSF
    
    

    View Slide

42. 88%$Ͱಥવͷ%FQSFDBUFEɻ
    ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ.FUBMͰ࣮૷͢Δ
    ࣄʹͳͬͨ࿩
    ແࣄλΠτϧճऩ
    ☺
    JPTEDC TIBSF
    
    

    View Slide

43. .FUBMͰॻ͍ͯΈ·ͨ͠
    JPTEDC TIBSF
    
    

    View Slide

44. .FUBMͷར఺
    w γΣʔμʔͷࣄલίϯύΠϧ
    w (16$16ͷڞ༗ϝϞϦۭؒ
    w (16΁ͷ໋ྩΛ·ͱΊͯૹΕΔ
    FUD
    %ΛΰϦΰϦʹ࢖͏ਓ͘Β͍͔͠ت͹ͳ͍ʜ
    JPTEDC TIBSF
    
    

    View Slide

45. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    (16ϓϩάϥϛϯάॳ৺ऀʹ༏͍͠.FUBM
    w (16Λ৮Δ্Ͱॻ͖΍͍͢*'
    w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ
    JPTEDC TIBSF
    
    

    View Slide

46. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    let desc = MTLRenderPipelineDescriptor()
    desc.vertexFunction = library.makeFunction(name: vertexFunctionName)
    desc.fragmentFunction = library.makeFunction(name: fragmentFunctionName)
    desc.colorAttachments[0].pixelFormat = pixelFormat
    4XJGU0CK$Ͱ΋ॻ͖΍͍͢Ϟμϯͳ*'
    ίϯύΠϧ࣌ʹ໰୊͕ൃ֮͠΍͍͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

47. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    γΣʔμʔͷΤϥʔ΋ίϯύΠϧ࣌ʹڭ͑ͯ͘ΕΔ
    JPTEDC TIBSF
    
    

    View Slide

48. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    w (16Λ৮Δ্Ͱॻ͖΍͍͢*'
    w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ
    σόοΨʔʹ͍ͭͯ͸͜ͷޙ঺հ͠·͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

49. .FUBMͰॻ͍ͯΈ·ͨ͠
    ͜ΕΒͷར఺Λ౿·͑ͯɺ.FUBMͰ
    ॻ͍ͯΈ·ͨ͠ɻ
    అ͞Μͷ.FUBMೖ໳͕ಡΈ΍͘͢
    ͓ͯ͢͢ΊͰ͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

50. J1IPOFT㲈GQT
    JPTEDC TIBSF
    
    

    View Slide

51. .FUBMͰॻ͍ͯΈ·ͨ͠
    (16'SBNF$BQUVSF
    w ΧϝϥϚʔΫͷϘλϯ
    JPTEDC TIBSF
    
    

    View Slide

52. JPTEDC TIBSF
    
    

    View Slide

53. ֤ςΫενϟΛQOH΍UJ⒎Ͱॻ͖ग़͢ࣄ΋Ͱ͖·͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

54. JPTEDC TIBSF
    
    

    View Slide

55. 0QFO(-&4൛ͷ(16'SBNF$BQUSVF΍*OTUSVNFOUT
    ͸ɺࣗ෼ͷ؀ڥͰ͸ಈ͔ͳ͘ͳ͍ͬͯ·ͨ͠
    JPTEDC TIBSF
    
    

    View Slide

56. .FUBMͷ*OTUSVNFOUT͸౰વར༻Ͱ͖·͢ɻ
    JPTEDC TIBSF
    
    

    View Slide

57. ·ͱΊ
    JPTEDC TIBSF
    
    

    View Slide

58. ΞϓϦΤϯδχΞ΋.FUBM͸৮Δ΂͖͔
    w ࢥͬͨΑΓ΋೉͘͠ແ͘ɺཧղ͠΍͍͢ੈք
    w "3,JU΍$PSF.-ʹܨ͕Δٕज़
    JPTEDC TIBSF
    
    

    View Slide

59. .FUBMͰ࣮૷͢Δඞཁ͸͋Δͷ͔
    w ը૾ฤूͳΒ͹$*'JMUFSΛ·ͣ͸ݕ౼
    w .FUBMͷσϝϦοτΛཧղ্ͨ͠Ͱ࠾༻͢Δ΂͖
    JPTEDC TIBSF
    
    

    View Slide

60. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺
    w J04ະຬͷσόΠεʹ͸.FUBMඇରԠͷ୺຤͕͋Γ
    ·͢
    w γϛϡϨʔλ͸.FUBMඇରԠͰ͢
    MTLCreateSystemDefaultDevice() != nil
    JPTEDC TIBSF
    
    

    View Slide

61. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺
    γϛϡϨʔλͰϏϧυͰ͖ͳ͍ࣄ΋͋Γ·͢
    Ұ෦ͷϔϧύʔ͸γϛϡϨʔλ4%,ʹؚ·Ε·ͤΜ
    w $".FUBM%SBXBCMF
    w $".FUBM-BZFS
    IUUQTHJTUHJUIVCDPNOPQQFGPYXPMGGBBBDCCEECG
    JPTEDC TIBSF
    
    

    View Slide

62. 1PDPDIBͰ͸
    .FUBMͰ࣮૷͠·͕ͨ͠ʜ
    w ࠓ͸0QFO(-&4൛Λ࢖͍ͬͯ·͢
    w J04΋αϙʔτ͢ΔͨΊ
    w γϛϡϨʔλ࢖͍ͬͯΔ
    0QFO(-&4͕ഇࢭ͞ΕͨΒͲ͏͢Δͷʁ
    ˠͦͷࠒʹ͸J04΋ରԠ͠ͳ͍ͩΖ͏͠ɺγϛϡϨʔλͰ΋.FUBM͕ಈ͘Α͏ʹ
    ͳ͍ͬͯ·͢ʂʂʢଟ෼Ͷʣ
    JPTEDC TIBSF
    
    

    View Slide

63. 4PVSDF$PEF
    ಁաॲཧΛࢪͯ͠࠶ੜ͢ΔϓϨʔϠʔϥΠϒϥϦΛ࡞͍ͬͯ·͢
    JPTEDC TIBSF
    
    

    View Slide

64. 4PVSDF$PEF
    w OPQQFGPYXPMG,JUTVOFCJ
    w OPQQFGPYXPMG,JUTVOFCJ@.FUBM
    !
    
    "
    
    !
    JPTEDC TIBSF
    
    

    View Slide

65. !
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    "TLUIFTQFBLFS
    %F/"4QFBLFS-PVOHF
    JPTEDC TIBSF
    
    

    View Slide