Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ライブ配信アプリのアイテム再生をMetalで実装する事になった話

noppefoxwolf
September 02, 2018

 ライブ配信アプリのアイテム再生をMetalで実装する事になった話

noppefoxwolf

September 02, 2018
Tweet

More Decks by noppefoxwolf

Other Decks in Technology

Transcript

  1. ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ
    .FUBMͰ࣮૷͢Δࣄʹͳͬͨ࿩
    J04%$+BQBO5SBDL#
    JPTEDC TIBSF

    View Slide

  2. OPQQF
    !
    גࣜձࣾσΟʔɾΤψɾΤʔ
    "
    ͖ͭͶ͔Θ͍͍
    #
    ΞϓϦ։ൃ̎̌̍̌ʙ
    JPTEDC TIBSF

    View Slide

  3. ࠓ೔ͷ࿩
    ɾ1PDPDIBͷ঺հ
    ɾΞΠςϜΤϑΣΫτͷ࠶ੜख๏
    ɾ࠶ੜख๏ͷ࣮૷ൺֱ
    ɾ·ͱΊ
    JPTEDC TIBSF

    View Slide

  4. ϥΠϒ഑৴ΞϓϦ1PDPDIBͷ঺հ
    JPTEDC TIBSF

    View Slide

  5. 1PDPDIB
    w ϥΠϒ഑৴ΞϓϦ
    w ʙ
    w ߴੑೳͳඒഽɾඒإϑΟϧλʔ
    w ΞΠςϜʹΑΔίϛϡχέʔ
    γϣϯ
    JPTEDC TIBSF

    View Slide

  6. ΞΠςϜͷΤϑΣΫτͷ࠶ੜख๏
    JPTEDC TIBSF

    View Slide

  7. JPTEDC TIBSF

    View Slide

  8. ΞΠςϜͷΤϑΣΫτ
    w  º 
    w GQT
    w ಁա৭͋Γ
    w ඵ
    JPTEDC TIBSF

    View Slide

  9. ΞΠςϜͷΤϑΣΫτΛJ04্Ͱ࠶ੜ͢Δʹ͸
    JPTEDC TIBSF

    View Slide

  10. ᶃ"71MBZFSͰ࠶ੜ͢Δ
    let url = Bundle.main.url(forResource: "item_effect",
    withExtension: "mov")!
    let asset = AVURLAsset(url: url)
    let item = AVPlayerItem(asset: asset)
    let player = AVPlayer(playerItem: item)
    let playerLayer = AVPlayerLayer(player: player)
    playerLayer.backgroundColor = UIColor.clear.cgColor
    view.layer.addSubLayer(playerLayer)
    player.play()
    JPTEDC TIBSF

    View Slide

  11. "71MBZFS-BZFSͰ͸ɺಁա৘ใͷ
    ͋ΔಈըΛѻ͏ࣄ͕Ͱ͖ͳ͍ɻ
    *NQPSUBOU
    5IFWBMVFPGBQMBZFSMBZFS`T
    JOIFSJUFEDPOUFOUTQSPQFSUZJT
    PQBRVFBOETIPVMEOPUCF
    VTFE "71-BZFS-BZFSυΩϡϝ
    ϯτΑΓ

    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO
    BWGPVOEBUJPOBWQMBZFSMBZFS
    JPTEDC TIBSF

    View Slide

  12. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    let imageView = UIImageView()
    let images = (0...300).compactMap({
    UIImage(named: "\($0).png")
    })
    imageView.animationImages = images
    imageView.startAnimating()
    JPTEDC TIBSF

    View Slide

  13. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    $ ffmpeg -i input.mov images/output_%04d.png
    JPTEDC TIBSF

    View Slide

  14. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    $ ffmpeg -i input.mov images/output_%04d.png
    $ du -sh ./images
    302M ./images

    JPTEDC TIBSF

    View Slide

  15. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    JPTEDC TIBSF

    View Slide

  16. ᶄ6**NBHF7JFXͰ࠶ੜ͢Δ
    Message from debugger: Terminated due to memory
    issue
    ͰΫϥογϡ
    JPTEDC TIBSF

    View Slide

  17. ݕূ͔Β෼͔ͬͨ̎ͭͷ՝୊ײ
    w ಈըࣗମͷ༰ྔ
    w ࠶ੜύϑΥʔϚϯε
    JPTEDC TIBSF

    View Slide

  18. ಈըࣗମͷ༰ྔ
    ඇѹॖNPWϑΝΠϧ .#
    ࿈൪1/(ϑΝΠϧ .#
    ѹॖ͢Δඞཁ͕͋Δ
    JPTEDC TIBSF

    View Slide

  19. ϑΝΠϧͷѹॖͰඞਢͳཁ݅
    w ಁա৘ใΛ͍࣋ͬͯΔ
    w GQTΛམͱ͞ͳ͍
    w ॎԣͷαΠζΛมߋ͠ͳ͍
    JPTEDC TIBSF

    View Slide

  20. ϑΥʔϚοτ ಁա ѹॖ཰ σίʔυෛՙ
    BQOH ˓ .# º
    XFCQ ˓ .# º TG

    XFCN ˓ ,# º
    NQ º .# ˓ I

    "1/(͸༰ྔͷ࣌఺Ͱෆ࠾༻ͩͬͨͷͰະଌఆɻ8.͸σίʔμ͕ແͦ͞͏ͳͷͰະଌఆ
    WQXJUIBMQIB
    JPTEDC TIBSF

    View Slide

  21. NQΛͳΜͱ͔ಁաʹରԠग़དྷͳ͍͔
    ͜͜ͰɺੲήʔϜΛ࡞͍ͬͯͨ࣌ͷ͜ͱΛࢥ͍ग़͢
    JPTEDC TIBSF

    View Slide

  22. JPTEDC TIBSF

    View Slide

  23. JPTEDC TIBSF

    View Slide

  24. ಁա߹੒ͱඳը
    ͔͜͜Β͸ɺಁաಈըϓϨʔϠʔΛ࣮૷͍ͯ͘͠࿩
    w $PSF*NBHF
    w 0QFO(-&4
    w .FUBM
    ΛͦΕͧΕ࢖ͬͯ࡞ͬͨ݁ՌΛൺֱ͠·͢ɻ
    JPTEDC TIBSF

    View Slide

  25. ಁա߹੒ͱඳը
    w  º 
    w GQT
    w ಁա৭ͳ͠
    w ඵ
    w NQ
    JPTEDC TIBSF

    View Slide

  26. $*'JMUFSΛ࢖ͬͨ߹੒
    $*'JMUFS
    J04
    Ҏ্ͷϓϦηοτϑΟϧλ
    $16͔(16Ͱॲཧ
    JPTEDC TIBSF

    View Slide

  27. $*'JMUFSΛ࢖ͬͨ߹੒
    $*'JMUFS
    J04
    Ҏ্ͷϓϦηοτϑΟϧλ
    $16͔(16Ͱॲཧ
    $*#MFOE8JUI.BTL
    ΞϧϑΝ஋ͷը૾Λ࢖ͬͯಁա
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO
    DPSFJNBHFDJpMUFS
    JPTEDC TIBSF

    View Slide

  28. $*'JMUFSΛ࢖ͬͨ߹੒
    let filter = CIFilter(name: "CIBlendWithMask")!
    filter.setValue(baseCIImage,
    forKey: kCIInputImageKey)
    filter.setValue(alphaCIImage,
    forKey: kCIInputMaskImageKey)
    let filteredImage = filter.outputImage!
    let image = UIImage(ciImage: filteredImage)
    imageView.image = image
    JPTEDC TIBSF

    View Slide

  29. J1IPOFDGQT
    JPTEDC TIBSF

    View Slide

  30. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    $"&"(--BZFS
    J04r
    (16Ͱॲཧ
    JPTEDC TIBSF

    View Slide

  31. 0QFO(-&45*14
    JPTEDC TIBSF

    View Slide

  32. 0QFO(-&45*14
    &"(-$POUFYU
    ςΫενϟ΍Τϥʔঢ়ଶͳͲΛ؅ཧ
    JPTEDC TIBSF

    View Slide

  33. 0QFO(-&45*14
    // ͲͷίϯςΩετͰಈ࡞͢Δ͔ෆఆ
    glDisable(GLenum(GL_DEPTH_TEST))
    // ར༻͢ΔcontextΛઃఆ
    EAGLContext.setCurrent(context)
    // ↑Ͱࢦఆͨ͠contextͰ࣮ߦ͞ΕΔ
    glDisable(GLenum(GL_DEPTH_TEST))
    JPTEDC TIBSF

    View Slide

  34. 0QFO(-&45*14
    glGetError()
    ྫɿGL_INVALID_ENUM
    ແޮͳFOVN஋Λࢦఆ͍ͯ͠Δɻ
    JPTEDC TIBSF

    View Slide

  35. 0QFO(-&45*14
    9DPEFͰ͸ϒϨʔΫϙΠϯτΛ࢓ࠐΉ͜ͱ΋Մ
    JPTEDC TIBSF

    View Slide

  36. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    $PSF*NBHF 0QFO(-&4
    $**NBHF ςΫενϟ
    $*'JMUFS ϑϥάϝϯτγΣʔμ
    6**NBHF7JFX Ϩϯμϥ $"&"(--BZFS

    JPTEDC TIBSF

    View Slide

  37. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    varying lowp vec2 textureCoordinate;
    uniform sampler2D videoFrame;
    void main() {
    // ϕʔεͷ৭ͷx
    lowp float x = textureCoordinate.x / 2.0;
    // ϕʔε৭ͷऔಘ
    lowp vec2 baseCoord = vec2(x, textureCoordinate.y);
    lowp vec4 baseColor = texture2D(videoFrame, baseCoord);
    // alpha஋ͷऔಘ
    lowp vec2 alphaCoord = vec2(x + 0.5, textureCoordinate.y);
    lowp vec4 alphaColor = texture2D(videoFrame, alphaCoord);
    // RGB͸ϕʔε৭ɺA͸alpha஋Λར༻͢Δ
    gl_FragColor = vec4(baseColor.r, baseColor.g, baseColor.b, alphaColor.r);
    }
    JPTEDC TIBSF

    View Slide

  38. HM@'SBH$PMPSWFD CBTF$PMPSS CBTF$PMPSH CBTF$PMPSC BMQIB$PMPSS

    JPTEDC TIBSF

    View Slide

  39. 0QFO(-&4Λ࢖ͬͨ߹੒ͱඳը
    J1IPOFD㲈GQT
    JPTEDC TIBSF

    View Slide

  40. ͔͠͠ʜ
    JPTEDC TIBSF

    View Slide

  41. JPTEDC TIBSF

    View Slide

  42. 88%$Ͱಥવͷ%FQSFDBUFEɻ
    ϥΠϒ഑৴ΞϓϦͷΞΠςϜ࠶ੜΛ.FUBMͰ࣮૷͢Δ
    ࣄʹͳͬͨ࿩
    ແࣄλΠτϧճऩ

    JPTEDC TIBSF

    View Slide

  43. .FUBMͰॻ͍ͯΈ·ͨ͠
    JPTEDC TIBSF

    View Slide

  44. .FUBMͷར఺
    w γΣʔμʔͷࣄલίϯύΠϧ
    w (16$16ͷڞ༗ϝϞϦۭؒ
    w (16΁ͷ໋ྩΛ·ͱΊͯૹΕΔ
    FUD
    %ΛΰϦΰϦʹ࢖͏ਓ͘Β͍͔͠ت͹ͳ͍ʜ
    JPTEDC TIBSF

    View Slide

  45. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    (16ϓϩάϥϛϯάॳ৺ऀʹ༏͍͠.FUBM
    w (16Λ৮Δ্Ͱॻ͖΍͍͢*'
    w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ
    JPTEDC TIBSF

    View Slide

  46. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    let desc = MTLRenderPipelineDescriptor()
    desc.vertexFunction = library.makeFunction(name: vertexFunctionName)
    desc.fragmentFunction = library.makeFunction(name: fragmentFunctionName)
    desc.colorAttachments[0].pixelFormat = pixelFormat
    4XJGU0CK$Ͱ΋ॻ͖΍͍͢Ϟμϯͳ*'
    ίϯύΠϧ࣌ʹ໰୊͕ൃ֮͠΍͍͢ɻ
    JPTEDC TIBSF

    View Slide

  47. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    γΣʔμʔͷΤϥʔ΋ίϯύΠϧ࣌ʹڭ͑ͯ͘ΕΔ
    JPTEDC TIBSF

    View Slide

  48. .FUBMΛ࢖ͬͯΈͯײͨ͡ར఺
    w (16Λ৮Δ্Ͱॻ͖΍͍͢*'
    w ཧղΛਂΊΔ্Ͱ΋ॏཁͳσόοΨʔ
    σόοΨʔʹ͍ͭͯ͸͜ͷޙ঺հ͠·͢ɻ
    JPTEDC TIBSF

    View Slide

  49. .FUBMͰॻ͍ͯΈ·ͨ͠
    ͜ΕΒͷར఺Λ౿·͑ͯɺ.FUBMͰ
    ॻ͍ͯΈ·ͨ͠ɻ
    అ͞Μͷ.FUBMೖ໳͕ಡΈ΍͘͢
    ͓ͯ͢͢ΊͰ͢ɻ
    JPTEDC TIBSF

    View Slide

  50. J1IPOFT㲈GQT
    JPTEDC TIBSF

    View Slide

  51. .FUBMͰॻ͍ͯΈ·ͨ͠
    (16'SBNF$BQUVSF
    w ΧϝϥϚʔΫͷϘλϯ
    JPTEDC TIBSF

    View Slide

  52. JPTEDC TIBSF

    View Slide

  53. ֤ςΫενϟΛQOH΍UJ⒎Ͱॻ͖ग़͢ࣄ΋Ͱ͖·͢ɻ
    JPTEDC TIBSF

    View Slide

  54. JPTEDC TIBSF

    View Slide

  55. 0QFO(-&4൛ͷ(16'SBNF$BQUSVF΍*OTUSVNFOUT
    ͸ɺࣗ෼ͷ؀ڥͰ͸ಈ͔ͳ͘ͳ͍ͬͯ·ͨ͠
    JPTEDC TIBSF

    View Slide

  56. .FUBMͷ*OTUSVNFOUT͸౰વར༻Ͱ͖·͢ɻ
    JPTEDC TIBSF

    View Slide

  57. ·ͱΊ
    JPTEDC TIBSF

    View Slide

  58. ΞϓϦΤϯδχΞ΋.FUBM͸৮Δ΂͖͔
    w ࢥͬͨΑΓ΋೉͘͠ແ͘ɺཧղ͠΍͍͢ੈք
    w "3,JU΍$PSF.-ʹܨ͕Δٕज़
    JPTEDC TIBSF

    View Slide

  59. .FUBMͰ࣮૷͢Δඞཁ͸͋Δͷ͔
    w ը૾ฤूͳΒ͹$*'JMUFSΛ·ͣ͸ݕ౼
    w .FUBMͷσϝϦοτΛཧղ্ͨ͠Ͱ࠾༻͢Δ΂͖
    JPTEDC TIBSF

    View Slide

  60. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺
    w J04ະຬͷσόΠεʹ͸.FUBMඇରԠͷ୺຤͕͋Γ
    ·͢
    w γϛϡϨʔλ͸.FUBMඇରԠͰ͢
    MTLCreateSystemDefaultDevice() != nil
    JPTEDC TIBSF

    View Slide

  61. .FUBMͰ࣮૷͢Δ্Ͱͷ஫ҙ఺
    γϛϡϨʔλͰϏϧυͰ͖ͳ͍ࣄ΋͋Γ·͢
    Ұ෦ͷϔϧύʔ͸γϛϡϨʔλ4%,ʹؚ·Ε·ͤΜ
    w $".FUBM%SBXBCMF
    w $".FUBM-BZFS
    IUUQTHJTUHJUIVCDPNOPQQFGPYXPMGGBBBDCCEECG
    JPTEDC TIBSF

    View Slide

  62. 1PDPDIBͰ͸
    .FUBMͰ࣮૷͠·͕ͨ͠ʜ
    w ࠓ͸0QFO(-&4൛Λ࢖͍ͬͯ·͢
    w J04΋αϙʔτ͢ΔͨΊ
    w γϛϡϨʔλ࢖͍ͬͯΔ
    0QFO(-&4͕ഇࢭ͞ΕͨΒͲ͏͢Δͷʁ
    ˠͦͷࠒʹ͸J04΋ରԠ͠ͳ͍ͩΖ͏͠ɺγϛϡϨʔλͰ΋.FUBM͕ಈ͘Α͏ʹ
    ͳ͍ͬͯ·͢ʂʂʢଟ෼Ͷʣ
    JPTEDC TIBSF

    View Slide

  63. 4PVSDF$PEF
    ಁաॲཧΛࢪͯ͠࠶ੜ͢ΔϓϨʔϠʔϥΠϒϥϦΛ࡞͍ͬͯ·͢
    JPTEDC TIBSF

    View Slide

  64. 4PVSDF$PEF
    w OPQQFGPYXPMG,JUTVOFCJ
    w OPQQFGPYXPMG,[email protected]
    !

    "

    !
    JPTEDC TIBSF

    View Slide

  65. !
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    "TLUIFTQFBLFS
    %F/"4QFBLFS-PVOHF
    JPTEDC TIBSF

    View Slide