Scalable Microservice for Media Transcoding / Transcoder

Rails Developers Meetup 2019の発表に利用した資料です。

Transcript

1. Scalable Microservice for Media Transcoding
   ञҪ ಞ @_atsushisakai
   גࣜձࣾϛΫγΟ
   Rails Developers Meetup 2019
   

   View Slide

2. ञҪ ಞ @_atsushisakai
   • Ո଒ΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢
   • SRE
   • Ruby on Rails
   • AWS
   • iOS/Android
   • Engineering Manager
   • iOSDC/DroidKaigi
   • RailsDM͸͸͡Ίͯ
   

   View Slide

3. Agenda
   ʮΈͯͶʯͰ࢖͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓࿩ɻ
   • ͳͥ࡞ͬͨͷ͔ʁ
   • ղܾ͍ͨ͠ཁ݅ɾ՝୊ͩͬͨͷ͔ʁ
   • Ͳ͏΍ͬͯղܾͨ͠ͷ͔ʁ
   

   View Slide

4. ·ͣ͸ͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
   

   View Slide

5. ಈը(WIP)
   

   View Slide

6. ͜Ε͸Ұମ…ʁ
   • ʮ1ඵಈըʯͱ͍͏ࢠͲ΋ͷ੒௕ΛৼΓฦΔػೳ
   • େྔʹΞοϓϩʔυ͞Εͨಈը΍ࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞੒
   • ൢച͢ΔDVDʹऩ࿥ͨ͠ΓɺΞϓϦʹ഑৴ͨ͠Γ͢Δ
   • ଟ༷ͳޮՌΛ෇͚ͯࢠҭͯͷࢥ͍ग़Λ࡞඼ʹ͢Δ
   • ௕͍ಈըΛ੾ͬͯɺผͷಈըͱ݁߹͢Δ
   • WatermarkΛ͚ͭΔʢ೔෇ͳͲͷը૾Λ͸Γ͚ͭΔʣ
   • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ
   • BGMΛ͚ͭΔ
   

   View Slide

7. ಈը(WIP)
   

   View Slide

8. 1ඵಈըΛࣗಈͰେྔʹ࡞Δ
   

   View Slide

9. ͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ
   • ྫ͑͹ɺAWS ElasticTranscoder
   • Ͱ͖Δ͜ͱ
   • Watermark͸͚ͭΕΔ
   • ಈըͷ݁߹΋Ͱ͖Δ
   • Ͱ͖ͳ͍͜ͱ
   • ෼ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍
   • Ի੠ͱͷ߹੒ͱ͍͏ͷ΋Ͱ͖ͳ͍
   

   View Slide

10. ࡞Δ͔͠ͳ͍…
    

    View Slide

11. Ͳ͏΍ͬͯʁ
    ੾Γग़͠΍
    ϦαΠζͳͲͷ
    લॲཧ
    ෳ਺ͷಈըΛ
    ݁߹
    Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ
    

    View Slide

12. Ͳ͏΍ͬͯʁ
    ੾Γग़͠΍
    ϦαΠζͳͲͷ
    લॲཧ
    ෳ਺ͷಈըΛ
    ݁߹
    Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ
    ͍ΘΏΔύΠϓϥΠϯॲཧ%
    

    View Slide

13. ͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ͸
    • ffmpegʹΑΔม׵ॲཧΛ௚ྻʹ࿈݁ͨ͠΋ͷ
    • લͷॲཧͷΞ΢τϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
    

    View Slide

14. Έͳ͞ΜͳΒͲ͏࣮૷͠·͔͢❓
    

    View Slide

15. ࠷΋۪௚ͳ࣮૷
    

    View Slide

16. ࠷΋۪௚ͳ࣮૷
    ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘
    • ύϑΥʔϚϯεͷ໰୊͕ग़ͯ͘Δ
    • ௚ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ΍͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ
    • ٸʹ഑৴ର৅ͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ
    • ίετͷ໰୊΋ग़ͯ͘Δ
    • CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͸΋͍ͬͨͳ͍
    • ޮ཰తͳϦτϥΠ͕Ͱ͖ͳ͍
    • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
    

    View Slide

17. ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮૷
    

    View Slide

18. ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮૷
    ύΠϓϥΠϯΛಉ࣌ʹෳ਺૸ΒͤΔͱɺ͖ͬͱύϑΥʔϚϯε͸ྑ͘ͳΔ
    • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌ೉ͦ͠͏
    • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢
    • ύϑΥʔϚϯε͸͋Δఔ౓Α͘ͳΔ͔΋͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏
    • Ҿ͖ଓ͖ϦτϥΠॲཧ΋೉ͦ͠͏
    

    View Slide

19. View Slide

20. Sidekiq
    Ruby੡ͷ Job Queue γεςϜͰͱͯ΋༗໊ͳgemɻΦεεϝɻ
    ʮΈͯͶʯͰ͸100୆Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ
    • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍
    • Resque/DelayedJobͷ20ഒۙ͘ߴ଎
    • https://github.com/mperham/sidekiq#performance
    • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ
    • ଞͷGemґଘ͕ͱͯ΋গͳ͍
    • ๛෋ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ
    • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
    

    View Slide

21. Sidekiq
    

    View Slide

22. Sidekiq Pro
    Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ
    https://sidekiq.org/products/pro.html
    • ϝʔϧαϙʔτΛड͚ΒΕΔ
    • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ
    • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
    

    View Slide

23. Sidekiq Pro
    Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ
    https://sidekiq.org/products/pro.html
    • ϝʔϧαϙʔτΛड͚ΒΕΔ
    • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ
    • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
    

    View Slide

24. Sidekiq Batch
    • Sidekiq Pro ݶఆͷػೳ
    • ୯७ͳWorkerͷॲཧʹґଘؔ܎Λ࡞Δ
    • ෳ਺ͷ Worker ΛଋͶͯɺͦͷ Batch
    ͝ͱʹॲཧΛਐΊ͍ͯ͘
    • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback
    ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏
    ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ
    • ͨͱ͑ D ͕ࣦഊͯ͠΋ D ͚ͩϦτϥΠ
    ͯ͘͠ΕΔ
    A B C
    D
    

    View Slide

25. View Slide

26. ΋ͬͱෳࡶͳBatchॲཧ
    • A͕ऴΘΔͷΛ଴ͬͯBʙF͕ىಈ
    • BʙF͸ύϥϨϧʹىಈ͢Δ
    • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ
    • …
    ͓΍ɺ͜Ε͸…ʁʁ
    https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
    

    View Slide

27. Ͳ͏΍ͬͯ࡞Δʁ
    ੾Γग़͠΍
    ϦαΠζͳͲͷ
    લॲཧ
    ෳ਺ͷಈըΛ
    ݁߹
    Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ
    Sidekiq BatchΛ࢖͏͜ͱͰ
    ໨తͷύΠϓϥΠϯ͕࣮૷Ͱ͖ͦ͏‼
    

    View Slide

28. ύΠϓϥΠϯ࣮૷Ͱ͖Δ͔΋ʁ
    ׂͱཧ૝ʹ͍ۙ☺
    • جຊతʹ͸ͨͩͷSidekiq Workerͳͷ
    Ͱ͓खܰʹฒྻॲཧͰ͖Δ
    • ϓϩάϥϛϯά΋؆୯
    • ޮ཰తͳϦτϥΠ͕ߦΘΕΔ
    • Sidekiqͷฒྻ౓ઃఆΛ্͛ΔͱCPU͕
    ༨͍ͬͯΔݶΓߴ଎Խ͍ͯ͘͠
    • CPU͕଍Γͳ͚Ε͹εέʔϧͤ͞Δ
    

    View Slide

29. Transcoder
    

    View Slide

30. ϚΠΫϩαʔϏεͱͯ͠ͷ࣮૷
    • ผαʔϏε͔ΒTranscoder΁ͷϦΫΤετ͸AWS SQSΛܦ༝͢Δɻ
    • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ
    • ࣮ߦ݁Ռ΋AWS SQSʹ৐ͤͯฦ͢
    • தؒϑΝΠϧ΍੒Ռ෺͸AWS S3ʹอଘ͢Δ
    • αʔϏεؒ͸ΦϒδΣΫτ໊ͷ΍ΓͱΓͷΈ
    • ࣮ϑΝΠϧ͸࢖͏ଆ͕औΓʹߦ͘
    

    View Slide

31. αʔϏεͷεέʔϥϏϦςΟ
    • ffmpeg͸ϞϦϞϦͱCPUΛ࢖͏
    • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ
    ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale
    ػೳͰ༰қʹରԠՄೳʣ
    

    View Slide

32. Transcoder γεςϜߏ੒
    • શͯAWSͷαʔϏεΛར༻͠
    ͨઃܭ
    • DBͱͷ΍ΓऔΓͷՕॴͰখن
    ໛ʹRails͕ಈ͘γεςϜ
    

    View Slide

33. Transcoder γεςϜߏ੒
    1. ଞͷαʔϏεͱͷೖग़ྗΠϯ
    λʔϑΣʔε͸SQSΛ༻͍Δ
    1
    2
    3
    4 5
    6
    

    View Slide

34. Transcoder γεςϜߏ੒
    1. ଞͷαʔϏεͱͷೖग़ྗΠϯ
    λʔϑΣʔε͸SQSΛ༻͍Δ
    2. SQSΛPolling͢ΔΠϯελϯ
    ε
    1
    2
    3
    4 5
    6
    

    View Slide

35. Transcoder γεςϜߏ੒
    1. ଞͷαʔϏεͱͷೖग़ྗΠϯ
    λʔϑΣʔε͸SQSΛ༻͍Δ
    2. SQSΛPolling͢ΔΠϯελϯ
    ε
    3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
    ΛSidekiq BatchͰॲཧ͢Δ
    1
    2
    3
    4 5
    6
    

    View Slide

36. Transcoder γεςϜߏ੒
    1. ଞͷαʔϏεͱͷೖग़ྗΠϯ
    λʔϑΣʔε͸SQSΛ༻͍Δ
    2. SQSΛPolling͢ΔΠϯελϯ
    ε
    3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
    ΛSidekiq BatchͰॲཧ͢Δ
    4. RDSʹґཔ͞Εͨ಺༰Λอ
    ࣋ɾࢀর͢Δ
    ʢActiveRecordʣ
    1
    2
    3
    4 5
    6
    

    View Slide

37. Transcoder γεςϜߏ੒
    5. Sidekiq ͷ Job ΛӬଓԽ͢Δ
    όοΫΤϯυͱͯ͠ͷRedis
    1
    2
    3
    4 5
    6
    

    View Slide

38. Transcoder γεςϜߏ੒
    5. Sidekiq ͷ Job ΛӬଓԽ͢Δ
    όοΫΤϯυͱͯ͠ͷRedis
    6. தؒੜ੒෺ɾ࠷ऴ੒Ռ෺Λอ
    ଘ͢ΔS3όέοτ
    1
    2
    3
    4 5
    6
    

    View Slide

39. Transcoder ͷΞʔΩςΫνϟ
    JobͱResult
    • Job͸ɺSQSʹ৐ͤΔJSONͷ͜ͱ
    • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ
    • SQSΛPolling͠ɺTranscoder͸ඇಉظʹॲཧΛ։࢝͢Δ
    • Result͸ɺSQSʹ৐ͤΔॲཧ݁ՌͷJSONͷ͜ͱ
    • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ
    • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ
    • ݁Ռ͕Errorͷ৔߹ʹ͸ErrorResultΛฦ͢
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. Transcoder ͷΞʔΩςΫνϟ
    Pipeline
    • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮૷
    • = Sidekiq Batchͷ࣮૷ͦͷ΋ͷɻ͜Εࣗମ΋Sidekiq Worker
    • ಺෦ʹෳ਺ͷStepΛ࣋ͭ
    • Pipeline͝ͱʹ੒Ռ෺͕ఆٛ͞Ε͍ͯΔ
    • ʮΞϓϦ഑৴ʯ༻ͷಈըΛͭ͘ΔPipeline
    • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
    

    View Slide

44. Transcoder ͷΞʔΩςΫνϟ
    Step
    • Pipeline͕ఆٛͨ͠ෳ਺ͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ
    • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ
    • Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
    

    View Slide

45. Transcoder ͷΞʔΩςΫνϟ
    Worker
    • Step಺ͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ
    • ͢΂ͯSidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ
    • جຊ͸ffmpegͷίϚϯυΛҰճ࣮ߦ͢Δཻ౓Ͱ࣮૷͢Δ
    • ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧ͸S3ʹอଘ͢Δ
    

    View Slide

46. Transcoder ͷΞʔΩςΫνϟ
    Command
    • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠΋ͷ
    • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
    

    View Slide

47. Pipeline
    Step Step Step
    Worker
    + Command
    

    View Slide

48. Pipeline
    Step Step Step
    Worker
    + Command
    Sidekiq BatchͰղܾ
    

    View Slide

49. Sidekiq Batch࣮૷ͷίπ
    • Batch಺෦ͷWorker͸ґଘ౓Λ௿͘ɺཻ͔ͭ౓Λখ࣮͘͞૷͢Δɻ
    • εέʔϧ͠΍͍͢ɺϦτϥΠ͠΍͍͢ɺͳͲͷ໨తɻ
    • ෳࡶͳBatchΛ࡞Δ৔߹ɺPipelineΛදݱ͢Δίʔυ͸ͲΜͲΜංେԽ͢Δ
    • Callbackͷ࣮ߦQueue͸ͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
    • Callbackॲཧ΋࣮ମ͸Sidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ
    • େྔʹߴ଎ͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
    

    View Slide

50. ͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ
    • ಈըม׵ύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ
    • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠͹ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺ೚ҙͷύϥ
    ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷ੒Ռ෺Λग़ͤΔΑ͏ʹ͔ͨͬͨ͠
    • TV༻ɺεϚϗ༻ɺPC༻ͱ͔…
    • Pipelineͷࣗಈੜ੒
    • Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ
    Λࣗಈੜ੒͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠
    • Pipelineͷ࣮૷͕ංେԽͯ͠΋ؾʹͳΒͳ͍
    

    View Slide

51. ޻෉ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Ε͹ʁʣ
    • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏΍ͬͯ΍Δ͔ʁ
    • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
    

    View Slide

52. View Slide

53. ޻෉ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Ε͹ʁʣ
    • ffmpegίϚϯυͷೖग़ྗ࢓༷͕ಛघ͗͢Δ
    • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ
    • ෳ਺ͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ
    • ͔͠΋ೖྗͷॱং͸ॏཁ
    • CommandͷجఈΫϥεΛ࡞ͬͯந৅Խ
    • Pipeline → Step → Worker ΁ͱɺෳ਺ͷInputΛॱংΛอͪͳ͕Β͍͍ײ
    ͡ʹCommandʹ౉͢
    

    View Slide

54. View Slide

55. ·ͱΊ
    • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ࢖͍͍ͬͯײ͡
    ʹ࡞ͬͨ
    • Sidekiq͸ͱͯ΋ྑ͍
    • Sidekiq Pro͸΋ͬͱྑ͍
    • Sidekiq BatchΛ࢖͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ
    • ಈըॲཧҎ֎ʹ΋ɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔…
    • ந৅తͳ࿩͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉ͸Sidekiq BatchΛੋඇ͓ࢼͩ͘͠
    ͍͞ʢ$950/yearʣ
    

    View Slide

56. ·ͱΊ
    

    View Slide

57. View Slide

58. THANK YOU!!
    

    View Slide