Scalable Microservice for Media Transcoding / Transcoder

Scalable Microservice for Media Transcoding / Transcoder

Rails Developers Meetup 2019の発表に利用した資料です。

D62ad26d5db87a3a62ae97c77a252c22?s=128

atsushisakai

March 23, 2019
Tweet

Transcript

  1. Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails

    Developers Meetup 2019
  2. ञҪ ಞ @_atsushisakai • Ո଒ΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on

    Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͸͸͡Ίͯ
  3. Agenda ʮΈͯͶʯͰ࢖͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓࿩ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝୊ͩͬͨͷ͔ʁ • Ͳ͏΍ͬͯղܾͨ͠ͷ͔ʁ

  4. ·ͣ͸ͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!

  5. ಈը(WIP)

  6. ͜Ε͸Ұମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲ΋ͷ੒௕ΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞Εͨಈը΍ࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞੒ • ൢച͢ΔDVDʹऩ࿥ͨ͠ΓɺΞϓϦʹ഑৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ෇͚ͯࢠҭͯͷࢥ͍ग़Λ࡞඼ʹ͢Δ •

    ௕͍ಈըΛ੾ͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢ೔෇ͳͲͷը૾Λ͸Γ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
  7. ಈը(WIP)

  8. 1ඵಈըΛࣗಈͰେྔʹ࡞Δ

  9. ͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑͹ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͸͚ͭΕΔ • ಈըͷ݁߹΋Ͱ͖Δ

    • Ͱ͖ͳ͍͜ͱ • ෼ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Ի੠ͱͷ߹੒ͱ͍͏ͷ΋Ͱ͖ͳ͍
  10. ࡞Δ͔͠ͳ͍…

  11. Ͳ͏΍ͬͯʁ ੾Γग़͠΍ ϦαΠζͳͲͷ લॲཧ ෳ਺ͷಈըΛ ݁߹ Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ

  12. Ͳ͏΍ͬͯʁ ੾Γग़͠΍ ϦαΠζͳͲͷ લॲཧ ෳ਺ͷಈըΛ ݁߹ Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%

  13. ͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ͸ • ffmpegʹΑΔม׵ॲཧΛ௚ྻʹ࿈݁ͨ͠΋ͷ • લͷॲཧͷΞ΢τϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ

  14. Έͳ͞ΜͳΒͲ͏࣮૷͠·͔͢❓

  15. ࠷΋۪௚ͳ࣮૷

  16. ࠷΋۪௚ͳ࣮૷ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ໰୊͕ग़ͯ͘Δ • ௚ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ΍͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ഑৴ର৅ͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷ໰୊΋ग़ͯ͘Δ

    • CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͸΋͍ͬͨͳ͍ • ޮ཰తͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
  17. ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮૷

  18. ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮૷ ύΠϓϥΠϯΛಉ࣌ʹෳ਺૸ΒͤΔͱɺ͖ͬͱύϑΥʔϚϯε͸ྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌ೉ͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͸͋Δఔ౓Α͘ͳΔ͔΋͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧ΋೉ͦ͠͏

  19. None
  20. Sidekiq Ruby੡ͷ Job Queue γεςϜͰͱͯ΋༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ͸100୆Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ଎

    • https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯ΋গͳ͍ • ๛෋ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
  21. Sidekiq

  22. Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ

  23. Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ

  24. Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔ܎Λ࡞Δ • ෳ਺ͷ

    Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠΋ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
  25. None
  26. ΋ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛ଴ͬͯBʙF͕ىಈ • BʙF͸ύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓΍ɺ͜Ε͸…ʁʁ

    https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
  27. Ͳ͏΍ͬͯ࡞Δʁ ੾Γग़͠΍ ϦαΠζͳͲͷ લॲཧ ෳ਺ͷಈըΛ ݁߹ Իָͱ߹੒ ੒Ռ෺͕׬੒ʂ Sidekiq BatchΛ࢖͏͜ͱͰ

    ໨తͷύΠϓϥΠϯ͕࣮૷Ͱ͖ͦ͏‼
  28. ύΠϓϥΠϯ࣮૷Ͱ͖Δ͔΋ʁ ׂͱཧ૝ʹ͍ۙ☺ • جຊతʹ͸ͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά΋؆୯ • ޮ཰తͳϦτϥΠ͕ߦΘΕΔ

    • Sidekiqͷฒྻ౓ઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴ଎Խ͍ͯ͘͠ • CPU͕଍Γͳ͚Ε͹εέʔϧͤ͞Δ
  29. Transcoder

  30. ϚΠΫϩαʔϏεͱͯ͠ͷ࣮૷ • ผαʔϏε͔ΒTranscoder΁ͷϦΫΤετ͸AWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁Ռ΋AWS SQSʹ৐ͤͯฦ͢ •

    தؒϑΝΠϧ΍੒Ռ෺͸AWS S3ʹอଘ͢Δ • αʔϏεؒ͸ΦϒδΣΫτ໊ͷ΍ΓͱΓͷΈ • ࣮ϑΝΠϧ͸࢖͏ଆ͕औΓʹߦ͘
  31. αʔϏεͷεέʔϥϏϦςΟ • ffmpeg͸ϞϦϞϦͱCPUΛ࢖͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ

  32. Transcoder γεςϜߏ੒ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷ΍ΓऔΓͷՕॴͰখن ໛ʹRails͕ಈ͘γεςϜ

  33. Transcoder γεςϜߏ੒ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔε͸SQSΛ༻͍Δ 1 2 3 4 5

    6
  34. Transcoder γεςϜߏ੒ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔε͸SQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2

    3 4 5 6
  35. Transcoder γεςϜߏ੒ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔε͸SQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ

    ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
  36. Transcoder γεςϜߏ੒ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔε͸SQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ

    ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ಺༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
  37. Transcoder γεςϜߏ੒ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2

    3 4 5 6
  38. Transcoder γεςϜߏ੒ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜ੒෺ɾ࠷ऴ੒Ռ෺Λอ

    ଘ͢ΔS3όέοτ 1 2 3 4 5 6
  39. Transcoder ͷΞʔΩςΫνϟ JobͱResult • Job͸ɺSQSʹ৐ͤΔJSONͷ͜ͱ • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoder͸ඇಉظʹॲཧΛ։࢝͢Δ •

    Result͸ɺSQSʹ৐ͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ৔߹ʹ͸ErrorResultΛฦ͢
  40. None
  41. None
  42. None
  43. Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮૷ • = Sidekiq Batchͷ࣮૷ͦͷ΋ͷɻ͜Εࣗମ΋Sidekiq Worker

    • ಺෦ʹෳ਺ͷStepΛ࣋ͭ • Pipeline͝ͱʹ੒Ռ෺͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ഑৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
  44. Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳ਺ͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •

    Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
  45. Transcoder ͷΞʔΩςΫνϟ Worker • Step಺ͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͢΂ͯSidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊ͸ffmpegͷίϚϯυΛҰճ࣮ߦ͢Δཻ౓Ͱ࣮૷͢Δ

    • ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧ͸S3ʹอଘ͢Δ
  46. Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠΋ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ

  47. Pipeline Step Step Step Worker + Command

  48. Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ

  49. Sidekiq Batch࣮૷ͷίπ • Batch಺෦ͷWorker͸ґଘ౓Λ௿͘ɺཻ͔ͭ౓Λখ࣮͘͞૷͢Δɻ • εέʔϧ͠΍͍͢ɺϦτϥΠ͠΍͍͢ɺͳͲͷ໨తɻ • ෳࡶͳBatchΛ࡞Δ৔߹ɺPipelineΛදݱ͢Δίʔυ͸ͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueue͸ͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ

    • Callbackॲཧ΋࣮ମ͸Sidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴ଎ͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
  50. ͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըม׵ύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠͹ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺ೚ҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷ੒Ռ෺Λग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ੒

    • Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ੒͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮૷͕ංେԽͯ͠΋ؾʹͳΒͳ͍
  51. ޻෉ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Ε͹ʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏΍ͬͯ΍Δ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ

  52. None
  53. ޻෉ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Ε͹ʁʣ • ffmpegίϚϯυͷೖग़ྗ࢓༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳ਺ͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠΋ೖྗͷॱং͸ॏཁ

    • CommandͷجఈΫϥεΛ࡞ͬͯந৅Խ • Pipeline → Step → Worker ΁ͱɺෳ਺ͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ౉͢
  54. None
  55. ·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ࢖͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiq͸ͱͯ΋ྑ͍ •

    Sidekiq Pro͸΋ͬͱྑ͍ • Sidekiq BatchΛ࢖͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹ΋ɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • ந৅తͳ࿩͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉ͸Sidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
  56. ·ͱΊ

  57. None
  58. THANK YOU!!