Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalable Microservice for Media Transcoding / T...
Search
atsushisakai
March 23, 2019
Programming
4
1.5k
Scalable Microservice for Media Transcoding / Transcoder
Rails Developers Meetup 2019の発表に利用した資料です。
atsushisakai
March 23, 2019
Tweet
Share
More Decks by atsushisakai
See All by atsushisakai
大規模 SaaS の技術的意思決定を支える三要素 / Three elements that support technical decision-making for large-scale SaaS
_atsushisakai
0
850
一方、私はなぜEMでありつづけるか / TECH TRACK vol.1
_atsushisakai
0
400
子育てとEMと転職と
_atsushisakai
2
730
「家族アルバム みてね」を支える エンジニアリング組織/introduce-familyalbum-engineering-team
_atsushisakai
2
210
ミクシィのマネージャーは悩んでいる / mixi's manager is in trouble
_atsushisakai
16
14k
実践 Engineering Manager / practice engineering manager
_atsushisakai
19
8k
New Relic {FUTURE} STACK 3/14, 2019 / newrelic_future_stack
_atsushisakai
1
1.5k
実践 WorkManager / WorkManager
_atsushisakai
3
7.3k
How to use New Relic in Mitene Development Team
_atsushisakai
0
560
Other Decks in Programming
See All in Programming
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
130
開発効率向上のためのリファクタリングの一歩目の選択肢 ~コード分割~ / JJUG CCC 2024 Fall
ryounasso
0
280
The future of development – Are our jobs getting harder or easier?
hollycummins
1
240
Nuxt UI Pro、NuxtHub、Nuxt Scripts、Nuxtエコシステムをふんだんに利用して開発するコーポレートサイト@Vue Fes Japan 2024
shingangan
3
810
Piniaの現状と今後
waka292
5
1.3k
Nuxtベースの「WXT」でChrome拡張を作成する | Vue Fes 2024 ランチセッション
moshi1121
1
190
Honoの来た道とこれから
yusukebe
18
2.9k
Going Staff - Keynote edition
pragtob
0
420
Importmapを使ったJavaScriptの 読み込みとブラウザアドオンの影響
swamp09
3
1.2k
Golang と Erlang
taiyow
8
1.8k
讓數據說話:用 Python、Prometheus 和 Grafana 講故事
eddie
0
310
Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines
sucitw
1
170
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
The Pragmatic Product Professional
lauravandoore
31
6.3k
Speed Design
sergeychernyshev
24
560
Building an army of robots
kneath
302
42k
Scaling GitHub
holman
458
140k
What's new in Ruby 2.0
geeforr
342
31k
A Tale of Four Properties
chriscoyier
156
23k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Producing Creativity
orderedlist
PRO
341
39k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Transcript
Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails
Developers Meetup 2019
ञҪ ಞ @_atsushisakai • ՈΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on
Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͡Ίͯ
Agenda ʮΈͯͶʯͰ͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝ͩͬͨͷ͔ʁ • Ͳ͏ͬͯղܾͨ͠ͷ͔ʁ
·ͣͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
ಈը(WIP)
͜ΕҰମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲͷΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞ΕͨಈըࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞ • ൢച͢ΔDVDʹऩͨ͠ΓɺΞϓϦʹ৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ͚ͯࢠҭͯͷࢥ͍ग़Λ࡞ʹ͢Δ •
͍ಈըΛͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢͳͲͷը૾ΛΓ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
ಈը(WIP)
1ඵಈըΛࣗಈͰେྔʹ࡞Δ
͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͚ͭΕΔ • ಈըͷ݁߹Ͱ͖Δ
• Ͱ͖ͳ͍͜ͱ • ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Իͱͷ߹ͱ͍͏ͷͰ͖ͳ͍
࡞Δ͔͠ͳ͍…
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%
͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ • ffmpegʹΑΔมॲཧΛྻʹ࿈݁ͨ͠ͷ • લͷॲཧͷΞτϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
Έͳ͞ΜͳΒͲ͏࣮͠·͔͢❓
࠷۪ͳ࣮
࠷۪ͳ࣮ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ͕ग़ͯ͘Δ • ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ৴ରͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷग़ͯ͘Δ
• CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͍ͬͨͳ͍ • ޮతͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮ ύΠϓϥΠϯΛಉ࣌ʹෳΒͤΔͱɺ͖ͬͱύϑΥʔϚϯεྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͋ΔఔΑ͘ͳΔ͔͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧͦ͠͏
None
Sidekiq Rubyͷ Job Queue γεςϜͰͱͯ༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ100Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ
• https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯগͳ͍ • ๛ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
Sidekiq
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔΛ࡞Δ • ෳͷ
Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
None
ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛͬͯBʙF͕ىಈ • BʙFύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓ɺ͜Ε…ʁʁ
https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
Ͳ͏ͬͯ࡞Δʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ Sidekiq BatchΛ͏͜ͱͰ
తͷύΠϓϥΠϯ͕࣮Ͱ͖ͦ͏‼
ύΠϓϥΠϯ࣮Ͱ͖Δ͔ʁ ׂͱཧʹ͍ۙ☺ • جຊతʹͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά؆୯ • ޮతͳϦτϥΠ͕ߦΘΕΔ
• SidekiqͷฒྻઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴԽ͍ͯ͘͠ • CPU͕Γͳ͚Εεέʔϧͤ͞Δ
Transcoder
ϚΠΫϩαʔϏεͱͯ͠ͷ࣮ • ผαʔϏε͔ΒTranscoderͷϦΫΤετAWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁ՌAWS SQSʹͤͯฦ͢ •
தؒϑΝΠϧՌAWS S3ʹอଘ͢Δ • αʔϏεؒΦϒδΣΫτ໊ͷΓͱΓͷΈ • ࣮ϑΝΠϧ͏ଆ͕औΓʹߦ͘
αʔϏεͷεέʔϥϏϦςΟ • ffmpegϞϦϞϦͱCPUΛ͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ
Transcoder γεςϜߏ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷΓऔΓͷՕॴͰখن ʹRails͕ಈ͘γεςϜ
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 1 2 3 4 5
6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2
3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2
3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜɾ࠷ऴՌΛอ
ଘ͢ΔS3όέοτ 1 2 3 4 5 6
Transcoder ͷΞʔΩςΫνϟ JobͱResult • JobɺSQSʹͤΔJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoderඇಉظʹॲཧΛ։࢝͢Δ •
ResultɺSQSʹͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ߹ʹErrorResultΛฦ͢
None
None
None
Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮ • = Sidekiq Batchͷ࣮ͦͷͷɻ͜ΕࣗମSidekiq Worker
• ෦ʹෳͷStepΛ࣋ͭ • Pipeline͝ͱʹՌ͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •
Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
Transcoder ͷΞʔΩςΫνϟ Worker • StepͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͯ͢Sidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊffmpegͷίϚϯυΛҰճ࣮ߦ͢ΔཻͰ࣮͢Δ
• ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧS3ʹอଘ͢Δ
Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
Pipeline Step Step Step Worker + Command
Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ
Sidekiq Batch࣮ͷίπ • Batch෦ͷWorkerґଘΛ͘ɺཻ͔ͭΛখ࣮͘͢͞Δɻ • εέʔϧ͍͢͠ɺϦτϥΠ͍͢͠ɺͳͲͷతɻ • ෳࡶͳBatchΛ࡞Δ߹ɺPipelineΛදݱ͢ΔίʔυͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueueͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
• Callbackॲཧ࣮ମSidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըมύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷՌΛग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ
• Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮͕ංେԽͯ͠ؾʹͳΒͳ͍
ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Εʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏ͬͯΔ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
None
ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Εʁʣ • ffmpegίϚϯυͷೖग़ྗ༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠ೖྗͷॱংॏཁ
• CommandͷجఈΫϥεΛ࡞ͬͯநԽ • Pipeline → Step → Worker ͱɺෳͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ͢
None
·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiqͱͯྑ͍ •
Sidekiq Proͬͱྑ͍ • Sidekiq BatchΛ͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • நతͳ͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉSidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
·ͱΊ
None
THANK YOU!!