Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalable Microservice for Media Transcoding / T...
Search
atsushisakai
March 23, 2019
Programming
4
1.5k
Scalable Microservice for Media Transcoding / Transcoder
Rails Developers Meetup 2019の発表に利用した資料です。
atsushisakai
March 23, 2019
Tweet
Share
More Decks by atsushisakai
See All by atsushisakai
大規模 SaaS の技術的意思決定を支える三要素 / Three elements that support technical decision-making for large-scale SaaS
_atsushisakai
0
1k
一方、私はなぜEMでありつづけるか / TECH TRACK vol.1
_atsushisakai
0
440
子育てとEMと転職と
_atsushisakai
2
800
「家族アルバム みてね」を支える エンジニアリング組織/introduce-familyalbum-engineering-team
_atsushisakai
2
220
ミクシィのマネージャーは悩んでいる / mixi's manager is in trouble
_atsushisakai
16
14k
実践 Engineering Manager / practice engineering manager
_atsushisakai
19
8.1k
New Relic {FUTURE} STACK 3/14, 2019 / newrelic_future_stack
_atsushisakai
1
1.5k
実践 WorkManager / WorkManager
_atsushisakai
3
7.4k
How to use New Relic in Mitene Development Team
_atsushisakai
0
580
Other Decks in Programming
See All in Programming
第3回関東Kaggler会_AtCoderはKaggleの役に立つ
chettub
3
1.1k
Multi Step Form, Decentralized Autonomous Organization
pumpkiinbell
1
830
color-scheme: light dark; を完全に理解する
uhyo
7
480
Bedrock Agentsレスポンス解析によるAgentのOps
licux
3
900
Jakarta EE meets AI
ivargrimstad
0
220
ML.NETで始める機械学習
ymd65536
0
220
DROBEの生成AI活用事例 with AWS
ippey
0
140
Generating OpenAPI schema from serializers throughout the Rails stack - Kyobashi.rb #5
envek
1
330
プログラミング言語学習のススメ / why-do-i-learn-programming-language
yashi8484
0
150
新宿駅構内を三人称視点で探索してみる
satoshi7190
2
110
5分で理解する SOLID 原則 #phpcon_nagoya
shogogg
1
280
もう僕は OpenAPI を書きたくない
sgash708
5
1.9k
Featured
See All Featured
YesSQL, Process and Tooling at Scale
rocio
172
14k
Producing Creativity
orderedlist
PRO
344
40k
Mobile First: as difficult as doing things right
swwweet
223
9.4k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
Visualization
eitanlees
146
15k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.3k
The Language of Interfaces
destraynor
156
24k
Transcript
Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails
Developers Meetup 2019
ञҪ ಞ @_atsushisakai • ՈΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on
Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͡Ίͯ
Agenda ʮΈͯͶʯͰ͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝ͩͬͨͷ͔ʁ • Ͳ͏ͬͯղܾͨ͠ͷ͔ʁ
·ͣͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
ಈը(WIP)
͜ΕҰମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲͷΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞ΕͨಈըࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞ • ൢച͢ΔDVDʹऩͨ͠ΓɺΞϓϦʹ৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ͚ͯࢠҭͯͷࢥ͍ग़Λ࡞ʹ͢Δ •
͍ಈըΛͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢͳͲͷը૾ΛΓ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
ಈը(WIP)
1ඵಈըΛࣗಈͰେྔʹ࡞Δ
͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͚ͭΕΔ • ಈըͷ݁߹Ͱ͖Δ
• Ͱ͖ͳ͍͜ͱ • ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Իͱͷ߹ͱ͍͏ͷͰ͖ͳ͍
࡞Δ͔͠ͳ͍…
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%
͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ • ffmpegʹΑΔมॲཧΛྻʹ࿈݁ͨ͠ͷ • લͷॲཧͷΞτϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
Έͳ͞ΜͳΒͲ͏࣮͠·͔͢❓
࠷۪ͳ࣮
࠷۪ͳ࣮ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ͕ग़ͯ͘Δ • ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ৴ରͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷग़ͯ͘Δ
• CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͍ͬͨͳ͍ • ޮతͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮ ύΠϓϥΠϯΛಉ࣌ʹෳΒͤΔͱɺ͖ͬͱύϑΥʔϚϯεྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͋ΔఔΑ͘ͳΔ͔͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧͦ͠͏
None
Sidekiq Rubyͷ Job Queue γεςϜͰͱͯ༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ100Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ
• https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯগͳ͍ • ๛ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
Sidekiq
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔΛ࡞Δ • ෳͷ
Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
None
ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛͬͯBʙF͕ىಈ • BʙFύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓ɺ͜Ε…ʁʁ
https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
Ͳ͏ͬͯ࡞Δʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ Sidekiq BatchΛ͏͜ͱͰ
తͷύΠϓϥΠϯ͕࣮Ͱ͖ͦ͏‼
ύΠϓϥΠϯ࣮Ͱ͖Δ͔ʁ ׂͱཧʹ͍ۙ☺ • جຊతʹͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά؆୯ • ޮతͳϦτϥΠ͕ߦΘΕΔ
• SidekiqͷฒྻઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴԽ͍ͯ͘͠ • CPU͕Γͳ͚Εεέʔϧͤ͞Δ
Transcoder
ϚΠΫϩαʔϏεͱͯ͠ͷ࣮ • ผαʔϏε͔ΒTranscoderͷϦΫΤετAWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁ՌAWS SQSʹͤͯฦ͢ •
தؒϑΝΠϧՌAWS S3ʹอଘ͢Δ • αʔϏεؒΦϒδΣΫτ໊ͷΓͱΓͷΈ • ࣮ϑΝΠϧ͏ଆ͕औΓʹߦ͘
αʔϏεͷεέʔϥϏϦςΟ • ffmpegϞϦϞϦͱCPUΛ͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ
Transcoder γεςϜߏ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷΓऔΓͷՕॴͰখن ʹRails͕ಈ͘γεςϜ
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 1 2 3 4 5
6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2
3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2
3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜɾ࠷ऴՌΛอ
ଘ͢ΔS3όέοτ 1 2 3 4 5 6
Transcoder ͷΞʔΩςΫνϟ JobͱResult • JobɺSQSʹͤΔJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoderඇಉظʹॲཧΛ։࢝͢Δ •
ResultɺSQSʹͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ߹ʹErrorResultΛฦ͢
None
None
None
Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮ • = Sidekiq Batchͷ࣮ͦͷͷɻ͜ΕࣗମSidekiq Worker
• ෦ʹෳͷStepΛ࣋ͭ • Pipeline͝ͱʹՌ͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •
Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
Transcoder ͷΞʔΩςΫνϟ Worker • StepͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͯ͢Sidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊffmpegͷίϚϯυΛҰճ࣮ߦ͢ΔཻͰ࣮͢Δ
• ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧS3ʹอଘ͢Δ
Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
Pipeline Step Step Step Worker + Command
Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ
Sidekiq Batch࣮ͷίπ • Batch෦ͷWorkerґଘΛ͘ɺཻ͔ͭΛখ࣮͘͢͞Δɻ • εέʔϧ͍͢͠ɺϦτϥΠ͍͢͠ɺͳͲͷతɻ • ෳࡶͳBatchΛ࡞Δ߹ɺPipelineΛදݱ͢ΔίʔυͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueueͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
• Callbackॲཧ࣮ମSidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըมύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷՌΛग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ
• Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮͕ංେԽͯ͠ؾʹͳΒͳ͍
ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Εʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏ͬͯΔ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
None
ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Εʁʣ • ffmpegίϚϯυͷೖग़ྗ༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠ೖྗͷॱংॏཁ
• CommandͷجఈΫϥεΛ࡞ͬͯநԽ • Pipeline → Step → Worker ͱɺෳͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ͢
None
·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiqͱͯྑ͍ •
Sidekiq Proͬͱྑ͍ • Sidekiq BatchΛ͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • நతͳ͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉSidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
·ͱΊ
None
THANK YOU!!