実際運用してみてわかった、大規模Mastodonインスタンスを運用するコツ 2017-04-25 at pixiv night!
@harukasan / Shunsuke Michii࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
View Slide
@Harukasan Shunsuke MICHIILead engineer of pixiv Inc., Product manager of ImageFlux, Infrastructure engineer, Kibana, Gopher
࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
Mastodon
͍͖ͭͬ͞v1.2.2Λͩͨ͠
github.com/pixiv/mastodon:pawoo
Ruby on Railsnode.jsPostgreSQLRedisSendGridSidekiqΫϥΠΞϯτWebsocket (streaming)HTTPSMTPϞμϯͳRuby on RailsΞϓϦέʔγϣϯnginx
Ruby on Railsnode.jsPostgreSQLRedisSendGridSidekiqΫϥΠΞϯτWebsocket (streaming)HTTPSMTPϞμϯͳRuby on RailsΞϓϦέʔγϣϯnginxRESTͬΆ͍API
Ruby on Railsnode.jsPostgreSQLRedisSendGridSidekiqΫϥΠΞϯτWebsocket (streaming)HTTPSMTPϞμϯͳRuby on RailsΞϓϦέʔγϣϯnginxRESTͬΆ͍APIετϦʔϛϯάAPI
10࣌ؒͰϦϦʔε
Pawooͷߏ
AWSͰγϡοͱΔ• ͕࣌ؒͳ͍͚Ͳ࠷༏ઌ҆ఆੑ• RDS/ElasticaceͷϚϧνAZߏ• ϝσΟΞS3ʹอଘ• EC2ฒͯALBͰϩʔυόϥϯε• CloudFrontͰྲྀ͢ʢ͏Ίͨʣ• ظؒݶఆαΠτΛཱͯΔ࣌ͷϊϋΛྲྀ༻
ΠϯελϯεΛฒΔ• ͯ͢ͷΠϯελϯεಉҰߏ• γϡοͱͭͬͯ͘AMIͰ૿͢• ԣʹฒͯεέʔϧΞτ• ͦͷ͏ͪΦʔτεέʔϦϯά͍ͨ͠
Docker͔Β͕͢• શ෦systemdαʔϏεʹஔ͖͑ͨ• ͦͷ··͏ͷͪΐͬͱ͍͜͠ͱ͕ଟ͍• σϓϩΠͷ͜ͱΒͳΜΒߟ͍͑ͯΔͱͪΐͬͱ༨༟͕ͳ͍• DockerͰ͙͢ΔͳΒϗοτσϓϩΠͱ͔ϦιʔεͷΈΛ͋Β͔͡Ί༻ҙ͓ͯ͘͠ඞཁ͕͋Γͦ͏ʢͦͷ͏ͪΓ͍ͨʣ
ͦͷͷ͏ͪʹϦϦʔε͞Ε͕͙ͨ͢ʹΤϥʔΛฦ͠͡ΊΔʢ͜͜ΒลͰνϟοτΛݟ࢝ΊΔʣ
nginx• ϦΫΤετόοϑΝϦϯάͷͨΊʹPumaͷલʹnginxΛ͓͘• Τϥʔ͕ग़͍ͯͨͷͰͳ͓ͨ͠• worker_rlimit_nofile: limitΛ૿͓ͯ͘͠• worker_connections: ίωΫγϣϯͨ͘͞ΜΔͷͰ• ϩάϩʔςʔτͷઃఆ͕ೖͬͯͳ͔ͬͨͷͰೖΕͨ• όϥϯγϯάEC2ͷnginx͡Όͳͯ͘ALBͰΔ
ৄ͘͜͠ͷຊʹ
PostgreSQL• PostgreSQLίωΫγϣϯ͝ͱʹfork͢Δ ʢฐࣾͯ͢MySQLͩͬͨͷͰݟͳ͠ʣ• ίωΫγϣϯϓʔϧΛ͏·͍۩߹ͷʹઃఆ͢Δ• ΠϯελϯεΛεέʔϧΞοϓͯ͠ରԠ• ίʔυʹ࠷దԽΛೖΕͯΠϯελϯεαΠζখͨ͘͞͠Γͨ͠• Մ༻ੑͱόοΫΞοϓRDSʹ͓ͤ• PgBouncerͬͨํ͕͍͍͔ʁͱࢥ͍ͳ͕Βාͯͬͯ͘ͳ͍
͜͜ͰεέʔϧΞοϓ
ݟٻΉ
Ruby on Railsnode.jsPostgreSQLRedisSendGridSidekiqΫϥΠΞϯτWebsocket (streaming)HTTPSMTPnginxϢʔβʔΠϯλʔϑΣʔεϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
Ruby on Railsnode.jsPostgreSQLRedisSendGridSidekiqΫϥΠΞϯτWebsocket (streaming)HTTPSMTPnginxϞμϯͳRuby on RailsΞϓϦέʔγϣϯϝΠϯϩδοΫ
Sidekiq• MastodonͷϝοηʔδύογϯάSidekiqͰߦΘΕΔ• τΡʔτͷ৴• ετϦʔϛϯά• ϦϞʔτΠϯελϯεͷ৴• 1τΡʔτ͝ͱʹϑΥϩϫʔͷ͚ͩenqueue͞ΕΔ• ϑΥϩϫʔ͕ଟ͍Ϣʔβʔͩͱ͍
ීஈͷΩϡʔͷ༷ࢠʢͪΐͬͱલʣ
ͨ·ʹ͜͏ͳΔ
Ωϡʔͷछྨdefaultpullpushmail• ΄͔ͷશ෦ͷδϣϒ• ϦϞʔτΠϯελϯεͷ৴• ϝʔϧͷૹ৴
Sidekiq٧·Δ• σϑΥϧτͩͱͯ͢ͷΩϡʔΛ̍ͭͷSidekiqϓϩηεͰॲཧ͢Δ• ϦϞʔτΠϯελϯε͕٧·Δͱpull/pushͷΩϡʔ͕٧·Δ• pull/pushͷ͍ͤͰλΠϜϥΠϯ͕Ԇ͢Δʂ• ͜ͷ··ͩͱαʔϏεͱͯ͠ऴΘͬͯ͠·͏ͷͰɺϓϩηεΛ͚Δ• Sidekiq1ϓϩηεͰ1ίΞʢ100%ʣ·Ͱ͔ͬͯ͘͠Εͳ͍• ϓϩηεΛͨ͘͞ΜཱͯͯΧόʔ
ίϚϯυϥΠϯҾͰࢦఆ͢Δ͚ͩ[Service]Type=simpleUser=mastodonWorkingDirectory=/home/mastodon/liveEnvironment="RAILS_ENV=production"Environment="DB_POOL=40"ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq -c 40 -q defaultTimeoutSec=15Restart=always
SidekiqϓϩηεΛ͚Δsidekiq-default-1.servicesidekiq-default-2.servicesidekiq-default-3.servicesidekiq-pullpush.servicesidekiq-mail.service1Πϯελϯεʹ͜Εཱ͚ͩͯͯΔɿσϑΥϧτ͚ͩͰ1,000εϨουҎ্404040105εϨου
Sidekiqͷϓϩηεը໘͕ ͍ͩͿݟͮΒ͍
Sidekiq٧·Δͷ՝• ϦϞʔτΠϯελϯε͕٧·Δͱ૬มΘΒͣpull/push͕٧·Δ• ϓϩηεΛཱͯͯΧόʔͯ͠Δ͚Ͳɺ΄͔ͷΠϯελϯε͔Β͢ΔͱٸʹϦΫΤετ͕૿͑ͯͭΒͦ͏……• ͍Πϯελϯεͱ͔େنΠϯελϯεͷॲཧΛͲ͏ʹ͔͍ͨ͠• ͦͦτΡʔτ͋ͨΓͷδϣϒ͕N+1ʹͳͬͯΔͷ͕͍
ղܾͯ͠ͳ͍͚Ͳ·ͱΊ• AWSͬͯγϡοͱཱͯͯӡ༻։࢝• ϚετυϯͷϝΠϯSidekiqδϣϒ• SidekiqͷΩϡʔΛ͏·͚͘͞ΔͱΑ͍• SidekiqϓϩηεΛεϙοτΠϯελϯεͰ૿͢Α͏ʹ͍ͨ͠• Ͳ͔ͬͪͱ͍͏ͱ࠷దԽͨ͠ํ͕Αͦ͞͏