大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

124da56a613b15fa980427533e4e3839?s=47 Harukasan
April 27, 2017

大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

実際運用してみてわかった、大規模Mastodonインスタンスを運用するコツ
2017-04-25 at pixiv night!

124da56a613b15fa980427533e4e3839?s=128

Harukasan

April 27, 2017
Tweet

Transcript

  1. @harukasan / Shunsuke Michii ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ

  2. @Harukasan Shunsuke MICHII Lead engineer of pixiv Inc.,
 Product manager

    of ImageFlux,
 Infrastructure engineer, Kibana, Gopher
  3. None
  4. ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ

  5. None
  6. Mastodon

  7. ͍͖ͭͬ͞v1.2.2Λͩͨ͠

  8. github.com/pixiv/mastodon:pawoo

  9. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx
  10. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx RESTͬΆ͍API
  11. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx RESTͬΆ͍API ετϦʔϛϯάAPI
  12. 10࣌ؒͰϦϦʔε

  13. Pawooͷߏ੒

  14. None
  15. AWSͰγϡοͱ΍Δ • ͕࣌ؒͳ͍͚Ͳ࠷༏ઌ͸҆ఆੑ • RDS/ElasticaceͷϚϧνAZߏ੒ • ϝσΟΞ͸S3ʹอଘ • EC2ฒ΂ͯALBͰϩʔυόϥϯε •

    CloudFrontͰྲྀ͢ʢ΋͏΍Ίͨʣ • ظؒݶఆαΠτΛཱͯΔ࣌ͷϊ΢ϋ΢Λྲྀ༻
  16. ΠϯελϯεΛฒ΂Δ • ͢΂ͯͷΠϯελϯε͸ಉҰߏ੒ • γϡοͱͭͬͯ͘AMIͰ૿΍͢ • ԣʹฒ΂ͯεέʔϧΞ΢τ • ͦͷ͏ͪΦʔτεέʔϦϯά͍ͨ͠

  17. Docker͔Β͸͕͢ • શ෦systemdαʔϏεʹஔ͖׵͑ͨ • ͦͷ··࢖͏ͷ͸ͪΐͬͱ೉͍͜͠ͱ͕ଟ͍ • σϓϩΠͷ͜ͱ΍ΒͳΜ΍Βߟ͍͑ͯΔͱͪΐͬͱ༨༟͕ͳ͍ • DockerͰ͙͢΍ΔͳΒϗοτσϓϩΠͱ͔Ϧιʔε഑෼ͷ࢓૊ΈΛ ͋Β͔͡Ί༻ҙ͓ͯ͘͠ඞཁ͕͋Γͦ͏ʢͦͷ͏ͪ΍Γ͍ͨʣ

  18. ͦͷ೔ͷ͏ͪʹϦϦʔε͞Ε͕ͨ ͙͢ʹΤϥʔΛฦ͠͸͡ΊΔ ʢ͜͜ΒลͰνϟοτΛݟ࢝ΊΔʣ

  19. nginx • ϦΫΤετόοϑΝϦϯάͷͨΊʹPumaͷલʹnginxΛ͓͘ • Τϥʔ͕ग़͍ͯͨͷͰͳ͓ͨ͠ • worker_rlimit_nofile: limitΛ૿΍͓ͯ͘͠ • worker_connections:

    ίωΫγϣϯͨ͘͞Μ͸ΔͷͰ • ϩάϩʔςʔτͷઃఆ͕ೖͬͯͳ͔ͬͨͷͰೖΕͨ • όϥϯγϯά͸EC2ͷnginx͡Όͳͯ͘ALBͰ΍Δ
  20. ৄ͘͠͸͜ͷຊʹ

  21. PostgreSQL • PostgreSQL͸ίωΫγϣϯ͝ͱʹfork͢Δ
 ʢฐࣾ͸͢΂ͯMySQLͩͬͨͷͰ஌ݟͳ͠ʣ • ίωΫγϣϯϓʔϧΛ͏·͍۩߹ͷ஋ʹઃఆ͢Δ • ΠϯελϯεΛεέʔϧΞοϓͯ͠ରԠ • ίʔυʹ࠷దԽΛೖΕͯΠϯελϯεαΠζখͨ͘͞͠Γͨ͠

    • Մ༻ੑͱόοΫΞοϓ͸RDSʹ͓೚ͤ • PgBouncer࢖ͬͨํ͕͍͍͔΋ʁͱࢥ͍ͳ͕Βාͯ͘࢖ͬͯͳ͍
  22. ͜͜ͰεέʔϧΞοϓ

  23. None
  24. ஌ݟٻΉ

  25. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx
  26. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP nginx ϢʔβʔΠϯλʔϑΣʔε ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
  27. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket


    (streaming) HTTP SMTP nginx ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ ϝΠϯϩδοΫ
  28. Sidekiq • Mastodonͷϝοηʔδύογϯά͸SidekiqͰߦΘΕΔ • τΡʔτͷ഑৴ • ετϦʔϛϯά • ϦϞʔτΠϯελϯε΁ͷ഑৴ •

    1τΡʔτ͝ͱʹϑΥϩϫʔͷ਺͚ͩenqueue͞ΕΔ • ϑΥϩϫʔ͕ଟ͍Ϣʔβʔͩͱ΍͹͍
  29. ීஈͷΩϡʔͷ༷ࢠʢͪΐͬͱલʣ

  30. ͨ·ʹ͜͏ͳΔ

  31. ͨ·ʹ͜͏ͳΔ

  32. Ωϡʔͷछྨ default pull push mail • ΄͔ͷશ෦ͷδϣϒ • ϦϞʔτΠϯελϯε΁ͷ഑৴ •

    ϝʔϧͷૹ৴
  33. Sidekiq٧·Δ໰୊ • σϑΥϧτͩͱ͢΂ͯͷΩϡʔΛ̍ͭͷSidekiqϓϩηεͰॲཧ͢Δ • ϦϞʔτΠϯελϯε͕٧·Δͱpull/pushͷΩϡʔ͕٧·Δ • pull/pushͷ͍ͤͰλΠϜϥΠϯ͕஗Ԇ͢Δʂ • ͜ͷ··ͩͱαʔϏεͱͯ͠ऴΘͬͯ͠·͏ͷͰɺϓϩηεΛ෼͚Δ •

    Sidekiq͸1ϓϩηεͰ1ίΞʢ100%ʣ·Ͱ͔͠࢖ͬͯ͘Εͳ͍ • ϓϩηεΛͨ͘͞ΜཱͯͯΧόʔ
  34. ίϚϯυϥΠϯҾ਺Ͱࢦఆ͢Δ͚ͩ [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="RAILS_ENV=production" Environment="DB_POOL=40" ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq

    -c 40 -q default TimeoutSec=15 Restart=always
  35. SidekiqϓϩηεΛ෼͚Δ sidekiq-default-1.service sidekiq-default-2.service sidekiq-default-3.service sidekiq-pullpush.service sidekiq-mail.service 1Πϯελϯεʹ͜Εཱ͚ͩͯͯΔɿ σϑΥϧτ͚ͩͰ1,000εϨουҎ্ 40 40

    40 10 5 εϨου਺
  36. Sidekiqͷϓϩηεը໘͕
 ͍ͩͿݟͮΒ͍

  37. Sidekiq٧·Δ໰୊ͷ՝୊ • ϦϞʔτΠϯελϯε͕٧·Δͱ૬มΘΒͣpull/push͕٧·Δ • ϓϩηεΛཱͯͯΧόʔͯ͠Δ͚Ͳɺ΄͔ͷΠϯελϯε͔Β͢Δͱ ٸʹϦΫΤετ͕૿͑ͯͭΒͦ͏…… • ஗͍Πϯελϯεͱ͔େن໛ΠϯελϯεͷॲཧΛͲ͏ʹ͔͍ͨ͠ • ͦ΋ͦ΋τΡʔτ͋ͨΓͷδϣϒ਺͕N+1ʹͳͬͯΔͷ͕΍͹͍

  38. ղܾͯ͠ͳ͍͚Ͳ·ͱΊ • AWS࢖ͬͯγϡοͱཱͯͯӡ༻։࢝ • ϚετυϯͷϝΠϯ͸Sidekiqδϣϒ • SidekiqͷΩϡʔΛ͏·͘͞͹͚ΔͱΑ͍ • SidekiqϓϩηεΛεϙοτΠϯελϯεͰ૿΍͢Α͏ʹ͍ͨ͠ •

    Ͳ͔ͬͪͱ͍͏ͱ࠷దԽͨ͠ํ͕Αͦ͞͏