Upgrade to Pro — share decks privately, control downloads, hide ads and more …

大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

Harukasan
April 27, 2017

大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

実際運用してみてわかった、大規模Mastodonインスタンスを運用するコツ
2017-04-25 at pixiv night!

Harukasan

April 27, 2017
Tweet

More Decks by Harukasan

Other Decks in Technology

Transcript

  1. @harukasan / Shunsuke Michii
    ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ

    View Slide

  2. @Harukasan Shunsuke MICHII
    Lead engineer of pixiv Inc.,

    Product manager of ImageFlux,

    Infrastructure engineer, Kibana, Gopher

    View Slide

  3. View Slide

  4. ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ

    View Slide

  5. View Slide

  6. Mastodon

    View Slide

  7. ͍͖ͭͬ͞v1.2.2Λͩͨ͠

    View Slide

  8. github.com/pixiv/mastodon:pawoo

    View Slide

  9. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx

    View Slide

  10. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx
    RESTͬΆ͍API

    View Slide

  11. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx
    RESTͬΆ͍API
    ετϦʔϛϯάAPI

    View Slide

  12. 10࣌ؒͰϦϦʔε

    View Slide

  13. Pawooͷߏ੒

    View Slide

  14. View Slide

  15. AWSͰγϡοͱ΍Δ
    • ͕࣌ؒͳ͍͚Ͳ࠷༏ઌ͸҆ఆੑ
    • RDS/ElasticaceͷϚϧνAZߏ੒
    • ϝσΟΞ͸S3ʹอଘ
    • EC2ฒ΂ͯALBͰϩʔυόϥϯε
    • CloudFrontͰྲྀ͢ʢ΋͏΍Ίͨʣ
    • ظؒݶఆαΠτΛཱͯΔ࣌ͷϊ΢ϋ΢Λྲྀ༻

    View Slide

  16. ΠϯελϯεΛฒ΂Δ
    • ͢΂ͯͷΠϯελϯε͸ಉҰߏ੒
    • γϡοͱͭͬͯ͘AMIͰ૿΍͢
    • ԣʹฒ΂ͯεέʔϧΞ΢τ
    • ͦͷ͏ͪΦʔτεέʔϦϯά͍ͨ͠

    View Slide

  17. Docker͔Β͸͕͢
    • શ෦systemdαʔϏεʹஔ͖׵͑ͨ
    • ͦͷ··࢖͏ͷ͸ͪΐͬͱ೉͍͜͠ͱ͕ଟ͍
    • σϓϩΠͷ͜ͱ΍ΒͳΜ΍Βߟ͍͑ͯΔͱͪΐͬͱ༨༟͕ͳ͍
    • DockerͰ͙͢΍ΔͳΒϗοτσϓϩΠͱ͔Ϧιʔε഑෼ͷ࢓૊ΈΛ
    ͋Β͔͡Ί༻ҙ͓ͯ͘͠ඞཁ͕͋Γͦ͏ʢͦͷ͏ͪ΍Γ͍ͨʣ

    View Slide

  18. ͦͷ೔ͷ͏ͪʹϦϦʔε͞Ε͕ͨ
    ͙͢ʹΤϥʔΛฦ͠͸͡ΊΔ
    ʢ͜͜ΒลͰνϟοτΛݟ࢝ΊΔʣ

    View Slide

  19. nginx
    • ϦΫΤετόοϑΝϦϯάͷͨΊʹPumaͷલʹnginxΛ͓͘
    • Τϥʔ͕ग़͍ͯͨͷͰͳ͓ͨ͠
    • worker_rlimit_nofile: limitΛ૿΍͓ͯ͘͠
    • worker_connections: ίωΫγϣϯͨ͘͞Μ͸ΔͷͰ
    • ϩάϩʔςʔτͷઃఆ͕ೖͬͯͳ͔ͬͨͷͰೖΕͨ
    • όϥϯγϯά͸EC2ͷnginx͡Όͳͯ͘ALBͰ΍Δ

    View Slide

  20. ৄ͘͠͸͜ͷຊʹ

    View Slide

  21. PostgreSQL
    • PostgreSQL͸ίωΫγϣϯ͝ͱʹfork͢Δ

    ʢฐࣾ͸͢΂ͯMySQLͩͬͨͷͰ஌ݟͳ͠ʣ
    • ίωΫγϣϯϓʔϧΛ͏·͍۩߹ͷ஋ʹઃఆ͢Δ
    • ΠϯελϯεΛεέʔϧΞοϓͯ͠ରԠ
    • ίʔυʹ࠷దԽΛೖΕͯΠϯελϯεαΠζখͨ͘͞͠Γͨ͠
    • Մ༻ੑͱόοΫΞοϓ͸RDSʹ͓೚ͤ
    • PgBouncer࢖ͬͨํ͕͍͍͔΋ʁͱࢥ͍ͳ͕Βාͯ͘࢖ͬͯͳ͍

    View Slide

  22. ͜͜ͰεέʔϧΞοϓ

    View Slide

  23. View Slide

  24. ஌ݟٻΉ

    View Slide

  25. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx

    View Slide

  26. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    nginx
    ϢʔβʔΠϯλʔϑΣʔε
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ

    View Slide

  27. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    nginx
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    ϝΠϯϩδοΫ

    View Slide

  28. Sidekiq
    • Mastodonͷϝοηʔδύογϯά͸SidekiqͰߦΘΕΔ
    • τΡʔτͷ഑৴
    • ετϦʔϛϯά
    • ϦϞʔτΠϯελϯε΁ͷ഑৴
    • 1τΡʔτ͝ͱʹϑΥϩϫʔͷ਺͚ͩenqueue͞ΕΔ
    • ϑΥϩϫʔ͕ଟ͍Ϣʔβʔͩͱ΍͹͍

    View Slide

  29. ීஈͷΩϡʔͷ༷ࢠʢͪΐͬͱલʣ

    View Slide

  30. ͨ·ʹ͜͏ͳΔ

    View Slide

  31. ͨ·ʹ͜͏ͳΔ

    View Slide

  32. Ωϡʔͷछྨ
    default
    pull
    push
    mail
    • ΄͔ͷશ෦ͷδϣϒ
    • ϦϞʔτΠϯελϯε΁ͷ഑৴
    • ϝʔϧͷૹ৴

    View Slide

  33. Sidekiq٧·Δ໰୊
    • σϑΥϧτͩͱ͢΂ͯͷΩϡʔΛ̍ͭͷSidekiqϓϩηεͰॲཧ͢Δ
    • ϦϞʔτΠϯελϯε͕٧·Δͱpull/pushͷΩϡʔ͕٧·Δ
    • pull/pushͷ͍ͤͰλΠϜϥΠϯ͕஗Ԇ͢Δʂ
    • ͜ͷ··ͩͱαʔϏεͱͯ͠ऴΘͬͯ͠·͏ͷͰɺϓϩηεΛ෼͚Δ
    • Sidekiq͸1ϓϩηεͰ1ίΞʢ100%ʣ·Ͱ͔͠࢖ͬͯ͘Εͳ͍
    • ϓϩηεΛͨ͘͞ΜཱͯͯΧόʔ

    View Slide

  34. ίϚϯυϥΠϯҾ਺Ͱࢦఆ͢Δ͚ͩ
    [Service]
    Type=simple
    User=mastodon
    WorkingDirectory=/home/mastodon/live
    Environment="RAILS_ENV=production"
    Environment="DB_POOL=40"
    ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq -c 40 -q default
    TimeoutSec=15
    Restart=always

    View Slide

  35. SidekiqϓϩηεΛ෼͚Δ
    sidekiq-default-1.service
    sidekiq-default-2.service
    sidekiq-default-3.service
    sidekiq-pullpush.service
    sidekiq-mail.service
    1Πϯελϯεʹ͜Εཱ͚ͩͯͯΔɿ
    σϑΥϧτ͚ͩͰ1,000εϨουҎ্
    40
    40
    40
    10
    5
    εϨου਺

    View Slide

  36. Sidekiqͷϓϩηεը໘͕

    ͍ͩͿݟͮΒ͍

    View Slide

  37. Sidekiq٧·Δ໰୊ͷ՝୊
    • ϦϞʔτΠϯελϯε͕٧·Δͱ૬มΘΒͣpull/push͕٧·Δ
    • ϓϩηεΛཱͯͯΧόʔͯ͠Δ͚Ͳɺ΄͔ͷΠϯελϯε͔Β͢Δͱ
    ٸʹϦΫΤετ͕૿͑ͯͭΒͦ͏……
    • ஗͍Πϯελϯεͱ͔େن໛ΠϯελϯεͷॲཧΛͲ͏ʹ͔͍ͨ͠
    • ͦ΋ͦ΋τΡʔτ͋ͨΓͷδϣϒ਺͕N+1ʹͳͬͯΔͷ͕΍͹͍

    View Slide

  38. ղܾͯ͠ͳ͍͚Ͳ·ͱΊ
    • AWS࢖ͬͯγϡοͱཱͯͯӡ༻։࢝
    • ϚετυϯͷϝΠϯ͸Sidekiqδϣϒ
    • SidekiqͷΩϡʔΛ͏·͘͞͹͚ΔͱΑ͍
    • SidekiqϓϩηεΛεϙοτΠϯελϯεͰ૿΍͢Α͏ʹ͍ͨ͠
    • Ͳ͔ͬͪͱ͍͏ͱ࠷దԽͨ͠ํ͕Αͦ͞͏

    View Slide