大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

実際運用してみてわかった、大規模Mastodonインスタンスを運用するコツ
2017-04-25 at pixiv night!

Transcript

1. @harukasan / Shunsuke Michii
   ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
   

   View Slide

2. @Harukasan Shunsuke MICHII
   Lead engineer of pixiv Inc.,

   Product manager of ImageFlux,

   Infrastructure engineer, Kibana, Gopher
   

   View Slide

3. View Slide

4. ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
   

   View Slide

5. View Slide

6. Mastodon
   

   View Slide

7. ͍͖ͭͬ͞v1.2.2Λͩͨ͠
   

   View Slide

8. github.com/pixiv/mastodon:pawoo
   

   View Slide

9. Ruby on Rails
   node.js
   PostgreSQL
   Redis
   SendGrid
   Sidekiq
   ΫϥΠΞϯτ
   Websocket

   (streaming)
   HTTP
   SMTP
   ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
   nginx
   

   View Slide

10. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx
    RESTͬΆ͍API
    

    View Slide

11. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx
    RESTͬΆ͍API
    ετϦʔϛϯάAPI
    

    View Slide

12. 10࣌ؒͰϦϦʔε
    
    

    View Slide

13. Pawooͷߏ੒
    

    View Slide

14. View Slide

15. AWSͰγϡοͱ΍Δ
    • ͕࣌ؒͳ͍͚Ͳ࠷༏ઌ͸҆ఆੑ
    • RDS/ElasticaceͷϚϧνAZߏ੒
    • ϝσΟΞ͸S3ʹอଘ
    • EC2ฒ΂ͯALBͰϩʔυόϥϯε
    • CloudFrontͰྲྀ͢ʢ΋͏΍Ίͨʣ
    • ظؒݶఆαΠτΛཱͯΔ࣌ͷϊ΢ϋ΢Λྲྀ༻
    

    View Slide

16. ΠϯελϯεΛฒ΂Δ
    • ͢΂ͯͷΠϯελϯε͸ಉҰߏ੒
    • γϡοͱͭͬͯ͘AMIͰ૿΍͢
    • ԣʹฒ΂ͯεέʔϧΞ΢τ
    • ͦͷ͏ͪΦʔτεέʔϦϯά͍ͨ͠
    

    View Slide

17. Docker͔Β͸͕͢
    • શ෦systemdαʔϏεʹஔ͖׵͑ͨ
    • ͦͷ··࢖͏ͷ͸ͪΐͬͱ೉͍͜͠ͱ͕ଟ͍
    • σϓϩΠͷ͜ͱ΍ΒͳΜ΍Βߟ͍͑ͯΔͱͪΐͬͱ༨༟͕ͳ͍
    • DockerͰ͙͢΍ΔͳΒϗοτσϓϩΠͱ͔Ϧιʔε഑෼ͷ࢓૊ΈΛ
    ͋Β͔͡Ί༻ҙ͓ͯ͘͠ඞཁ͕͋Γͦ͏ʢͦͷ͏ͪ΍Γ͍ͨʣ
    

    View Slide

18. ͦͷ೔ͷ͏ͪʹϦϦʔε͞Ε͕ͨ
    ͙͢ʹΤϥʔΛฦ͠͸͡ΊΔ
    ʢ͜͜ΒลͰνϟοτΛݟ࢝ΊΔʣ
    

    View Slide

19. nginx
    • ϦΫΤετόοϑΝϦϯάͷͨΊʹPumaͷલʹnginxΛ͓͘
    • Τϥʔ͕ग़͍ͯͨͷͰͳ͓ͨ͠
    • worker_rlimit_nofile: limitΛ૿΍͓ͯ͘͠
    • worker_connections: ίωΫγϣϯͨ͘͞Μ͸ΔͷͰ
    • ϩάϩʔςʔτͷઃఆ͕ೖͬͯͳ͔ͬͨͷͰೖΕͨ
    • όϥϯγϯά͸EC2ͷnginx͡Όͳͯ͘ALBͰ΍Δ
    

    View Slide

20. ৄ͘͠͸͜ͷຊʹ
    

    View Slide

21. PostgreSQL
    • PostgreSQL͸ίωΫγϣϯ͝ͱʹfork͢Δ

    ʢฐࣾ͸͢΂ͯMySQLͩͬͨͷͰ஌ݟͳ͠ʣ
    • ίωΫγϣϯϓʔϧΛ͏·͍۩߹ͷ஋ʹઃఆ͢Δ
    • ΠϯελϯεΛεέʔϧΞοϓͯ͠ରԠ
    • ίʔυʹ࠷దԽΛೖΕͯΠϯελϯεαΠζখͨ͘͞͠Γͨ͠
    • Մ༻ੑͱόοΫΞοϓ͸RDSʹ͓೚ͤ
    • PgBouncer࢖ͬͨํ͕͍͍͔΋ʁͱࢥ͍ͳ͕Βාͯ͘࢖ͬͯͳ͍
    

    View Slide

22. ͜͜ͰεέʔϧΞοϓ
    

    View Slide

23. View Slide

24. ஌ݟٻΉ
    

    View Slide

25. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    nginx
    

    View Slide

26. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    nginx
    ϢʔβʔΠϯλʔϑΣʔε
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    

    View Slide

27. Ruby on Rails
    node.js
    PostgreSQL
    Redis
    SendGrid
    Sidekiq
    ΫϥΠΞϯτ
    Websocket

    (streaming)
    HTTP
    SMTP
    nginx
    ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
    ϝΠϯϩδοΫ
    

    View Slide

28. Sidekiq
    • Mastodonͷϝοηʔδύογϯά͸SidekiqͰߦΘΕΔ
    • τΡʔτͷ഑৴
    • ετϦʔϛϯά
    • ϦϞʔτΠϯελϯε΁ͷ഑৴
    • 1τΡʔτ͝ͱʹϑΥϩϫʔͷ਺͚ͩenqueue͞ΕΔ
    • ϑΥϩϫʔ͕ଟ͍Ϣʔβʔͩͱ΍͹͍
    

    View Slide

29. ීஈͷΩϡʔͷ༷ࢠʢͪΐͬͱલʣ
    

    View Slide

30. ͨ·ʹ͜͏ͳΔ
    

    View Slide

31. ͨ·ʹ͜͏ͳΔ
    

    View Slide

32. Ωϡʔͷछྨ
    default
    pull
    push
    mail
    • ΄͔ͷશ෦ͷδϣϒ
    • ϦϞʔτΠϯελϯε΁ͷ഑৴
    • ϝʔϧͷૹ৴
    

    View Slide

33. Sidekiq٧·Δ໰୊
    • σϑΥϧτͩͱ͢΂ͯͷΩϡʔΛ̍ͭͷSidekiqϓϩηεͰॲཧ͢Δ
    • ϦϞʔτΠϯελϯε͕٧·Δͱpull/pushͷΩϡʔ͕٧·Δ
    • pull/pushͷ͍ͤͰλΠϜϥΠϯ͕஗Ԇ͢Δʂ
    • ͜ͷ··ͩͱαʔϏεͱͯ͠ऴΘͬͯ͠·͏ͷͰɺϓϩηεΛ෼͚Δ
    • Sidekiq͸1ϓϩηεͰ1ίΞʢ100%ʣ·Ͱ͔͠࢖ͬͯ͘Εͳ͍
    • ϓϩηεΛͨ͘͞ΜཱͯͯΧόʔ
    

    View Slide

34. ίϚϯυϥΠϯҾ਺Ͱࢦఆ͢Δ͚ͩ
    [Service]
    Type=simple
    User=mastodon
    WorkingDirectory=/home/mastodon/live
    Environment="RAILS_ENV=production"
    Environment="DB_POOL=40"
    ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq -c 40 -q default
    TimeoutSec=15
    Restart=always
    

    View Slide

35. SidekiqϓϩηεΛ෼͚Δ
    sidekiq-default-1.service
    sidekiq-default-2.service
    sidekiq-default-3.service
    sidekiq-pullpush.service
    sidekiq-mail.service
    1Πϯελϯεʹ͜Εཱ͚ͩͯͯΔɿ
    σϑΥϧτ͚ͩͰ1,000εϨουҎ্
    40
    40
    40
    10
    5
    εϨου਺
    

    View Slide

36. Sidekiqͷϓϩηεը໘͕

    ͍ͩͿݟͮΒ͍
    

    View Slide

37. Sidekiq٧·Δ໰୊ͷ՝୊
    • ϦϞʔτΠϯελϯε͕٧·Δͱ૬มΘΒͣpull/push͕٧·Δ
    • ϓϩηεΛཱͯͯΧόʔͯ͠Δ͚Ͳɺ΄͔ͷΠϯελϯε͔Β͢Δͱ
    ٸʹϦΫΤετ͕૿͑ͯͭΒͦ͏……
    • ஗͍Πϯελϯεͱ͔େن໛ΠϯελϯεͷॲཧΛͲ͏ʹ͔͍ͨ͠
    • ͦ΋ͦ΋τΡʔτ͋ͨΓͷδϣϒ਺͕N+1ʹͳͬͯΔͷ͕΍͹͍
    

    View Slide

38. ղܾͯ͠ͳ͍͚Ͳ·ͱΊ
    • AWS࢖ͬͯγϡοͱཱͯͯӡ༻։࢝
    • ϚετυϯͷϝΠϯ͸Sidekiqδϣϒ
    • SidekiqͷΩϡʔΛ͏·͘͞͹͚ΔͱΑ͍
    • SidekiqϓϩηεΛεϙοτΠϯελϯεͰ૿΍͢Α͏ʹ͍ͨ͠
    • Ͳ͔ͬͪͱ͍͏ͱ࠷దԽͨ͠ํ͕Αͦ͞͏
    

    View Slide