Upgrade to Pro — share decks privately, control downloads, hide ads and more …

fluentdによる大規模キュー設計

 fluentdによる大規模キュー設計

「Fluentd Meetup 2015 夏」で発表した内容です。

Atsushi Takayama

June 01, 2015
Tweet

More Decks by Atsushi Takayama

Other Decks in Technology

Transcript

  1. fluentdʹΑΔ
    େن໛Ωϡʔઃܭ
    A large scale queue design with fluentd

    View full-size slide

  2. ࣗݾ঺հ
    • ߴࢁ Թ @edvakf
    • pixivͷອըࣄۀͷ։ൃ੹೚ऀ
    • ීஈॻ͍ͯΔͷ͸Scala΍PHPɺͨ·ʹJS΍Ruby΍Go
    • ࠷ۙϦʔυΤϯδχΞͱ͍͏ݞॻʹ
    • ձࣾͷٕज़બ୒ʹ੹೚Λ࣋ͭ
    • ʮࣾ಺ͷΤϯδχΞ͕޾ͤʹಇͨ͘Ίʹ͸ʯΛߟ࣮͑ݱ͢Δۀ

    View full-size slide

  3. ී௨ͷ
    ΢ΣϒΞϓϦ+δϣϒΩϡʔ

    View full-size slide

  4. Resque
    Resque

    View full-size slide

  5. MySQL Q4M
    PHP
    PHP
    PHP
    PHP
    Q4M batch

    View full-size slide

  6. MongoDB
    PHP
    PHP
    PHP
    PHP
    Mongo batch

    View full-size slide

  7. ྑ͍఺
    • Sidekiq΍Resque͸൚༻δϣϒΩϡʔ
    • Rubyͷϝιου୯ҐͰ؆୯ʹδϣϒʹͰ͖Δ
    • ޿͘࢖ΘΕ͍ͯΔ
    • ΋͠Redis΍MongoDBͳͲΛطʹ࢖͍ͬͯΔͳΒ৽͍͠
    ϛυϧ΢ΣΞΛಋೖ͠ͳͯ͘ྑ͍

    View full-size slide

  8. ΋͠
    • 1෼ؒʹ10ສճͷδϣϒΛ࣮ߦ͠ͳ͍ͱ͍͚ͳ͔ͬͨΒ
    • ਺࣌ؒલͱ͔਺೔લͷঢ়ଶ·Ͱר͖໭ͬͯδϣϒΛ࣮ߦ
    ͠௚͞ͳ͍ͱ͍͚ͳ͔ͬͨΒ

    View full-size slide

  9. ΞΫηεղੳ
    www.pixiv.net

    View full-size slide

  10. ࠷ॳʹߟ͑ͨҊ

    View full-size slide

  11. MongoDB
    PHP
    PHP
    PHP
    PHP
    Mongo batch

    View full-size slide

  12. ݒ೦఺
    • MongoDB͕٧·ΔorམͪΔ
    • ͢΂ͯͷPVͰॻ͖ࠐΉΑ͏ͳͱ͜Ζʹ౤ೖͰ͖Δ΄Ͳ
    ৴པͰ͖ͳ͍

    View full-size slide

  13. ͦ͜Ͱfluentd

    View full-size slide

  14. ߏ੒
    • ΞϓϦέʔγϣϯ͸ϩʔΧϧͷϑΝΠϧʹॻ͚ͩ͘
    • fluentd͕ͦͬ͘Γͦͷ··సૹ͠ɺసૹઌͰ΋ϑΝΠϧ
    ʹॻ͖ࠐΉ
    • σʔϞϯͬΆ͍όονεΫϦϓτ͕ϑΝΠϧΛݹ͍ॱʹ
    ಡΜͰ͍ͬͯॲཧ͢Δ
    • exec_filterʹ͢Δख΋͕͋ͬͨɺॾʑͷཧ༝͔ΒPHP
    ͰϑΝΠϧΛtailͯ͠ϙδγϣϯͷ؅ཧͳͲ΋͍ͯ͠Δ

    View full-size slide

  15. ར఺ʢ̍ʣ
    • ϢʔβʔͷϦΫΤετΛॲཧ͍ͯ͠Δؒʹ௨৴͕ൃੜ͠
    ͳ͍
    • ϑΝΠϧ΁ͷॻ͖ࠐΈ͸γʔέϯγϟϧͳͷͰઈରʹ٧
    ·Βͳ͍
    • సૹઌ΋ಉ༷
    • ϑΝΠϧγεςϜ͸Ұ൪ރΕͯΔετϨʔδ

    View full-size slide

  16. ར఺ʢ̎ʣ
    • ॲཧͨ͠ޙ΋ϩά͕࢒Δ
    • ԿΒ͔ͷΤϥʔͰόον͕͏·࣮͘ߦͰ͖͍ͯͳ͔ͬ
    ͨͱͯ͠΋ɺݩʹ໭ͬͯ΍Γ௚ͤΔ
    • ࣦഊ͢Δͱઌʹਐ·ࣦͣഊ͠ଓ͚ΔΑ͏ʹͳ͍ͬͯͯɺ
    ରࡦͯ͠σϓϩΠ͢ΔͱਐΜͰ͘ΕΔ

    View full-size slide

  17. ར఺ʢ̏ʣ
    • సૹઌΛΞΫςΟϒɾελϯόΠߏ੒ʹͰ͖Δ
    • ྆ํಉ͡ߏ੒Ͱಉ͡όον͕ಈ͍͍ͯΔ

    View full-size slide

  18. • “When an active node goes down, the standby node is
    promoted to an active node.”
    • http://docs.fluentd.org/articles/out_forward#standby

    name host1
    host 192.168.xx.1
    standby


    name host2
    host 192.168.xx.2

    View full-size slide

  19. ·ͱΊ
    • fluentdͱϑΝΠϧγεςϜΛ࢖ͬͯ৴པੑͷߴ͍େن໛
    ΩϡʔγεςϜΛߏஙͨ͠
    • ϑΝΠϧʹγʔέϯγϟϧʹॻ͖ࠐΉ
    • 2೥Ҏ্ී௨ʹಈ͍͍ͯͯɺಉ͡࢓૊Ͱ·ͩ·ͩϩάΛ
    ૹͬͯ΋໰୊ͳͦ͞͏
    • ϑΝΠϧ͕࢒͍ͬͯΔ҆৺ײ

    View full-size slide