fluentdによる大規模キュー設計

by Atsushi Takayama

Slide 1

Slide 1 text

ﬂuentdʹΑΔ େن໛Ωϡʔઃܭ A large scale queue design with ﬂuentd

Slide 2

Slide 2 text

ࣗݾ঺հ • ߴࢁ Թ @edvakf • pixivͷອըࣄۀͷ։ൃ੹೚ऀ • ීஈॻ͍ͯΔͷ͸Scala΍PHPɺͨ·ʹJS΍Ruby΍Go • ࠷ۙϦʔυΤϯδχΞͱ͍͏ݞॻʹ • ձࣾͷٕज़બ୒ʹ੹೚Λ࣋ͭ • ʮࣾ಺ͷΤϯδχΞ͕޾ͤʹಇͨ͘Ίʹ͸ʯΛߟ࣮͑ݱ͢Δۀ

Slide 3

Slide 3 text

ී௨ͷ ΢ΣϒΞϓϦ+δϣϒΩϡʔ

Slide 4

Slide 4 text

Sidekiq

Slide 5

Slide 5 text

Resque Resque

Slide 6

Slide 6 text

MySQL Q4M PHP PHP PHP PHP Q4M batch

Slide 7

Slide 7 text

MongoDB PHP PHP PHP PHP Mongo batch

Slide 8

Slide 8 text

ྑ͍఺ • Sidekiq΍Resque͸൚༻δϣϒΩϡʔ • Rubyͷϝιου୯ҐͰ؆୯ʹδϣϒʹͰ͖Δ • ޿͘࢖ΘΕ͍ͯΔ • ΋͠Redis΍MongoDBͳͲΛطʹ࢖͍ͬͯΔͳΒ৽͍͠ ϛυϧ΢ΣΞΛಋೖ͠ͳͯ͘ྑ͍

Slide 9

Slide 9 text

΋͠ • 1෼ؒʹ10ສճͷδϣϒΛ࣮ߦ͠ͳ͍ͱ͍͚ͳ͔ͬͨΒ • ਺࣌ؒલͱ͔਺೔લͷঢ়ଶ·Ͱר͖໭ͬͯδϣϒΛ࣮ߦ ͠௚͞ͳ͍ͱ͍͚ͳ͔ͬͨΒ

Slide 10

Slide 10 text

ΞΫηεղੳ www.pixiv.net

Slide 11

Slide 11 text

࠷ॳʹߟ͑ͨҊ

Slide 12

Slide 12 text

MongoDB PHP PHP PHP PHP Mongo batch

Slide 13

Slide 13 text

ݒ೦఺ • MongoDB͕٧·ΔorམͪΔ • ͢΂ͯͷPVͰॻ͖ࠐΉΑ͏ͳͱ͜Ζʹ౤ೖͰ͖Δ΄Ͳ ৴པͰ͖ͳ͍

Slide 14

Slide 14 text

ͦ͜Ͱﬂuentd

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

ߏ੒ • ΞϓϦέʔγϣϯ͸ϩʔΧϧͷϑΝΠϧʹॻ͚ͩ͘ • ﬂuentd͕ͦͬ͘Γͦͷ··సૹ͠ɺసૹઌͰ΋ϑΝΠϧ ʹॻ͖ࠐΉ • σʔϞϯͬΆ͍όονεΫϦϓτ͕ϑΝΠϧΛݹ͍ॱʹ ಡΜͰ͍ͬͯॲཧ͢Δ • exec_ﬁlterʹ͢Δख΋͕͋ͬͨɺॾʑͷཧ༝͔ΒPHP ͰϑΝΠϧΛtailͯ͠ϙδγϣϯͷ؅ཧͳͲ΋͍ͯ͠Δ

Slide 17

Slide 17 text

ར఺ʢ̍ʣ • ϢʔβʔͷϦΫΤετΛॲཧ͍ͯ͠Δؒʹ௨৴͕ൃੜ͠ ͳ͍ • ϑΝΠϧ΁ͷॻ͖ࠐΈ͸γʔέϯγϟϧͳͷͰઈରʹ٧ ·Βͳ͍ • సૹઌ΋ಉ༷ • ϑΝΠϧγεςϜ͸Ұ൪ރΕͯΔετϨʔδ

Slide 18

Slide 18 text

ར఺ʢ̎ʣ • ॲཧͨ͠ޙ΋ϩά͕࢒Δ • ԿΒ͔ͷΤϥʔͰόον͕͏·࣮͘ߦͰ͖͍ͯͳ͔ͬ ͨͱͯ͠΋ɺݩʹ໭ͬͯ΍Γ௚ͤΔ • ࣦഊ͢Δͱઌʹਐ·ࣦͣഊ͠ଓ͚ΔΑ͏ʹͳ͍ͬͯͯɺ ରࡦͯ͠σϓϩΠ͢ΔͱਐΜͰ͘ΕΔ

Slide 19

Slide 19 text

ར఺ʢ̏ʣ • సૹઌΛΞΫςΟϒɾελϯόΠߏ੒ʹͰ͖Δ • ྆ํಉ͡ߏ੒Ͱಉ͡όον͕ಈ͍͍ͯΔ

Slide 20

Slide 20 text

• “When an active node goes down, the standby node is promoted to an active node.” • http://docs.ﬂuentd.org/articles/out_forward#standby name host1 host 192.168.xx.1 standby name host2 host 192.168.xx.2

Slide 21

Slide 21 text

·ͱΊ • ﬂuentdͱϑΝΠϧγεςϜΛ࢖ͬͯ৴པੑͷߴ͍େن໛ ΩϡʔγεςϜΛߏஙͨ͠ • ϑΝΠϧʹγʔέϯγϟϧʹॻ͖ࠐΉ • 2೥Ҏ্ී௨ʹಈ͍͍ͯͯɺಉ͡࢓૊Ͱ·ͩ·ͩϩάΛ ૹͬͯ΋໰୊ͳͦ͞͏ • ϑΝΠϧ͕࢒͍ͬͯΔ҆৺ײ