teratailͷղੳج൫Λ EFKͰ࡞ͬͯ ͍Ζ͍Ζָ͍͠࿩ @ikuwow ϨόϨδʔζגࣜձࣾɹςΫϊϩδʔϝσΟΞϥϘ ΏͱΓੈ୅ΤϯδχΞަྲྀձʢ2016/03/04ʣ

ࣗݾ঺հ • ϨόϨδʔζגࣜձࣾɺςΫϊϩδʔϝ σΟΞϥϘɺteratailͷ։ൃͯ͠Δਓɻ • ֶੜͷ࣌εϩʔΨϯגࣜձࣾͰ1.5೥͙ Β͍Πϯλʔϯͯͨ͠ • ίʔυॻ͘ͱ͖͸PHPͰ͕͢ɺϑϩϯτ ΋Πϯϑϥ΋΍ͬͨΓ͍Ζ͍Ζ΍Γ· ͢ • ࠷ۙ΍ͬͨ͜ͱɿteratailͷϩάղੳج ൫࡞Δ @ikuwow

teratail ஌ͬͯΔਓʙʁ

teratail • ΤϯδχΞɾϓϩάϥ ϚͷͨΊͷQ&AαΠτ • ຖ೔࣭໰͕70-80݅ • ճ౴཰໿93% • 3/17ʹϢʔβʔձʮू ·ͬtailʯୈ࢛ճ։࠵༧ ఆ

ࠓ೔࿩͢͜ͱ • teratailͷϢʔβʔߦಈϩάΛEFKελοΫ (Elasticsearch, Fluentd, KibanaʣͰՄࢹԽ͢ Δ࢓૊Έ࡞ͬͨ • ָ͍͠ʂ • ਏ͍ʂʂ

ϢʔβʔͷߦಈΛݟ͍ͨʂ 1. ϦΞϧλΠϜʹ؂ࢹͯ͠ϦεΫݕ஌ͨ͠Γɺ ΧδϡΞϧʹ࠷ۙͷϢʔβʔͷಈ͖Λ௥ͬͨ Γ͍ͨ͠ʂ 2. KPIΛݟΔͷʹ࠷దԽͨ͠ܗͰσʔλΛ࣋ͬ ͯਂ͘ૣ͘ՄࢹԽ͍ͨ͠ ʢ͋ͱHiveQLॻ͘ͷΊΜͲ͍͘͢͠͝஗͍͔Βૣ͍ͷʹ͍ͨ͠ɾɾɾʣ

࡞ͬͨج൫ Amazon S3 Amazon Redshift ໼ҹ͸ϩάͷྲྀΕ 1. ϦΞϧλΠϜՄࢹԽ 2. ਂ͘ՄࢹԽ

΋͏গ͚ͩ͠ৄ͘͠ node.master: false node.data: false node.master: true node.data: true node.master: false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ

Fluentdͱ͸ • ϩάͷύʔεɺू໿Λ͢Δπʔ ϧ • Treasuredata੡ʢ೔ຊͰΘΓͱ ਓؾʣ • Α͘Logstashͱൺֱ͞ΕΔ • όοϑΝϦϯάݡͯ͘ɺ5෼͙ Β͍ࢭΊͯ΋શ͘໰୊ͳ͍

Elasticsearchͱ͸ • ࠷ۙྲྀߦΓͷશจݕࡧΤϯδϯɻ2ܥ ͕࠷৽ɻ • Elasticࣾ੡ʢLogstashͱಉ͡ʣ • ͖Ε͍ʹRESTfulͳAPIͰѻ͍΍͍͢ • ͱΓ͋͑ͣಉ͡ωοτϫʔΫʹஔ͍ ͓͚ͯ͹Ϋϥελ࡞ͬͯ͘ΕΔ • ࠷ۙAWS͕Elasticsearch Serviceͱ ͍͏ͷΛग़ͨ͠Γ

Kibanaͱ͸ • ElasticsearchΛόοΫͱ͠ ͯɺͦΕΒͷσʔλΛ௒͔ͬ ͜ྑ͘ՄࢹԽ͢Δπʔϧ • nodeΞϓϦέʔγϣϯͳͷ Ͱಋೖָ͕͘͢͝ • Ϛ΢εϙνϙνͰϩά͕ݟΒ ΕΔ

EFKελοΫͷಛ௃ • Πϯετʔϧ΍؅ཧ͕ൺֱతΧϯλϯ • Fluentd͸ϫϯϥΠφʔ͚ͩͰ͍͚Δ • Elasticsearch͸উखʹ͏·͍͜ͱΫϥελ࡞ͬͯ͘ΕΔ • Kibana΋ೖΕΔͷ؆୯ͩ͠ݟͨΒ͍͍ͩͨ࢖͑Δ • ͦͦ͜͜ރΕ͖ͯͨײ͋Δʁ • ࢼͯ͠ΈΔͱ࢖͍΍͕͙͢͢͞

࡞ͬͯԿ͕มΘ͔ͬͨʁ • ϩά͕௒؆୯ʹૣ͔ͬ͘͜Α͘ݟΒΕΔ༷ʹͳͬͨ • ࣌ؒͷ୹ॖ • νʔϜશһʹɺ͍ܰؾ࣋ͪͰ͍͍͢͢ϩάΛूܭɾՄࢹԽɾ෼ੳ͢ Δश׳͕͍ͭͯɺΠϕϯτࣄͷͨͼʹߦಈྔ૿͑ͨΓ͢Δͷ͕Έͯ ָ͍͠ • ϩάʹײ৘ҠೖͰ͖ΔΑ͏ʹͳͬͨʂ • ͓໰͍߹Θͤ࣌ʹࠔͬͯΔϢʔβʔͷߦಈΛ௥͑ΔΑ͏ʹͳͬͨ • όάͷݪҼ͕ɺϩά͔ΒϢʔβʔͷಈ͖Λ࠶ݱͯ͠ΈͨΒ൑໌ͨ͠

ָ͍͠ʂ

΄͔ʹ΍Γ͍ͨ͜ͱ • ApacheͷΤϥʔϩάɺΞΫηεϩάͷՄࢹԽɾ෼ੳ • fluentdͰTemplate͕༻ҙ͞Ε͍ͯΔͷͰ௒؆୯ʹͰ͖Δ • ϨεϙϯελΠϜͱ͔ग़͓ͯ͘͠ͱ΋ͬͱָ͍͠ • ΞϓϦέʔγϣϯϑϨʔϜϫʔΫͷΤϥʔϩά • Fluentd͸ෳ਺ߦϩά΋͍͚Δ • slow queryͷϩάݟͯΨϯΨϯѱ͍ΫΤϦΛ௵͢ ϦΞϧλΠϜੑ͕ٻΊΒΕΔ৘ใΛݟ΍͍͔͢Β͘͢͝Ԡ༻ར͘

ਏ͔ͬͨ͜ͱ • HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔϚοτΛਖ਼ن දݱͰॻ͘ͷͭΒ͍ • ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Index template͚ͭͨΒಡΊͳ͍ͬͯݴΘΕΔ • Autoscaling͕ݡ͗ͯ͢࢖ͬͯͨͷterminate͞Εͨ

@type tail path /home/ikuo.degawa/hogehoge.logs pos_file /tmp/hogehoge.logs.pos format /^(?

[^\t]+)\t(?[^\t]*)\t(?[^\t]*)\t(? [^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(? [^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(? [^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(? [^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(?[^\t]*)\t(?[^ \u0001]*)\u0001(?[^\u0001]*)\u0001(?[^\t]*)\t(?[^ \u0001]*)\u0001(?[^\u0001]*)\u0001(?[^\u0001]*)\u0001(? [^\u0001]*)\u0001(?[^\u0001]*)\u0001(?[^\u0001]*) \u0001(?[^\u0001]*)\u0001(?[^\t]*)\t(?[^\u0001]*) \u0001(?[^\u0001]*)\u0001(?[^\u0001]*)\u0001(?[^\t]*)\t(? [^\u0001]*)\u0001(?[^\u0001]*)\u0001(?.*)$/ tag mogmog-logs.gerogero

HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔ ϚοτΛਖ਼نදݱͰॻ͘ͷͭΒ͍

ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Kibanaͷ݅਺ͱɺcat hoge.log | wc -l ͨ݁͠Ռ ͕ҧ͏ʂʂ • lotateͨ͠ઌͷϑΝΠϧΛ ಡΈ࢝ΊΔλΠϛϯά͕ ஗͍ͱ͍͏࢓༷Λൃݟ • read_from_headΛ࢖ͬͨ Β࣏ͬͨ લͷ ࣍ͷ ͜ͷล͔ΒಡΜͰͨ

Index template͚ͭͨΒಡΊͳ ͍ͬͯݴΘΕΔ • index template: elasticsearchʹೖΔ ࣌ͷmappingΛࢦ ఆͰ͖Δ • index໊Λ৚݅ʹܕ ΛܾΊΒΕΔ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1 }, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": {

ύϑΥʔϚϯε্͕Δͱࢥͬͨ Βɾɾɾ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1 }, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": { • ࣮͸intΛظ଴͍ͯ͠Δͱ͜ ΖʹstringඈΜͰ͖ͨΓ͠ ͯͨʢϩάͷ࣮૷ϛεʣ • ϩά͕ೖͬͨͱ͖ʹΤϥʔ ు͍ͯͯɺfluentdͷόο ϑΝʹཷ·Γଓ͚ͯͨ • ݁ہnot_analyzedΛ͚ͭͨ ͷΈ

Autoscaling͕ݡ͗ͯ͢terminate ͞Εͨ ʂʁ

ʮavailability zone͕௥Ճ͞Ε͔ͨΒɺόϥϯε Αͯ͘͠Մ༻ੑ͋͛ΔͨΊʹ͍ͬ͜ফͯ࣍͠ͷ ݐͯΔΑʂʯ

ڭ܇ɾɾɾ • Fluentd͸͓ੈ࿩গͳͯ͘ࡁΉ͕ɺϩάͷಡΈ ํΛ஌ͬͱ͚ • Elasticsearch͸Elasticʹ͓͍ͯͨ͠΄͏͕͍͍ • Auto Scaling Group͸ݡ͍

·ͱΊ • KibanaͰϩάΛ͔ͬ͜Α͘ݟΒΕΔͱσʔλ ʹײ৘ҠೖͰ͖ΔΑ͏ʹͳΓɺνʔϜશһ͕ ϢʔβʔͷߦಈΛݟΒΕΔਓʹͳΕΔ • ָ͍͠

ฐࣾͰ͸ΤϯδχΞΛืूதͰ͢ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

͜ͷຊʹ͸͓ੈ࿩ʹͳΓ·ͨ͠ • ͍͍ຊͰ͢

@ikuwow