teratailの解析基盤をEFKで作っていろいろ楽しい話

Transcript

1. teratailͷղੳج൫Λ EFKͰ࡞ͬͯ ͍Ζ͍Ζָ͍͠࿩ @ikuwow ϨόϨδʔζגࣜձࣾɹςΫϊϩδʔϝσΟΞϥϘ ΏͱΓੈ୅ΤϯδχΞަྲྀձʢ2016/03/04ʣ

2. ࣗݾ঺հ • ϨόϨδʔζגࣜձࣾɺςΫϊϩδʔϝ σΟΞϥϘɺteratailͷ։ൃͯ͠Δਓɻ • ֶੜͷ࣌εϩʔΨϯגࣜձࣾͰ1.5೥͙ Β͍Πϯλʔϯͯͨ͠ • ίʔυॻ͘ͱ͖͸PHPͰ͕͢ɺϑϩϯτ ΋Πϯϑϥ΋΍ͬͨΓ͍Ζ͍Ζ΍Γ·

   ͢ • ࠷ۙ΍ͬͨ͜ͱɿteratailͷϩάղੳج ൫࡞Δ @ikuwow

3. teratail ஌ͬͯΔਓʙʁ

4. teratail • ΤϯδχΞɾϓϩάϥ ϚͷͨΊͷQ&AαΠτ • ຖ೔࣭໰͕70-80݅ • ճ౴཰໿93% • 3/17ʹϢʔβʔձʮू

   ·ͬtailʯୈ࢛ճ։࠵༧ ఆ

5. ࠓ೔࿩͢͜ͱ • teratailͷϢʔβʔߦಈϩάΛEFKελοΫ (Elasticsearch, Fluentd, KibanaʣͰՄࢹԽ͢ Δ࢓૊Έ࡞ͬͨ • ָ͍͠ʂ •

   ਏ͍ʂʂ

6. ϢʔβʔͷߦಈΛݟ͍ͨʂ 1. ϦΞϧλΠϜʹ؂ࢹͯ͠ϦεΫݕ஌ͨ͠Γɺ ΧδϡΞϧʹ࠷ۙͷϢʔβʔͷಈ͖Λ௥ͬͨ Γ͍ͨ͠ʂ 2. KPIΛݟΔͷʹ࠷దԽͨ͠ܗͰσʔλΛ࣋ͬ ͯਂ͘ૣ͘ՄࢹԽ͍ͨ͠ ʢ͋ͱHiveQLॻ͘ͷΊΜͲ͍͘͢͠͝஗͍͔Βૣ͍ͷʹ͍ͨ͠ɾɾɾʣ

7. ࡞ͬͨج൫ Amazon S3 Amazon Redshift ໼ҹ͸ϩάͷྲྀΕ 1. ϦΞϧλΠϜՄࢹԽ 2. ਂ͘ՄࢹԽ

8. ΋͏গ͚ͩ͠ৄ͘͠ node.master: false node.data: false node.master: true node.data: true node.master:

   false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ

9. Fluentdͱ͸ • ϩάͷύʔεɺू໿Λ͢Δπʔ ϧ • Treasuredata੡ʢ೔ຊͰΘΓͱ ਓؾʣ • Α͘Logstashͱൺֱ͞ΕΔ •

   όοϑΝϦϯάݡͯ͘ɺ5෼͙ Β͍ࢭΊͯ΋શ͘໰୊ͳ͍

10. Elasticsearchͱ͸ • ࠷ۙྲྀߦΓͷશจݕࡧΤϯδϯɻ2ܥ ͕࠷৽ɻ • Elasticࣾ੡ʢLogstashͱಉ͡ʣ • ͖Ε͍ʹRESTfulͳAPIͰѻ͍΍͍͢ • ͱΓ͋͑ͣಉ͡ωοτϫʔΫʹஔ͍

    ͓͚ͯ͹Ϋϥελ࡞ͬͯ͘ΕΔ • ࠷ۙAWS͕Elasticsearch Serviceͱ ͍͏ͷΛग़ͨ͠Γ

11. Kibanaͱ͸ • ElasticsearchΛόοΫͱ͠ ͯɺͦΕΒͷσʔλΛ௒͔ͬ ͜ྑ͘ՄࢹԽ͢Δπʔϧ • nodeΞϓϦέʔγϣϯͳͷ Ͱಋೖָ͕͘͢͝ • Ϛ΢εϙνϙνͰϩά͕ݟΒ

    ΕΔ

12. EFKελοΫͷಛ௃ • Πϯετʔϧ΍؅ཧ͕ൺֱతΧϯλϯ • Fluentd͸ϫϯϥΠφʔ͚ͩͰ͍͚Δ • Elasticsearch͸উखʹ͏·͍͜ͱΫϥελ࡞ͬͯ͘ΕΔ • Kibana΋ೖΕΔͷ؆୯ͩ͠ݟͨΒ͍͍ͩͨ࢖͑Δ •

    ͦͦ͜͜ރΕ͖ͯͨײ͋Δʁ • ࢼͯ͠ΈΔͱ࢖͍΍͕͙͢͢͞

13. ࡞ͬͯԿ͕มΘ͔ͬͨʁ • ϩά͕௒؆୯ʹૣ͔ͬ͘͜Α͘ݟΒΕΔ༷ʹͳͬͨ • ࣌ؒͷ୹ॖ • νʔϜશһʹɺ͍ܰؾ࣋ͪͰ͍͍͢͢ϩάΛूܭɾՄࢹԽɾ෼ੳ͢ Δश׳͕͍ͭͯɺΠϕϯτࣄͷͨͼʹߦಈྔ૿͑ͨΓ͢Δͷ͕Έͯ ָ͍͠ •

    ϩάʹײ৘ҠೖͰ͖ΔΑ͏ʹͳͬͨʂ • ͓໰͍߹Θͤ࣌ʹࠔͬͯΔϢʔβʔͷߦಈΛ௥͑ΔΑ͏ʹͳͬͨ • όάͷݪҼ͕ɺϩά͔ΒϢʔβʔͷಈ͖Λ࠶ݱͯ͠ΈͨΒ൑໌ͨ͠

14. ָ͍͠ʂ

15. ΄͔ʹ΍Γ͍ͨ͜ͱ • ApacheͷΤϥʔϩάɺΞΫηεϩάͷՄࢹԽɾ෼ੳ • fluentdͰTemplate͕༻ҙ͞Ε͍ͯΔͷͰ௒؆୯ʹͰ͖Δ • ϨεϙϯελΠϜͱ͔ग़͓ͯ͘͠ͱ΋ͬͱָ͍͠ • ΞϓϦέʔγϣϯϑϨʔϜϫʔΫͷΤϥʔϩά •

    Fluentd͸ෳ਺ߦϩά΋͍͚Δ • slow queryͷϩάݟͯΨϯΨϯѱ͍ΫΤϦΛ௵͢ ϦΞϧλΠϜੑ͕ٻΊΒΕΔ৘ใΛݟ΍͍͔͢Β͘͢͝Ԡ༻ར͘

16. ਏ͔ͬͨ͜ͱ • HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔϚοτΛਖ਼ن දݱͰॻ͘ͷͭΒ͍ • ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Index

    template͚ͭͨΒಡΊͳ͍ͬͯݴΘΕΔ • Autoscaling͕ݡ͗ͯ͢࢖ͬͯͨͷterminate͞Εͨ

17. <source> @type tail path /home/ikuo.degawa/hogehoge.logs pos_file /tmp/hogehoge.logs.pos format /^(?<dt>[^\t]+)\t(?<site_id>[^\t]*)\t(?<action>[^\t]*)\t(? <option>[^\t]*)\t(?<user_id>[^\t]*)\t(?<session_cookie>[^\t]*)\t(?

    <storage_cookie>[^\t]*)\t(?<view_type>[^\t]*)\t(?<user_agent>[^\t]*)\t(? <page_id>[^\t]*)\t(?<url>[^\t]*)\t(?<time>[^\t]*)\t(?<ip>[^\t]*)\t(? <segment>[^\t]*)\t(?<var>[^\t]*)\t(?<view>[^\t]*)\t(?<act>[^\t]*)\t(?<post0>[^ \u0001]*)\u0001(?<post1>[^\u0001]*)\u0001(?<post2>[^\t]*)\t(?<search0>[^ \u0001]*)\u0001(?<search1>[^\u0001]*)\u0001(?<search2>[^\u0001]*)\u0001(? <search3>[^\u0001]*)\u0001(?<search4>[^\u0001]*)\u0001(?<search5>[^\u0001]*) \u0001(?<search6>[^\u0001]*)\u0001(?<search7>[^\t]*)\t(?<user0>[^\u0001]*) \u0001(?<user1>[^\u0001]*)\u0001(?<user2>[^\u0001]*)\u0001(?<user3>[^\t]*)\t(? <other0>[^\u0001]*)\u0001(?<other1>[^\u0001]*)\u0001(?<other2>.*)$/ tag mogmog-logs.gerogero </source> HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔ ϚοτΛਖ਼نදݱͰॻ͘ͷͭΒ͍

18. ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Kibanaͷ݅਺ͱɺcat hoge.log | wc -l ͨ݁͠Ռ

    ͕ҧ͏ʂʂ • lotateͨ͠ઌͷϑΝΠϧΛ ಡΈ࢝ΊΔλΠϛϯά͕ ஗͍ͱ͍͏࢓༷Λൃݟ • read_from_headΛ࢖ͬͨ Β࣏ͬͨ લͷ ࣍ͷ ͜ͷล͔ΒಡΜͰͨ

19. Index template͚ͭͨΒಡΊͳ ͍ͬͯݴΘΕΔ • index template: elasticsearchʹೖΔ ࣌ͷmappingΛࢦ ఆͰ͖Δ •

    index໊Λ৚݅ʹܕ ΛܾΊΒΕΔ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1 }, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": {

20. ύϑΥʔϚϯε্͕Δͱࢥͬͨ Βɾɾɾ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1

    }, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": { • ࣮͸intΛظ଴͍ͯ͠Δͱ͜ ΖʹstringඈΜͰ͖ͨΓ͠ ͯͨʢϩάͷ࣮૷ϛεʣ • ϩά͕ೖͬͨͱ͖ʹΤϥʔ ు͍ͯͯɺfluentdͷόο ϑΝʹཷ·Γଓ͚ͯͨ • ݁ہnot_analyzedΛ͚ͭͨ ͷΈ

21. Autoscaling͕ݡ͗ͯ͢terminate ͞Εͨ ʂʁ

22. ʮavailability zone͕௥Ճ͞Ε͔ͨΒɺόϥϯε Αͯ͘͠Մ༻ੑ͋͛ΔͨΊʹ͍ͬ͜ফͯ࣍͠ͷ ݐͯΔΑʂʯ

23. ڭ܇ɾɾɾ • Fluentd͸͓ੈ࿩গͳͯ͘ࡁΉ͕ɺϩάͷಡΈ ํΛ஌ͬͱ͚ • Elasticsearch͸Elasticʹ͓͍ͯͨ͠΄͏͕͍͍ • Auto Scaling Group͸ݡ͍

24. ·ͱΊ • KibanaͰϩάΛ͔ͬ͜Α͘ݟΒΕΔͱσʔλ ʹײ৘ҠೖͰ͖ΔΑ͏ʹͳΓɺνʔϜશһ͕ ϢʔβʔͷߦಈΛݟΒΕΔਓʹͳΕΔ • ָ͍͠

25. ฐࣾͰ͸ΤϯδχΞΛืूதͰ͢ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

26. ͜ͷຊʹ͸͓ੈ࿩ʹͳΓ·ͨ͠ • ͍͍ຊͰ͢

27. @ikuwow