ログ・係数集約と可視化・分析

Fluentd,Embulk,ElasticStack6.0

Transcript

1. ϩάɾ܎਺ू໿ͱՄࢹԽɾ෼ੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
2. Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 2
3. Fluentdͱ͸ — Fluentd͸γϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 3
4. Fluentdͱ͸ — ʮCNCF(Cloud Native Computing Foundation)ʯͷ؅ཧ͢ΔϓϩδΣΫτ — Kubernetes ΍ Prometheus ͱ͍ͬͨΫϥ΢υωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetes؀ڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
5. Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯ͸v1.1.3 - 2018/04/03 — FluentdͷίΞιϑτ΢ΣΞ — ϓϥάΠϯ͸ݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠৔߹͸FluentdຊମΛ࢖͏ — td-agent — Fluentdʹ֤छϓϥάΠϯ΍Ruby؀ڥΛ૊ΈࠐΜͩύοέʔδ — ओཁ؀ڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — Linux΁ͷΠϯετʔϧ΋༰қ — αϙʔτOSʹ੍໿͕͋Δ͕ґଘؔ܎ͷ໰୊͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷ͸td-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
6. Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDB౳ʹ౉͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛ͸ߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ๷͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛ஗ΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲ΋Ͱ͖Δɻ 3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔Βήʔτ΢ΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
7. ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛ΋ॏෳ΋ڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 7
8. ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε඼࣭؅ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε ͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ౤͛Δଆ͕Ұ౓Ͱ౤͛Δ͜ͱ ౤͛Δଆ͕Ұ౓Ͱ౤͛Δ͜ͱ ౤͛Δଆɺड͚औΔଆͱ΋ʹ Ұ౓Ͱ഑৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
9. v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer - ҎԼͷΑ͏ͳ໰୊͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
10. v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter, Storage, Buffer - վળ఺ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
11. v0.14Ҏ߱ͷόʔδϣϯ v1.0͸v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯ͸v1.1.3 - 2018/04/03 td-agent3͸2017೥ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 11
12. v0.12ͱv1 — v0.12 APIΛ࢖༻͢ΔϓϥάΠϯ͸ɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1͸ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม׵͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ — Fluentd v1.0ͷ৽ػೳ͸ɺ৽͍͠APIΛ࢖༻͢ΔϓϥάΠϯͰͷΈ࢖༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ͸৽͍͠APIΛ࢖༻͢ΔϓϥάΠϯ͸ಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
13. v0.12ͱv1ͷઃఆͷҧ͍ v1͸outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ࢖͍ͬͯΔ # v1 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
14. fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱ͸σʔλ͕invalidͳͷʹretry͔͚ͯ΋ҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛௚ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹ͸retry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
15. fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset, table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
16. Fluentd v1.2ʹ͸retryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput olugin͸ɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ secondary Λ࢖͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠࡟আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
17. όοϑΝઃܭ(v0.12) Input͔ΒOutput΁৘ใ͕౉͞ΕΔ࢓૊Έʹ͓͍ͯ OutputଆͰ͸BufferͱQueueͱ͍͏࢓૊Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍࢓૊Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
18. όοϑΝઃܭ(v0.12) — ࠷ॳʹ৘ใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦෼ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ஝͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit Λ௒͑ͨ৔߹ԡ͠ग़͞ΕΔ৔߹ͱ flush_interval Λܦաͨ͠৔߹౉͞ΕΔέʔε — ͦΕͧΕઃఆ͢Ε͹ॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
19. όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ಺༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit chunck࠷େαΠζ bufferqueuelimit Queue಺chunck࠷େ਺ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
20. όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit 8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ஝͑Δ flush_interval 60s # Buffer͔ΒQueueʹ౉͞ΕΔ৔߹60ඵͨͬͨΒத਎ͷchunckΛQueueʹ౉͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ஝͑ΔDiskͷ༰ྔ΍ϝϞϦͷαΠζ͸ɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨ΋ͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ਺͚ͩ͜ͷ͔͚߹Θͤͨ஋͕ඞཁʹͳΔͷͰ஫ҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
21. όοϑΝઃܭ(v1) ಺෦తʹ͸ɺόοϑΝϓϥάΠϯʹ͸ɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹ଴ػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ෼཭͞Εͨ৔ॴ ͕͋Γ·͢ɻ ৽͘͠࡞੒͞Εͨ͢΂ͯͷνϟϯΫ͸ɺεςʔδ͔Β։࢝͠ɺ࣌ؒ಺ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering ঢ়ଶ — queued:flush଴ͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
22. όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
23. Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — Φʔϓϯιʔεͷฒྻ෼ࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
24. EmbulkͷϢʔεέʔε — աڈ෼ͷ৘ใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 24
25. FluentdͱEmbulkͷ࢖͍෼͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ؂ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू — ϦΞϧλΠϜੑͷߴ͍෼ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍΋ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ೔͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμ΢ϯϩʔυ — DWH΁ͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
26. όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream - Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3೥͝ͱͷϦϦʔεΛ໨ඪ - ػೳϦϦʔε͸ɺ6ϲ݄͝ͱΛ໨ඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
27. όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕࢖༻͞Ε͍ͯͳ͍৔߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕଎͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
28. embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨஌ɺޓ׵ੑʹؔ͢Δ௨஌ ͳͲ։ൃऀ͔ΒͷΞφ΢ϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
29. ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱ෼ੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats - X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
30. ֤ίϯϙʔωϯτͷ໾ׂ — Elasticsearch ͢΂ͯͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱ෼ੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛ೚ҙͷܗࣜͰूத؅ཧ — Beats Filebeat͸ɺαʔόʔ͔ΒLogstash΍ElasticsearchʹϩάϑΝΠϧΛ഑৴͢ΔͨΊʹߏங͞ΕͨBeat Metricbeat͸ɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOS΍αʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔ؂ࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺ؂ࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛ௥Ճ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
31. ͓໾ཱͪ৘ใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd v1.0ͷ׆༻๏ fluentd ͷجૅ஌ࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31