Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ログ・係数集約と可視化・分析
Search
Shuhei Ozawa
April 26, 2018
Technology
0
140
ログ・係数集約と可視化・分析
Fluentd,Embulk,ElasticStack6.0
Shuhei Ozawa
April 26, 2018
Tweet
Share
More Decks by Shuhei Ozawa
See All by Shuhei Ozawa
Amebaアフィリエイト基盤の GKEアーキテクチャと マイクロサービス
ozashu
0
190
production_ready_envoy
ozashu
2
1.1k
Python for web architectures
ozashu
0
910
PyQではじめるPython
ozashu
0
420
インフラエンジニアのWEBアプリ入門
ozashu
1
8k
Other Decks in Technology
See All in Technology
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
950
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
570
高速なプロダクト開発を実現、創業期から掲げるエンタープライズアーキテクチャ
kawauso
2
7.8k
Tech-Verse 2025 Global CTO Session
lycorptech_jp
PRO
0
1.6k
Connect 100+を支える技術
kanyamaguc
0
190
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
330
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
lycorptech_jp
PRO
0
150
CI/CD/IaC 久々に0から環境を作ったらこうなりました
kaz29
1
230
Witchcraft for Memory
pocke
1
750
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
330
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
3
280
WordPressから ヘッドレスCMSへ! Storyblokへの移行プロセス
nyata
0
430
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Done Done
chrislema
184
16k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
How STYLIGHT went responsive
nonsquared
100
5.6k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
For a Future-Friendly Web
brad_frost
179
9.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Facilitating Awesome Meetings
lara
54
6.4k
Transcript
ϩάɾूͱՄࢹԽɾੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 2
Fluentdͱ — Fluentdγϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 3
Fluentdͱ — ʮCNCF(Cloud Native Computing Foundation)ʯͷཧ͢ΔϓϩδΣΫτ — Kubernetes Prometheus
ͱ͍ͬͨΫϥυωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetesڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 — FluentdͷίΞιϑτΣΞ —
ϓϥάΠϯݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠߹FluentdຊମΛ͏ — td-agent — Fluentdʹ֤छϓϥάΠϯRubyڥΛΈࠐΜͩύοέʔδ — ओཁڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — LinuxͷΠϯετʔϧ༰қ — αϙʔτOSʹ੍͕͋Δ͕ґଘؔͷ͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷtd-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDBʹ͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲͰ͖Δɻ
3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔ΒήʔτΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛॏෳڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa
Shuhei 7
ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε࣭ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε
͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆɺड͚औΔଆͱʹ ҰͰ৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer
- ҎԼͷΑ͏ͳ͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,
Storage, Buffer - վળ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
v0.14Ҏ߱ͷόʔδϣϯ v1.0v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 td-agent32017ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 11
v0.12ͱv1 — v0.12 APIΛ༻͢ΔϓϥάΠϯɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ
— Fluentd v1.0ͷ৽ػೳɺ৽͍͠APIΛ༻͢ΔϓϥάΠϯͰͷΈ༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ৽͍͠APIΛ༻͢ΔϓϥάΠϯಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
v0.12ͱv1ͷઃఆͷҧ͍ v1outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ͍ͬͯΔ # v1 <match pattern> @type foo database db1
apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱσʔλ͕invalidͳͷʹretry͔͚ͯҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹretry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset,
table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
Fluentd v1.2ʹretryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput oluginɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ
secondary Λ͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
όοϑΝઃܭ(v0.12) Input͔ΒOutputใ͕͞ΕΔΈʹ͓͍ͯ OutputଆͰBufferͱQueueͱ͍͏Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
όοϑΝઃܭ(v0.12) — ࠷ॳʹใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit
Λ͑ͨ߹ԡ͠ग़͞ΕΔ߹ͱ flush_interval Λܦաͨ͠߹͞ΕΔέʔε — ͦΕͧΕઃఆ͢ΕॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit
chunck࠷େαΠζ bufferqueuelimit Queuechunck࠷େ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit
8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ͑Δ flush_interval 60s # Buffer͔ΒQueueʹ͞ΕΔ߹60ඵͨͬͨΒதͷchunckΛQueueʹ͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ͑ΔDiskͷ༰ྔϝϞϦͷαΠζɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ͚ͩ͜ͷ͔͚߹Θ͕ͤͨඞཁʹͳΔͷͰҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
όοϑΝઃܭ(v1) ෦తʹɺόοϑΝϓϥάΠϯʹɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ͞Εͨॴ ͕͋Γ·͢ɻ ৽͘͠࡞͞Εͨͯ͢ͷνϟϯΫɺεςʔδ͔Β։࢝͠ɺ࣌ؒ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering
ঢ়ଶ — queued:flushͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — ΦʔϓϯιʔεͷฒྻࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
EmbulkͷϢʔεέʔε — աڈͷใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 24
FluentdͱEmbulkͷ͍͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू —
ϦΞϧλΠϜੑͷߴ͍ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμϯϩʔυ — DWHͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream
- Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3͝ͱͷϦϦʔεΛඪ - ػೳϦϦʔεɺ6ϲ݄͝ͱΛඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕༻͞Ε͍ͯͳ͍߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨ɺޓੑʹؔ͢Δ௨ ͳͲ։ൃऀ͔ΒͷΞφϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats
- X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
֤ίϯϙʔωϯτͷׂ — Elasticsearch ͯ͢ͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛҙͷܗࣜͰूதཧ — Beats Filebeatɺαʔόʔ͔ΒLogstashElasticsearchʹϩάϑΝΠϧΛ৴͢ΔͨΊʹߏங͞ΕͨBeat
Metricbeatɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOSαʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛՃ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
ཱ͓ͪใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd
v1.0ͷ׆༻๏ fluentd ͷجૅࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31