Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ログ・係数集約と可視化・分析
Search
Shuhei Ozawa
April 26, 2018
Technology
0
140
ログ・係数集約と可視化・分析
Fluentd,Embulk,ElasticStack6.0
Shuhei Ozawa
April 26, 2018
Tweet
Share
More Decks by Shuhei Ozawa
See All by Shuhei Ozawa
Amebaアフィリエイト基盤の GKEアーキテクチャと マイクロサービス
ozashu
0
190
production_ready_envoy
ozashu
2
1.1k
Python for web architectures
ozashu
0
900
PyQではじめるPython
ozashu
0
420
インフラエンジニアのWEBアプリ入門
ozashu
1
8k
Other Decks in Technology
See All in Technology
GitHub Copilot の概要
tomokusaba
1
150
KubeCon + CloudNativeCon Japan 2025 Recap
ren510dev
1
340
「Chatwork」の認証基盤の移行とログ活用によるプロダクト改善
kubell_hr
1
240
生成AI開発案件におけるClineの業務活用事例とTips
shinya337
0
200
mrubyと micro-ROSが繋ぐロボットの世界
kishima
3
390
WordPressから ヘッドレスCMSへ! Storyblokへの移行プロセス
nyata
0
370
20250625 Snowflake Summit 2025活用事例 レポート / Nowcast Snowflake Summit 2025 Case Study Report
kkuv
1
380
AI専用のリンターを作る #yumemi_patch
bengo4com
5
3k
ネットワーク保護はどう変わるのか?re:Inforce 2025最新アップデート解説
tokushun
0
170
OPENLOGI Company Profile
hr01
0
67k
KubeCon + CloudNativeCon Japan 2025 Recap by CA
ponkio_o
PRO
0
270
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
1.5k
Featured
See All Featured
Designing for humans not robots
tammielis
253
25k
A Tale of Four Properties
chriscoyier
160
23k
Designing Experiences People Love
moore
142
24k
The Pragmatic Product Professional
lauravandoore
35
6.7k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
Documentation Writing (for coders)
carmenintech
72
4.9k
We Have a Design System, Now What?
morganepeng
53
7.7k
Optimizing for Happiness
mojombo
379
70k
Navigating Team Friction
lara
187
15k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Transcript
ϩάɾूͱՄࢹԽɾੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 2
Fluentdͱ — Fluentdγϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 3
Fluentdͱ — ʮCNCF(Cloud Native Computing Foundation)ʯͷཧ͢ΔϓϩδΣΫτ — Kubernetes Prometheus
ͱ͍ͬͨΫϥυωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetesڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 — FluentdͷίΞιϑτΣΞ —
ϓϥάΠϯݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠߹FluentdຊମΛ͏ — td-agent — Fluentdʹ֤छϓϥάΠϯRubyڥΛΈࠐΜͩύοέʔδ — ओཁڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — LinuxͷΠϯετʔϧ༰қ — αϙʔτOSʹ੍͕͋Δ͕ґଘؔͷ͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷtd-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDBʹ͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲͰ͖Δɻ
3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔ΒήʔτΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛॏෳڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa
Shuhei 7
ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε࣭ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε
͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆɺड͚औΔଆͱʹ ҰͰ৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer
- ҎԼͷΑ͏ͳ͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,
Storage, Buffer - վળ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
v0.14Ҏ߱ͷόʔδϣϯ v1.0v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 td-agent32017ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 11
v0.12ͱv1 — v0.12 APIΛ༻͢ΔϓϥάΠϯɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ
— Fluentd v1.0ͷ৽ػೳɺ৽͍͠APIΛ༻͢ΔϓϥάΠϯͰͷΈ༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ৽͍͠APIΛ༻͢ΔϓϥάΠϯಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
v0.12ͱv1ͷઃఆͷҧ͍ v1outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ͍ͬͯΔ # v1 <match pattern> @type foo database db1
apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱσʔλ͕invalidͳͷʹretry͔͚ͯҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹretry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset,
table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
Fluentd v1.2ʹretryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput oluginɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ
secondary Λ͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
όοϑΝઃܭ(v0.12) Input͔ΒOutputใ͕͞ΕΔΈʹ͓͍ͯ OutputଆͰBufferͱQueueͱ͍͏Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
όοϑΝઃܭ(v0.12) — ࠷ॳʹใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit
Λ͑ͨ߹ԡ͠ग़͞ΕΔ߹ͱ flush_interval Λܦաͨ͠߹͞ΕΔέʔε — ͦΕͧΕઃఆ͢ΕॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit
chunck࠷େαΠζ bufferqueuelimit Queuechunck࠷େ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit
8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ͑Δ flush_interval 60s # Buffer͔ΒQueueʹ͞ΕΔ߹60ඵͨͬͨΒதͷchunckΛQueueʹ͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ͑ΔDiskͷ༰ྔϝϞϦͷαΠζɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ͚ͩ͜ͷ͔͚߹Θ͕ͤͨඞཁʹͳΔͷͰҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
όοϑΝઃܭ(v1) ෦తʹɺόοϑΝϓϥάΠϯʹɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ͞Εͨॴ ͕͋Γ·͢ɻ ৽͘͠࡞͞Εͨͯ͢ͷνϟϯΫɺεςʔδ͔Β։࢝͠ɺ࣌ؒ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering
ঢ়ଶ — queued:flushͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — ΦʔϓϯιʔεͷฒྻࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
EmbulkͷϢʔεέʔε — աڈͷใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 24
FluentdͱEmbulkͷ͍͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू —
ϦΞϧλΠϜੑͷߴ͍ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμϯϩʔυ — DWHͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream
- Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3͝ͱͷϦϦʔεΛඪ - ػೳϦϦʔεɺ6ϲ݄͝ͱΛඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕༻͞Ε͍ͯͳ͍߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨ɺޓੑʹؔ͢Δ௨ ͳͲ։ൃऀ͔ΒͷΞφϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats
- X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
֤ίϯϙʔωϯτͷׂ — Elasticsearch ͯ͢ͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛҙͷܗࣜͰूதཧ — Beats Filebeatɺαʔόʔ͔ΒLogstashElasticsearchʹϩάϑΝΠϧΛ৴͢ΔͨΊʹߏங͞ΕͨBeat
Metricbeatɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOSαʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛՃ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
ཱ͓ͪใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd
v1.0ͷ׆༻๏ fluentd ͷجૅࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31