Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ログ・係数集約と可視化・分析
Search
Shuhei Ozawa
April 26, 2018
Technology
0
110
ログ・係数集約と可視化・分析
Fluentd,Embulk,ElasticStack6.0
Shuhei Ozawa
April 26, 2018
Tweet
Share
More Decks by Shuhei Ozawa
See All by Shuhei Ozawa
Amebaアフィリエイト基盤の GKEアーキテクチャと マイクロサービス
ozashu
0
120
production_ready_envoy
ozashu
2
1k
Python for web architectures
ozashu
0
830
PyQではじめるPython
ozashu
0
380
インフラエンジニアのWEBアプリ入門
ozashu
1
7.9k
Other Decks in Technology
See All in Technology
Python と Snowflake はズッ友だょ!~ Snowflake の Python 関連機能をふりかえる ~
__allllllllez__
1
120
MapLibreとAmazon Location Service
dayjournal
1
150
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
250
VS CodeでAWSを操作しよう
smt7174
7
1.6k
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
910
APIファーストなプロダクトマネジメントの実践 〜SaaSus Platformでの例〜 / "Practicing API-First Product Management - An Example with SaaSus Platform
oztick139
0
100
複雑な構成要素を持つUIとの向き合い方 〜新・支出グラフでの実例〜 / B43 TECH TALK
nakamuuu
0
140
Google Cloud Next '24でブログを10本書いた方法と勉強会を沸かせた方法
yasumuusan
0
290
開発生産性大幅アップ!Postman VS Code拡張機能
nagix
2
370
Além do else! Categorizando Pokemóns com Pattern Matching no JavaScript
wmsbill
0
510
web-application-security
matsuihidetoshi
0
160
継続的な改善 x ⾮連続的な進化
sansantech
PRO
3
150
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
19
1.7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
How GitHub (no longer) Works
holman
304
140k
Statistics for Hackers
jakevdp
789
220k
Creatively Recalculating Your Daily Design Routine
revolveconf
210
11k
Visualization
eitanlees
136
14k
How to Ace a Technical Interview
jacobian
272
22k
A Modern Web Designer's Workflow
chriscoyier
689
190k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Navigating Team Friction
lara
178
13k
Transcript
ϩάɾूͱՄࢹԽɾੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 2
Fluentdͱ — Fluentdγϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 3
Fluentdͱ — ʮCNCF(Cloud Native Computing Foundation)ʯͷཧ͢ΔϓϩδΣΫτ — Kubernetes Prometheus
ͱ͍ͬͨΫϥυωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetesڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 — FluentdͷίΞιϑτΣΞ —
ϓϥάΠϯݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠߹FluentdຊମΛ͏ — td-agent — Fluentdʹ֤छϓϥάΠϯRubyڥΛΈࠐΜͩύοέʔδ — ओཁڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — LinuxͷΠϯετʔϧ༰қ — αϙʔτOSʹ੍͕͋Δ͕ґଘؔͷ͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷtd-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDBʹ͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲͰ͖Δɻ
3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔ΒήʔτΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛॏෳڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa
Shuhei 7
ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε࣭ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε
͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆɺड͚औΔଆͱʹ ҰͰ৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer
- ҎԼͷΑ͏ͳ͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,
Storage, Buffer - վળ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
v0.14Ҏ߱ͷόʔδϣϯ v1.0v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 td-agent32017ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 11
v0.12ͱv1 — v0.12 APIΛ༻͢ΔϓϥάΠϯɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ
— Fluentd v1.0ͷ৽ػೳɺ৽͍͠APIΛ༻͢ΔϓϥάΠϯͰͷΈ༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ৽͍͠APIΛ༻͢ΔϓϥάΠϯಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
v0.12ͱv1ͷઃఆͷҧ͍ v1outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ͍ͬͯΔ # v1 <match pattern> @type foo database db1
apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱσʔλ͕invalidͳͷʹretry͔͚ͯҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹretry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset,
table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
Fluentd v1.2ʹretryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput oluginɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ
secondary Λ͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
όοϑΝઃܭ(v0.12) Input͔ΒOutputใ͕͞ΕΔΈʹ͓͍ͯ OutputଆͰBufferͱQueueͱ͍͏Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
όοϑΝઃܭ(v0.12) — ࠷ॳʹใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit
Λ͑ͨ߹ԡ͠ग़͞ΕΔ߹ͱ flush_interval Λܦաͨ͠߹͞ΕΔέʔε — ͦΕͧΕઃఆ͢ΕॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit
chunck࠷େαΠζ bufferqueuelimit Queuechunck࠷େ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit
8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ͑Δ flush_interval 60s # Buffer͔ΒQueueʹ͞ΕΔ߹60ඵͨͬͨΒதͷchunckΛQueueʹ͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ͑ΔDiskͷ༰ྔϝϞϦͷαΠζɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ͚ͩ͜ͷ͔͚߹Θ͕ͤͨඞཁʹͳΔͷͰҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
όοϑΝઃܭ(v1) ෦తʹɺόοϑΝϓϥάΠϯʹɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ͞Εͨॴ ͕͋Γ·͢ɻ ৽͘͠࡞͞Εͨͯ͢ͷνϟϯΫɺεςʔδ͔Β։࢝͠ɺ࣌ؒ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering
ঢ়ଶ — queued:flushͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — ΦʔϓϯιʔεͷฒྻࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
EmbulkͷϢʔεέʔε — աڈͷใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 24
FluentdͱEmbulkͷ͍͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू —
ϦΞϧλΠϜੑͷߴ͍ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμϯϩʔυ — DWHͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream
- Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3͝ͱͷϦϦʔεΛඪ - ػೳϦϦʔεɺ6ϲ݄͝ͱΛඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕༻͞Ε͍ͯͳ͍߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨ɺޓੑʹؔ͢Δ௨ ͳͲ։ൃऀ͔ΒͷΞφϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats
- X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
֤ίϯϙʔωϯτͷׂ — Elasticsearch ͯ͢ͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛҙͷܗࣜͰूதཧ — Beats Filebeatɺαʔόʔ͔ΒLogstashElasticsearchʹϩάϑΝΠϧΛ৴͢ΔͨΊʹߏங͞ΕͨBeat
Metricbeatɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOSαʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛՃ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
ཱ͓ͪใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd
v1.0ͷ׆༻๏ fluentd ͷجૅࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31