Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ログ・係数集約と可視化・分析
Search
Shuhei Ozawa
April 26, 2018
Technology
0
140
ログ・係数集約と可視化・分析
Fluentd,Embulk,ElasticStack6.0
Shuhei Ozawa
April 26, 2018
Tweet
Share
More Decks by Shuhei Ozawa
See All by Shuhei Ozawa
Amebaアフィリエイト基盤の GKEアーキテクチャと マイクロサービス
ozashu
0
200
production_ready_envoy
ozashu
2
1.2k
Python for web architectures
ozashu
0
920
PyQではじめるPython
ozashu
0
430
インフラエンジニアのWEBアプリ入門
ozashu
1
8k
Other Decks in Technology
See All in Technology
[ JAWS-UG 東京 CommunityBuilders Night #2 ]SlackとAmazon Q Developerで 運用効率化を模索する
sh_fk2
3
430
allow_retry と Arel.sql / allow_retry and Arel.sql
euglena1215
1
170
共有と分離 - Compose Multiplatform "本番導入" の設計指針
error96num
2
570
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
S3アクセス制御の設計ポイント
tommy0124
3
200
複数サービスを支えるマルチテナント型Batch MLプラットフォーム
lycorptech_jp
PRO
1
380
5分でカオスエンジニアリングを分かった気になろう
pandayumi
0
240
AI開発ツールCreateがAnythingになったよ
tendasato
0
130
初めてAWSを使うときのセキュリティ覚書〜初心者支部編〜
cmusudakeisuke
1
250
Webブラウザ向け動画配信プレイヤーの 大規模リプレイスから得た知見と学び
yud0uhu
0
230
💡Ruby 川辺で灯すPicoRubyからの光
bash0c7
0
120
バイブスに「型」を!Kent Beckに学ぶ、AI時代のテスト駆動開発
amixedcolor
2
560
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
For a Future-Friendly Web
brad_frost
180
9.9k
Mobile First: as difficult as doing things right
swwweet
224
9.9k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
13k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Scaling GitHub
holman
463
140k
Practical Orchestrator
shlominoach
190
11k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Bash Introduction
62gerente
615
210k
Transcript
ϩάɾूͱՄࢹԽɾੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 2
Fluentdͱ — Fluentdγϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 3
Fluentdͱ — ʮCNCF(Cloud Native Computing Foundation)ʯͷཧ͢ΔϓϩδΣΫτ — Kubernetes Prometheus
ͱ͍ͬͨΫϥυωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetesڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 — FluentdͷίΞιϑτΣΞ —
ϓϥάΠϯݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠߹FluentdຊମΛ͏ — td-agent — Fluentdʹ֤छϓϥάΠϯRubyڥΛΈࠐΜͩύοέʔδ — ओཁڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — LinuxͷΠϯετʔϧ༰қ — αϙʔτOSʹ੍͕͋Δ͕ґଘؔͷ͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷtd-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDBʹ͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲͰ͖Δɻ
3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔ΒήʔτΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛॏෳڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa
Shuhei 7
ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε࣭ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε
͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆɺड͚औΔଆͱʹ ҰͰ৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer
- ҎԼͷΑ͏ͳ͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,
Storage, Buffer - վળ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
v0.14Ҏ߱ͷόʔδϣϯ v1.0v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 td-agent32017ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 11
v0.12ͱv1 — v0.12 APIΛ༻͢ΔϓϥάΠϯɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ
— Fluentd v1.0ͷ৽ػೳɺ৽͍͠APIΛ༻͢ΔϓϥάΠϯͰͷΈ༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ৽͍͠APIΛ༻͢ΔϓϥάΠϯಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
v0.12ͱv1ͷઃఆͷҧ͍ v1outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ͍ͬͯΔ # v1 <match pattern> @type foo database db1
apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱσʔλ͕invalidͳͷʹretry͔͚ͯҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹretry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset,
table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
Fluentd v1.2ʹretryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput oluginɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ
secondary Λ͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
όοϑΝઃܭ(v0.12) Input͔ΒOutputใ͕͞ΕΔΈʹ͓͍ͯ OutputଆͰBufferͱQueueͱ͍͏Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
όοϑΝઃܭ(v0.12) — ࠷ॳʹใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit
Λ͑ͨ߹ԡ͠ग़͞ΕΔ߹ͱ flush_interval Λܦաͨ͠߹͞ΕΔέʔε — ͦΕͧΕઃఆ͢ΕॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit
chunck࠷େαΠζ bufferqueuelimit Queuechunck࠷େ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit
8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ͑Δ flush_interval 60s # Buffer͔ΒQueueʹ͞ΕΔ߹60ඵͨͬͨΒதͷchunckΛQueueʹ͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ͑ΔDiskͷ༰ྔϝϞϦͷαΠζɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ͚ͩ͜ͷ͔͚߹Θ͕ͤͨඞཁʹͳΔͷͰҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
όοϑΝઃܭ(v1) ෦తʹɺόοϑΝϓϥάΠϯʹɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ͞Εͨॴ ͕͋Γ·͢ɻ ৽͘͠࡞͞Εͨͯ͢ͷνϟϯΫɺεςʔδ͔Β։࢝͠ɺ࣌ؒ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering
ঢ়ଶ — queued:flushͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — ΦʔϓϯιʔεͷฒྻࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
EmbulkͷϢʔεέʔε — աڈͷใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 24
FluentdͱEmbulkͷ͍͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू —
ϦΞϧλΠϜੑͷߴ͍ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμϯϩʔυ — DWHͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream
- Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3͝ͱͷϦϦʔεΛඪ - ػೳϦϦʔεɺ6ϲ݄͝ͱΛඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕༻͞Ε͍ͯͳ͍߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨ɺޓੑʹؔ͢Δ௨ ͳͲ։ൃऀ͔ΒͷΞφϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats
- X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
֤ίϯϙʔωϯτͷׂ — Elasticsearch ͯ͢ͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛҙͷܗࣜͰूதཧ — Beats Filebeatɺαʔόʔ͔ΒLogstashElasticsearchʹϩάϑΝΠϧΛ৴͢ΔͨΊʹߏங͞ΕͨBeat
Metricbeatɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOSαʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛՃ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
ཱ͓ͪใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd
v1.0ͷ׆༻๏ fluentd ͷجૅࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31