Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
teratailの解析基盤をEFKで作っていろいろ楽しい話
Search
ikuwow
March 04, 2016
Technology
0
860
teratailの解析基盤をEFKで作っていろいろ楽しい話
teratailの解析基盤をEFKで作っていろいろ楽しい話 @ ゆとりエンジニア交流会
ikuwow
March 04, 2016
Tweet
Share
More Decks by ikuwow
See All by ikuwow
Elasticsearch on EC2からAmazon Elasticsearch Serviceに 移行してだいぶ楽になった話
ikuwow
0
3.5k
意外と使える! Alibaba Cloud
ikuwow
0
230
UNIXという考え方
ikuwow
1
2k
技術書紹介 パーフェクトPHP
ikuwow
0
2.1k
みんなもMiddlemanで技術ブログ作って幸せになろう!
ikuwow
0
960
PHPサイバーテロの技法 書籍紹介
ikuwow
0
930
Other Decks in Technology
See All in Technology
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
910
Windows 11 で AWS Documentation MCP Server 接続実践/practical-aws-documentation-mcp-server-connection-on-windows-11
emiki
0
1k
【PHPカンファレンス 2025】PHPを愛するひとに伝えたい PHPとキャリアの話
tenshoku_draft
0
120
CursorによるPMO業務の代替 / Automating PMO Tasks with Cursor
motoyoshi_kakaku
0
330
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
220
How Community Opened Global Doors
hiroramos4
PRO
1
120
2025-06-26_Lightning_Talk_for_Lightning_Talks
_hashimo2
2
100
Snowflake Summit 2025 データエンジニアリング関連新機能紹介 / Snowflake Summit 2025 What's New about Data Engineering
tiltmax3
0
320
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
5
3.9k
Amazon Bedrockで実現する 新たな学習体験
kzkmaeda
2
610
Prox Industries株式会社 会社紹介資料
proxindustries
0
330
作曲家がボカロを使うようにPdMはAIを使え
itotaxi
0
120
Featured
See All Featured
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
How GitHub (no longer) Works
holman
314
140k
RailsConf 2023
tenderlove
30
1.1k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
930
GitHub's CSS Performance
jonrohan
1031
460k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
How STYLIGHT went responsive
nonsquared
100
5.6k
Become a Pro
speakerdeck
PRO
28
5.4k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
800
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Transcript
teratailͷղੳج൫Λ EFKͰ࡞ͬͯ ͍Ζ͍Ζָ͍͠ @ikuwow ϨόϨδʔζגࣜձࣾɹςΫϊϩδʔϝσΟΞϥϘ ΏͱΓੈΤϯδχΞަྲྀձʢ2016/03/04ʣ
ࣗݾհ • ϨόϨδʔζגࣜձࣾɺςΫϊϩδʔϝ σΟΞϥϘɺteratailͷ։ൃͯ͠Δਓɻ • ֶੜͷ࣌εϩʔΨϯגࣜձࣾͰ1.5͙ Β͍Πϯλʔϯͯͨ͠ • ίʔυॻ͘ͱ͖PHPͰ͕͢ɺϑϩϯτ ΠϯϑϥͬͨΓ͍Ζ͍ΖΓ·
͢ • ࠷ۙͬͨ͜ͱɿteratailͷϩάղੳج ൫࡞Δ @ikuwow
teratail ͬͯΔਓʙʁ
teratail • ΤϯδχΞɾϓϩάϥ ϚͷͨΊͷQ&AαΠτ • ຖ࣭͕70-80݅ • ճ93% • 3/17ʹϢʔβʔձʮू
·ͬtailʯୈ࢛ճ։࠵༧ ఆ
ࠓ͢͜ͱ • teratailͷϢʔβʔߦಈϩάΛEFKελοΫ (Elasticsearch, Fluentd, KibanaʣͰՄࢹԽ͢ ΔΈ࡞ͬͨ • ָ͍͠ʂ •
ਏ͍ʂʂ
ϢʔβʔͷߦಈΛݟ͍ͨʂ 1. ϦΞϧλΠϜʹࢹͯ͠ϦεΫݕͨ͠Γɺ ΧδϡΞϧʹ࠷ۙͷϢʔβʔͷಈ͖Λͬͨ Γ͍ͨ͠ʂ 2. KPIΛݟΔͷʹ࠷దԽͨ͠ܗͰσʔλΛ࣋ͬ ͯਂ͘ૣ͘ՄࢹԽ͍ͨ͠ ʢ͋ͱHiveQLॻ͘ͷΊΜͲ͍͍͔͘͢͠͝Βૣ͍ͷʹ͍ͨ͠ɾɾɾʣ
࡞ͬͨج൫ Amazon S3 Amazon Redshift ҹϩάͷྲྀΕ 1. ϦΞϧλΠϜՄࢹԽ 2. ਂ͘ՄࢹԽ
͏গ͚ͩ͠ৄ͘͠ node.master: false node.data: false node.master: true node.data: true node.master:
false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ
Fluentdͱ • ϩάͷύʔεɺूΛ͢Δπʔ ϧ • TreasuredataʢຊͰΘΓͱ ਓؾʣ • Α͘Logstashͱൺֱ͞ΕΔ •
όοϑΝϦϯάݡͯ͘ɺ5͙ Β͍ࢭΊͯશ͘ͳ͍
Elasticsearchͱ • ࠷ۙྲྀߦΓͷશจݕࡧΤϯδϯɻ2ܥ ͕࠷৽ɻ • ElasticࣾʢLogstashͱಉ͡ʣ • ͖Ε͍ʹRESTfulͳAPIͰѻ͍͍͢ • ͱΓ͋͑ͣಉ͡ωοτϫʔΫʹஔ͍
͓͚ͯΫϥελ࡞ͬͯ͘ΕΔ • ࠷ۙAWS͕Elasticsearch Serviceͱ ͍͏ͷΛग़ͨ͠Γ
Kibanaͱ • ElasticsearchΛόοΫͱ͠ ͯɺͦΕΒͷσʔλΛ͔ͬ ͜ྑ͘ՄࢹԽ͢Δπʔϧ • nodeΞϓϦέʔγϣϯͳͷ Ͱಋೖָ͕͘͢͝ • ϚεϙνϙνͰϩά͕ݟΒ
ΕΔ
EFKελοΫͷಛ • Πϯετʔϧཧ͕ൺֱతΧϯλϯ • FluentdϫϯϥΠφʔ͚ͩͰ͍͚Δ • Elasticsearchউखʹ͏·͍͜ͱΫϥελ࡞ͬͯ͘ΕΔ • KibanaೖΕΔͷ؆୯ͩ͠ݟͨΒ͍͍ͩͨ͑Δ •
ͦͦ͜͜ރΕ͖ͯͨײ͋Δʁ • ࢼͯ͠ΈΔͱ͍͕͙͢͢͞
࡞ͬͯԿ͕มΘ͔ͬͨʁ • ϩά͕؆୯ʹૣ͔ͬ͘͜Α͘ݟΒΕΔ༷ʹͳͬͨ • ࣌ؒͷॖ • νʔϜશһʹɺ͍ܰؾ࣋ͪͰ͍͍͢͢ϩάΛूܭɾՄࢹԽɾੳ͢ Δश׳͕͍ͭͯɺΠϕϯτࣄͷͨͼʹߦಈྔ૿͑ͨΓ͢Δͷ͕Έͯ ָ͍͠ •
ϩάʹײҠೖͰ͖ΔΑ͏ʹͳͬͨʂ • ͓͍߹Θͤ࣌ʹࠔͬͯΔϢʔβʔͷߦಈΛ͑ΔΑ͏ʹͳͬͨ • όάͷݪҼ͕ɺϩά͔ΒϢʔβʔͷಈ͖Λ࠶ݱͯ͠ΈͨΒ໌ͨ͠
ָ͍͠ʂ
΄͔ʹΓ͍ͨ͜ͱ • ApacheͷΤϥʔϩάɺΞΫηεϩάͷՄࢹԽɾੳ • fluentdͰTemplate͕༻ҙ͞Ε͍ͯΔͷͰ؆୯ʹͰ͖Δ • ϨεϙϯελΠϜͱ͔ग़͓ͯ͘͠ͱͬͱָ͍͠ • ΞϓϦέʔγϣϯϑϨʔϜϫʔΫͷΤϥʔϩά •
Fluentdෳߦϩά͍͚Δ • slow queryͷϩάݟͯΨϯΨϯѱ͍ΫΤϦΛ௵͢ ϦΞϧλΠϜੑ͕ٻΊΒΕΔใΛݟ͍͔͢Β͘͢͝Ԡ༻ར͘
ਏ͔ͬͨ͜ͱ • HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔϚοτΛਖ਼ن දݱͰॻ͘ͷͭΒ͍ • ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Index
template͚ͭͨΒಡΊͳ͍ͬͯݴΘΕΔ • Autoscaling͕ݡ͗ͯͬͯͨ͢ͷterminate͞Εͨ
<source> @type tail path /home/ikuo.degawa/hogehoge.logs pos_file /tmp/hogehoge.logs.pos format /^(?<dt>[^\t]+)\t(?<site_id>[^\t]*)\t(?<action>[^\t]*)\t(? <option>[^\t]*)\t(?<user_id>[^\t]*)\t(?<session_cookie>[^\t]*)\t(?
<storage_cookie>[^\t]*)\t(?<view_type>[^\t]*)\t(?<user_agent>[^\t]*)\t(? <page_id>[^\t]*)\t(?<url>[^\t]*)\t(?<time>[^\t]*)\t(?<ip>[^\t]*)\t(? <segment>[^\t]*)\t(?<var>[^\t]*)\t(?<view>[^\t]*)\t(?<act>[^\t]*)\t(?<post0>[^ \u0001]*)\u0001(?<post1>[^\u0001]*)\u0001(?<post2>[^\t]*)\t(?<search0>[^ \u0001]*)\u0001(?<search1>[^\u0001]*)\u0001(?<search2>[^\u0001]*)\u0001(? <search3>[^\u0001]*)\u0001(?<search4>[^\u0001]*)\u0001(?<search5>[^\u0001]*) \u0001(?<search6>[^\u0001]*)\u0001(?<search7>[^\t]*)\t(?<user0>[^\u0001]*) \u0001(?<user1>[^\u0001]*)\u0001(?<user2>[^\u0001]*)\u0001(?<user3>[^\t]*)\t(? <other0>[^\u0001]*)\u0001(?<other1>[^\u0001]*)\u0001(?<other2>.*)$/ tag mogmog-logs.gerogero </source> HadoopʹೖΕ͍ͯͨಠࣗͷϑΥʔ ϚοτΛਖ਼نදݱͰॻ͘ͷͭΒ͍
ϩά͕1.3%͙Β͍ܽଛ͢Δ => ࣏ͬͨ • Kibanaͷ݅ͱɺcat hoge.log | wc -l ͨ݁͠Ռ
͕ҧ͏ʂʂ • lotateͨ͠ઌͷϑΝΠϧΛ ಡΈ࢝ΊΔλΠϛϯά͕ ͍ͱ͍͏༷Λൃݟ • read_from_headΛͬͨ Β࣏ͬͨ લͷ ࣍ͷ ͜ͷล͔ΒಡΜͰͨ
Index template͚ͭͨΒಡΊͳ ͍ͬͯݴΘΕΔ • index template: elasticsearchʹೖΔ ࣌ͷmappingΛࢦ ఆͰ͖Δ •
index໊Λ݅ʹܕ ΛܾΊΒΕΔ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1 }, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": {
ύϑΥʔϚϯε্͕Δͱࢥͬͨ Βɾɾɾ { "templates": “awesomelog-*", "settings": { "number_of_shards" : 1
}, "mappings": { "awesomelogs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "act0" : { "type" : "integer" }, "act1" : { "type" : "integer" }, "act10" : { "type" : "string", "index": "not_analyzed" }, "act11" : { "type" : "string" }, "act2" : { "type" : "integer" }, "act3" : { "type" : "integer" }, "act4" : { "type" : "string" }, "act5" : { "type": "multi_field", "fields": { • ࣮intΛظ͍ͯ͠Δͱ͜ ΖʹstringඈΜͰ͖ͨΓ͠ ͯͨʢϩάͷ࣮ϛεʣ • ϩά͕ೖͬͨͱ͖ʹΤϥʔ ు͍ͯͯɺfluentdͷόο ϑΝʹཷ·Γଓ͚ͯͨ • ݁ہnot_analyzedΛ͚ͭͨ ͷΈ
Autoscaling͕ݡ͗ͯ͢terminate ͞Εͨ ʂʁ
ʮavailability zone͕Ճ͞Ε͔ͨΒɺόϥϯε Αͯ͘͠Մ༻ੑ͋͛ΔͨΊʹ͍ͬ͜ফͯ࣍͠ͷ ݐͯΔΑʂʯ
ڭ܇ɾɾɾ • Fluentd͓ੈগͳͯ͘ࡁΉ͕ɺϩάͷಡΈ ํΛͬͱ͚ • ElasticsearchElasticʹ͓͍ͯͨ͠΄͏͕͍͍ • Auto Scaling Groupݡ͍
·ͱΊ • KibanaͰϩάΛ͔ͬ͜Α͘ݟΒΕΔͱσʔλ ʹײҠೖͰ͖ΔΑ͏ʹͳΓɺνʔϜશһ͕ ϢʔβʔͷߦಈΛݟΒΕΔਓʹͳΕΔ • ָ͍͠
ฐࣾͰΤϯδχΞΛืूதͰ͢ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
͜ͷຊʹ͓ੈʹͳΓ·ͨ͠ • ͍͍ຊͰ͢
@ikuwow