Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amaz...
Search
Shinji Fujimoto
October 08, 2016
Technology
3
2.7k
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration
Developers.IO 2016 in FUKUOKA
Shinji Fujimoto
October 08, 2016
Tweet
Share
More Decks by Shinji Fujimoto
See All by Shinji Fujimoto
AWS のサービスを活用して CI/CD #akibaaws
shinjifujimoto
0
18k
Amazon Elasticsearch Service の使いドコロ in 福岡
shinjifujimoto
1
740
Amazon Elasticsearch Service の使いドコロ
shinjifujimoto
3
54k
AWSのマネージドサービスを活用したログ可視化
shinjifujimoto
1
3.3k
はじめてのIoT
shinjifujimoto
0
5.1k
Beats開発の始め方 #cmdevio
shinjifujimoto
2
2.4k
Other Decks in Technology
See All in Technology
GopherCon Tour 概略
logica0419
2
170
GC25 Recap+: Advancing Go Garbage Collection with Green Tea
logica0419
1
360
20250929_QaaS_vol20
mura_shin
0
110
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
330
AI ReadyなData PlatformとしてのAutonomous Databaseアップデート
oracle4engineer
PRO
0
150
AI時代だからこそ考える、僕らが本当につくりたいスクラムチーム / A Scrum Team we really want to create in this AI era
takaking22
6
3k
コンテキストエンジニアリングとは? 考え方と応用方法
findy_eventslides
4
880
生成AIで「お客様の声」を ストーリーに変える 新潮流「Generative ETL」
ishikawa_satoru
1
290
空間を設計する力を考える / 20251004 Naoki Takahashi
shift_evolve
PRO
3
320
What is BigQuery?
aizack_harks
0
130
about #74462 go/token#FileSet
tomtwinkle
1
280
#普通の文系サラリーマンチャレンジ 自分でアプリ開発と電子工作を続けたら人生が変わった
tatsuya1970
0
940
Featured
See All Featured
Designing for humans not robots
tammielis
254
25k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
51k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.1k
Six Lessons from altMBA
skipperchong
28
4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
How to Think Like a Performance Engineer
csswizardry
27
2k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Code Review Best Practice
trishagee
72
19k
Transcript
Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘
Profile • Who ౻ຊ ਅ࢘ Ϋϥεϝιουגࣜձࣾ • Work AWSͷಋೖࢧԉɺϒϩά •
Interest Elastic Stack
ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ •
AWSʹ͓͚ΔϩάՄࢹԽɾੳͷϢʔεέʔε
ϩάͷ׆༻ͱʁ
͋Δͷӡ༻ݱ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ͍ͳ͊ɻɻ -# 8FC"1 %#
͋Δͷӡ༻ݱ Ϣʔβʔ γεςϜཧऀ %#୲ "104୲ /8୲ Կ͔શମతʹϖʔδ͕͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐͯ
͋Δͷӡ༻ݱ γεςϜཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜѱ͍ 42-ͷΫΤϦͳ͍ ݁ہݪҼɾɾʁ ɾ ɾ ɾ -#
8FC"1 %#
ϩά׆༻ ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
ϩά࣮ͷ • ΞϓϦέʔγϣϯϩάͷ࣮ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON
• LTSV
AWSʹ͓͚Δϩά׆༻
AWSͷϩάऩू ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
AWSͷϩάऩू S3ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3
EMR Data Pipeline Billing
AWSͷϩάऩू CloudWatch Logsͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC
Flow Log Lambda CloudWatch Logs IoT
AWSͷϩάऩू CloudTrailͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ CloudTrail
AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks
Management Console AWS
AWSͷϩάऩू AWS Config ͦͷଞ
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ɾɾɾ ՄࢹԽ ੳ ଞଟ
AWSͷϩάू $MPVE8BUDI-PHT ϩάετϦʔϜͱͯ͠ू 4 ϑΝΠϧͱͯ͠ू
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάࢹɾ௨ $MPVE8BUDI-PHT Ωʔϫʔυࢹ $MPVE8BUDI ݅ʹΑΔݕ 4/4 ௨ -BNCEB ΧελϚΠζ௨
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάՄࢹԽɾੳ 2VJDL4JUF &MBTUJDTFBSDI4FSWJDF
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
Amazon Elasticsearch Service
Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮
Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ
Elasticsearch/Kibanaͷߏ • Elasticsearch on EC2 • Elastic Cloud • Amazon
Elasticsearch Service
Elasticsearch on EC2 ࣗલͰOS͔Βཧ͢Δ • ϝϦοτ • ࣗ༝ • ηΩϡϦςΟੑ
• σϝϦοτ • ӡ༻ɺอक
Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक
• ༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺͯߴ͍
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
• ߏஙͷ؆ུԽ • ϚϧνAZஔ • Master / Data • αΠζࢦఆ
• ӡ༻ͷ؆ུԽ • ਫฏ/ਨεέʔϧ • োͷࣗಈݕ • ࣗಈϊʔυަ • ࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ
ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script
• Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap
Ϣʔεέʔε
Ϣʔεέʔε̍ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ
Կ͕Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? •
DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -#
8FC"1 %#
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 γεςϜཧऀ &4 ूɾՄࢹԽɾੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ IUUQE 3%4GPS.Z42- ग़ྗઌ 4 'JMF 5BCMFPS'JMF ΞΫηεํ๏
"84"1* ϑΝΠϧΞΫηε 42-ΫΤϦ PS "84"1* ܗࣜ εϖʔε۠Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ຖPSຖ ϦΞϧλΠϜ ϦΞϧλΠϜ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ EC2 Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ
• ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI-PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ϩάࢹ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ ELB Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ίετ &-# 4 &-# 4 -BNCEB
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ RDS for MySQL Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2ͷෛՙ • ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI &WFOU
Demo
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI &WFOU
͋Δͷӡ༻ݱΛAWSʹஔ͖͑
ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯεੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯεੳ •
EC2୯ҐͷύϑΥʔϚϯεੳ
Ϣʔεέʔε̍.̑ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ ɹʢServerless൛ʣ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ "1*(BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI
Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηεੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE8BUDI -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI-PHTͷઃఆʹΑΓࣗಈੜ ιʔείʔυ"84͕ఏڙ
ෆਖ਼ΞΫηεੳ • IAMϢʔβʔͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ͑ͨՄࢹԽ
Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹظؒอଘɺ౷߹ੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱʹ ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$
CloudWatchϝτϦΫε • 2िؒΛ͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ ΤϦΞνϟʔτͳͲͷՄࢹԽ • Ҏ֎ͷѻ͍
• ॊೈͳ݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ
Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4ग़ྗ &-#ͱಉ͡Α͏ʹ࣮Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮Մ
CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδੳ • Ωϟογϡώοτੳ • ΞΫηεҬੳ • ར༻ऀͷ/OSੳ •
ෆਖ਼ΞΫηεੳ ʢࢀߟURLʣ http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/
·ͱΊ • AWSͷαʔϏεͷଟ͘ϩάΛग़ྗ͢Δ • ϩάͷతอͰͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ •
Amazon Elasticsearch ServiceͰ͋Εɺ $20/month͔Β࢝ΊΒΕ·͢