Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amaz...
Search
Shinji Fujimoto
October 08, 2016
Technology
3
2.6k
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration
Developers.IO 2016 in FUKUOKA
Shinji Fujimoto
October 08, 2016
Tweet
Share
More Decks by Shinji Fujimoto
See All by Shinji Fujimoto
AWS のサービスを活用して CI/CD #akibaaws
shinjifujimoto
0
18k
Amazon Elasticsearch Service の使いドコロ in 福岡
shinjifujimoto
1
730
Amazon Elasticsearch Service の使いドコロ
shinjifujimoto
3
54k
AWSのマネージドサービスを活用したログ可視化
shinjifujimoto
1
3.2k
はじめてのIoT
shinjifujimoto
0
5.1k
Beats開発の始め方 #cmdevio
shinjifujimoto
2
2.3k
Other Decks in Technology
See All in Technology
RapidPen: AIエージェントによる高度なペネトレーションテスト自動化の研究開発
laysakura
1
400
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
12
2k
Tiptapで実現する堅牢で柔軟なエディター開発
kirik
1
140
Bliki (ja), and the Cathedral, and the Bazaar
koic
8
1.5k
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
1
160
CSPヘッダー導入で実現するWebサイトの多層防御:今すぐ試せる設定例と運用知見
llamakko
1
250
MCPと認可まわりの話 / mcp_and_authorization
convto
2
270
Datasets for Critical Operations by Dataform
kimujun
0
110
Shadow DOM & Security - Exploring the boundary between light and shadow
masatokinugawa
0
740
Kiro Hookを Terraformで検証
ao_inoue
0
130
増え続ける脆弱性に立ち向かう: 事前対策と優先度づけによる 持続可能な脆弱性管理 / Confronting the Rise of Vulnerabilities: Sustainable Management Through Proactive Measures and Prioritization
nttcom
1
210
モバイルゲームの開発を支える基盤の歩み ~再現性のある開発ラインを量産する秘訣~
qualiarts
0
570
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
54
11k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Rails Girls Zürich Keynote
gr2m
95
14k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Facilitating Awesome Meetings
lara
54
6.5k
Docker and Python
trallard
45
3.5k
We Have a Design System, Now What?
morganepeng
53
7.7k
Become a Pro
speakerdeck
PRO
29
5.4k
Building Applications with DynamoDB
mza
95
6.5k
Designing Experiences People Love
moore
142
24k
How to Think Like a Performance Engineer
csswizardry
25
1.8k
Transcript
Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘
Profile • Who ౻ຊ ਅ࢘ Ϋϥεϝιουגࣜձࣾ • Work AWSͷಋೖࢧԉɺϒϩά •
Interest Elastic Stack
ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ •
AWSʹ͓͚ΔϩάՄࢹԽɾੳͷϢʔεέʔε
ϩάͷ׆༻ͱʁ
͋Δͷӡ༻ݱ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ͍ͳ͊ɻɻ -# 8FC"1 %#
͋Δͷӡ༻ݱ Ϣʔβʔ γεςϜཧऀ %#୲ "104୲ /8୲ Կ͔શମతʹϖʔδ͕͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐͯ
͋Δͷӡ༻ݱ γεςϜཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜѱ͍ 42-ͷΫΤϦͳ͍ ݁ہݪҼɾɾʁ ɾ ɾ ɾ -#
8FC"1 %#
ϩά׆༻ ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
ϩά࣮ͷ • ΞϓϦέʔγϣϯϩάͷ࣮ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON
• LTSV
AWSʹ͓͚Δϩά׆༻
AWSͷϩάऩू ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
AWSͷϩάऩू S3ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3
EMR Data Pipeline Billing
AWSͷϩάऩू CloudWatch Logsͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC
Flow Log Lambda CloudWatch Logs IoT
AWSͷϩάऩू CloudTrailͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ CloudTrail
AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks
Management Console AWS
AWSͷϩάऩू AWS Config ͦͷଞ
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ɾɾɾ ՄࢹԽ ੳ ଞଟ
AWSͷϩάू $MPVE8BUDI-PHT ϩάετϦʔϜͱͯ͠ू 4 ϑΝΠϧͱͯ͠ू
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάࢹɾ௨ $MPVE8BUDI-PHT Ωʔϫʔυࢹ $MPVE8BUDI ݅ʹΑΔݕ 4/4 ௨ -BNCEB ΧελϚΠζ௨
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάՄࢹԽɾੳ 2VJDL4JUF &MBTUJDTFBSDI4FSWJDF
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
Amazon Elasticsearch Service
Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮
Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ
Elasticsearch/Kibanaͷߏ • Elasticsearch on EC2 • Elastic Cloud • Amazon
Elasticsearch Service
Elasticsearch on EC2 ࣗલͰOS͔Βཧ͢Δ • ϝϦοτ • ࣗ༝ • ηΩϡϦςΟੑ
• σϝϦοτ • ӡ༻ɺอक
Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक
• ༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺͯߴ͍
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
• ߏஙͷ؆ུԽ • ϚϧνAZஔ • Master / Data • αΠζࢦఆ
• ӡ༻ͷ؆ུԽ • ਫฏ/ਨεέʔϧ • োͷࣗಈݕ • ࣗಈϊʔυަ • ࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ
ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script
• Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap
Ϣʔεέʔε
Ϣʔεέʔε̍ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ
Կ͕Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? •
DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -#
8FC"1 %#
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 γεςϜཧऀ &4 ूɾՄࢹԽɾੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ IUUQE 3%4GPS.Z42- ग़ྗઌ 4 'JMF 5BCMFPS'JMF ΞΫηεํ๏
"84"1* ϑΝΠϧΞΫηε 42-ΫΤϦ PS "84"1* ܗࣜ εϖʔε۠Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ຖPSຖ ϦΞϧλΠϜ ϦΞϧλΠϜ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ EC2 Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ
• ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI-PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ϩάࢹ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ ELB Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ίετ &-# 4 &-# 4 -BNCEB
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ RDS for MySQL Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2ͷෛՙ • ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI &WFOU
Demo
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI &WFOU
͋Δͷӡ༻ݱΛAWSʹஔ͖͑
ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯεੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯεੳ •
EC2୯ҐͷύϑΥʔϚϯεੳ
Ϣʔεέʔε̍.̑ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ ɹʢServerless൛ʣ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ "1*(BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI
Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηεੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE8BUDI -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI-PHTͷઃఆʹΑΓࣗಈੜ ιʔείʔυ"84͕ఏڙ
ෆਖ਼ΞΫηεੳ • IAMϢʔβʔͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ͑ͨՄࢹԽ
Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹظؒอଘɺ౷߹ੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱʹ ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$
CloudWatchϝτϦΫε • 2िؒΛ͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ ΤϦΞνϟʔτͳͲͷՄࢹԽ • Ҏ֎ͷѻ͍
• ॊೈͳ݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ
Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4ग़ྗ &-#ͱಉ͡Α͏ʹ࣮Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮Մ
CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδੳ • Ωϟογϡώοτੳ • ΞΫηεҬੳ • ར༻ऀͷ/OSੳ •
ෆਖ਼ΞΫηεੳ ʢࢀߟURLʣ http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/
·ͱΊ • AWSͷαʔϏεͷଟ͘ϩάΛग़ྗ͢Δ • ϩάͷతอͰͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ •
Amazon Elasticsearch ServiceͰ͋Εɺ $20/month͔Β࢝ΊΒΕ·͢