Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration
Search
Shinji Fujimoto
October 08, 2016
Technology
3
2.5k
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration
Developers.IO 2016 in FUKUOKA
Shinji Fujimoto
October 08, 2016
Tweet
Share
More Decks by Shinji Fujimoto
See All by Shinji Fujimoto
AWS のサービスを活用して CI/CD #akibaaws
shinjifujimoto
0
16k
Amazon Elasticsearch Service の使いドコロ in 福岡
shinjifujimoto
1
670
Amazon Elasticsearch Service の使いドコロ
shinjifujimoto
3
51k
AWSのマネージドサービスを活用したログ可視化
shinjifujimoto
1
2.9k
はじめてのIoT
shinjifujimoto
0
4.6k
Beats開発の始め方 #cmdevio
shinjifujimoto
2
2.1k
Other Decks in Technology
See All in Technology
OpenTelemetry を使ったトレースエグザンプラーの活用 / otel-trace-exemplar
k6s4i53rx
2
630
AIQ株式会社 エンジニア向け会社紹介資料
aiqlab
0
360
WebアプリケーションにおけるPDOの使い方入門 / phpcon odawara 2024
meihei3
2
420
カオナビの利用実績をアウトカムへつなげる旅 / example-of-data-management-startup-in-kaonavi
kaonavi
0
120
反実仮想機械学習とは何か
usaito
PRO
6
1.6k
AWS を使う上で知っておきたいオンプレミス知識/aws-on-premise-essentials
emiki
1
4.1k
Amplify Gen2を 拡張してみよう JAWS-UG北陸新幹線 ( 福井開催 ) 2024-04-06/Let's extend Amplify Gen2
fossamagna
0
280
Aurora MySQL v3(MySQL8.0互換)の オンラインDDLの罠挙動を全バージョンで検証した
yutakikai
0
150
エンタープライズ環境下での Active Directory の運用 TIPS
tamaiyutaro
1
1.5k
小さな開発会社がWebサービスを作る理由
polidog
PRO
0
140
Tableau事例紹介 / Tableau Case Study of Eureka
kazuya_araki_tokyo
1
170
Microsoft Cloudで開発ライフサイクルを保護する
kkamegawa
0
140
Featured
See All Featured
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
13
1.5k
Art, The Web, and Tiny UX
lynnandtonic
288
19k
Building a Scalable Design System with Sketch
lauravandoore
455
32k
Robots, Beer and Maslow
schacon
PRO
154
7.9k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
24
2.3k
Building Flexible Design Systems
yeseniaperezcruz
318
37k
jQuery: Nuts, Bolts and Bling
dougneiner
59
7.1k
Building Adaptive Systems
keathley
29
1.8k
Visualization
eitanlees
135
14k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
219
21k
The Illustrated Children's Guide to Kubernetes
chrisshort
28
46k
Automating Front-end Workflow
addyosmani
1354
200k
Transcript
Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘
Profile • Who ౻ຊ ਅ࢘ Ϋϥεϝιουגࣜձࣾ • Work AWSͷಋೖࢧԉɺϒϩά •
Interest Elastic Stack
ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ •
AWSʹ͓͚ΔϩάՄࢹԽɾੳͷϢʔεέʔε
ϩάͷ׆༻ͱʁ
͋Δͷӡ༻ݱ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ͍ͳ͊ɻɻ -# 8FC"1 %#
͋Δͷӡ༻ݱ Ϣʔβʔ γεςϜཧऀ %#୲ "104୲ /8୲ Կ͔શମతʹϖʔδ͕͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐͯ
͋Δͷӡ༻ݱ γεςϜཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜѱ͍ 42-ͷΫΤϦͳ͍ ݁ہݪҼɾɾʁ ɾ ɾ ɾ -#
8FC"1 %#
ϩά׆༻ ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
ϩά࣮ͷ • ΞϓϦέʔγϣϯϩάͷ࣮ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON
• LTSV
AWSʹ͓͚Δϩά׆༻
AWSͷϩάऩू ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
AWSͷϩάऩू S3ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3
EMR Data Pipeline Billing
AWSͷϩάऩू CloudWatch Logsͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC
Flow Log Lambda CloudWatch Logs IoT
AWSͷϩάऩू CloudTrailͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ CloudTrail
AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks
Management Console AWS
AWSͷϩάऩू AWS Config ͦͷଞ
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ɾɾɾ ՄࢹԽ ੳ ଞଟ
AWSͷϩάू $MPVE8BUDI-PHT ϩάετϦʔϜͱͯ͠ू 4 ϑΝΠϧͱͯ͠ू
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάࢹɾ௨ $MPVE8BUDI-PHT Ωʔϫʔυࢹ $MPVE8BUDI ݅ʹΑΔݕ 4/4 ௨ -BNCEB ΧελϚΠζ௨
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάՄࢹԽɾੳ 2VJDL4JUF &MBTUJDTFBSDI4FSWJDF
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
Amazon Elasticsearch Service
Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮
Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ
Elasticsearch/Kibanaͷߏ • Elasticsearch on EC2 • Elastic Cloud • Amazon
Elasticsearch Service
Elasticsearch on EC2 ࣗલͰOS͔Βཧ͢Δ • ϝϦοτ • ࣗ༝ • ηΩϡϦςΟੑ
• σϝϦοτ • ӡ༻ɺอक
Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक
• ༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺͯߴ͍
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
• ߏஙͷ؆ུԽ • ϚϧνAZஔ • Master / Data • αΠζࢦఆ
• ӡ༻ͷ؆ུԽ • ਫฏ/ਨεέʔϧ • োͷࣗಈݕ • ࣗಈϊʔυަ • ࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ
ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script
• Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap
Ϣʔεέʔε
Ϣʔεέʔε̍ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ
Կ͕Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? •
DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -#
8FC"1 %#
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 γεςϜཧऀ &4 ूɾՄࢹԽɾੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ IUUQE 3%4GPS.Z42- ग़ྗઌ 4 'JMF 5BCMFPS'JMF ΞΫηεํ๏
"84"1* ϑΝΠϧΞΫηε 42-ΫΤϦ PS "84"1* ܗࣜ εϖʔε۠Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ຖPSຖ ϦΞϧλΠϜ ϦΞϧλΠϜ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ EC2 Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ
• ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI-PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ϩάࢹ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ ELB Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ίετ &-# 4 &-# 4 -BNCEB
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ RDS for MySQL Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2ͷෛՙ • ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI &WFOU
Demo
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI &WFOU
͋Δͷӡ༻ݱΛAWSʹஔ͖͑
ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯεੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯεੳ •
EC2୯ҐͷύϑΥʔϚϯεੳ
Ϣʔεέʔε̍.̑ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ ɹʢServerless൛ʣ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ "1*(BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI
Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηεੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE8BUDI -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI-PHTͷઃఆʹΑΓࣗಈੜ ιʔείʔυ"84͕ఏڙ
ෆਖ਼ΞΫηεੳ • IAMϢʔβʔͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ͑ͨՄࢹԽ
Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹظؒอଘɺ౷߹ੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱʹ ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$
CloudWatchϝτϦΫε • 2िؒΛ͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ ΤϦΞνϟʔτͳͲͷՄࢹԽ • Ҏ֎ͷѻ͍
• ॊೈͳ݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ
Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4ग़ྗ &-#ͱಉ͡Α͏ʹ࣮Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮Մ
CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδੳ • Ωϟογϡώοτੳ • ΞΫηεҬੳ • ར༻ऀͷ/OSੳ •
ෆਖ਼ΞΫηεੳ ʢࢀߟURLʣ http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/
·ͱΊ • AWSͷαʔϏεͷଟ͘ϩάΛग़ྗ͢Δ • ϩάͷతอͰͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ •
Amazon Elasticsearch ServiceͰ͋Εɺ $20/month͔Β࢝ΊΒΕ·͢