Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration

by Shinji Fujimoto

Slide 1

Slide 1 text

Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘

Slide 2

Slide 2 text

Proﬁle • Who  ౻ຊ ਅ࢘  Ϋϥεϝιουגࣜձࣾ • Work  AWSͷಋೖࢧԉɺϒϩά • Interest  Elastic Stack

Slide 3

Slide 3 text

ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ • AWSʹ͓͚ΔϩάՄࢹԽɾ෼ੳͷϢʔεέʔε

Slide 4

Slide 4 text

ϩάͷ׆༻ͱ͸ʁ

Slide 5

Slide 5 text

͋Δ೔ͷӡ༻ݱ৔ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ஗͍ͳ͊ɻɻ -# 8FC"1 %#

Slide 6

Slide 6 text

͋Δ೔ͷӡ༻ݱ৔ Ϣʔβʔ γεςϜ؅ཧऀ %#୲౰ "104୲౰ /8୲౰ Կ͔શମతʹϖʔδ͕஗͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐ΂ͯ

Slide 7

Slide 7 text

͋Δ೔ͷӡ༻ݱ৔ γεςϜ؅ཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜ͸ѱ͍ 42-ͷΫΤϦ͸໰୊ͳ͍ ݁ہݪҼ͸ɾɾʁ ɾ ɾ ɾ -# 8FC"1 %#

Slide 8

Slide 8 text

ϩά׆༻ ऩू ू໿ ɾɾɾ ɾɾɾ ՄࢹԽ ෼ੳ ؂ࢹ ௨஌

Slide 9

Slide 9 text

ϩά࣮૷ͷ࿩ • ΞϓϦέʔγϣϯϩάͷ࣮૷ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON • LTSV

Slide 10

Slide 10 text

AWSʹ͓͚Δϩά׆༻

Slide 11

Slide 11 text

AWSͷϩάऩू ऩू ू໿ ɾɾɾ ɾɾɾ ՄࢹԽ ෼ੳ ؂ࢹ ௨஌

Slide 12

Slide 12 text

AWSͷϩάऩू S3΁ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3 EMR Data Pipeline Billing

Slide 13

Slide 13 text

AWSͷϩάऩू CloudWatch Logs΁ͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC  Flow Log Lambda CloudWatch Logs IoT

Slide 14

Slide 14 text

AWSͷϩάऩू CloudTrail΁ͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ਺ CloudTrail

Slide 15

Slide 15 text

AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks Management Console AWS

Slide 16

Slide 16 text

AWSͷϩάऩू AWS Config ͦͷଞ

Slide 17

Slide 17 text

AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ɾɾɾ ՄࢹԽ ෼ੳ ଞଟ਺

Slide 18

Slide 18 text

AWSͷϩάू໿ $MPVE8BUDI-PHT ϩάετϦʔϜͱͯ͠ू໿ 4 ϑΝΠϧͱͯ͠ू໿

Slide 19

Slide 19 text

AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺

Slide 20

Slide 20 text

AWSͷϩά؂ࢹɾ௨஌ $MPVE8BUDI-PHT Ωʔϫʔυ؂ࢹ $MPVE8BUDI ৚݅ʹΑΔݕ஌ 4/4 ௨஌ -BNCEB ΧελϚΠζ௨஌

Slide 21

Slide 21 text

AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺

Slide 22

Slide 22 text

AWSͷϩάՄࢹԽɾ෼ੳ 2VJDL4JUF &MBTUJDTFBSDI4FSWJDF

Slide 23

Slide 23 text

AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺

Slide 24

Slide 24 text

Amazon Elasticsearch Service

Slide 25

Slide 25 text

Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮

Slide 26

Slide 26 text

Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜ෼ੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ

Slide 27

Slide 27 text

Elasticsearch/Kibanaͷߏ੒ • Elasticsearch on EC2 • Elastic Cloud • Amazon Elasticsearch Service

Slide 28

Slide 28 text

Elasticsearch on EC2 ࣗલͰOS͔Β؅ཧ͢Δ • ϝϦοτ • ࣗ༝౓ • ηΩϡϦςΟੑ • σϝϦοτ • ӡ༻ɺอक

Slide 29

Slide 29 text

Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक • ঎༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺ΂ͯߴ͍

Slide 30

Slide 30 text

Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍

Slide 31

Slide 31 text

• ߏஙͷ؆ུԽ • ϚϧνAZ഑ஔ • Master / Data෼཭ • αΠζࢦఆ • ӡ༻ͷ؆ུԽ • ਫฏ/ਨ௚εέʔϧ • ো֐ͷࣗಈݕ஌ • ࣗಈϊʔυަ׵ • ೔࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring

Slide 32

Slide 32 text

Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍

Slide 33

Slide 33 text

AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES

Slide 34

Slide 34 text

Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍

Slide 35

Slide 35 text

IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ

Slide 36

Slide 36 text

Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍

Slide 37

Slide 37 text

Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ୒ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script • Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap

Slide 38

Slide 38 text

Ϣʔεέʔε

Slide 39

Slide 39 text

Ϣʔεέʔε̍ ɹ͋Δ೔ͷӡ༻ݱ৔ͷ ɹύϑΥʔϚϯε෼ੳ

Slide 40

Slide 40 text

Կ͕஌Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? • DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?

Slide 41

Slide 41 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -# 8FC"1 %#

Slide 42

Slide 42 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ Ϣʔβʔ &-# &$ 3%4 γεςϜ؅ཧऀ &4 ू໿ɾՄࢹԽɾ෼ੳ

Slide 43

Slide 43 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ

Slide 44

Slide 44 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &-# &$ IUUQE 3%4GPS.Z42- ग़ྗઌ 4 'JMF 5BCMFPS'JMF ΞΫηεํ๏ "84"1* ϑΝΠϧΞΫηε 42-ΫΤϦ PS "84"1* ܗࣜ εϖʔε۠੾Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ෼ຖPS෼ຖ ϦΞϧλΠϜ ϦΞϧλΠϜ

Slide 45

Slide 45 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ EC2 Amazon ES

Slide 46

Slide 46 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • ௥Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI-PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2΁ͷ௿ෛՙ • ϩά؂ࢹ

Slide 47

Slide 47 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ ELB Amazon ES

Slide 48

Slide 48 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2΁ͷ௿ෛՙ • ௿ίετ &-# 4 &-# 4 -BNCEB

Slide 49

Slide 49 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ RDS for MySQL Amazon ES

Slide 50

Slide 50 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2΁ͷ௿ෛՙ • ௿ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI &WFOU

Slide 51

Slide 51 text

Demo

Slide 52

Slide 52 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI &WFOU

Slide 53

Slide 53 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑

Slide 54

Slide 54 text

ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯε෼ੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯε෼ੳ • EC2୯ҐͷύϑΥʔϚϯε෼ੳ

Slide 55

Slide 55 text

Ϣʔεέʔε̍.̑ ɹ͋Δ೔ͷӡ༻ݱ৔ͷ ɹύϑΥʔϚϯε෼ੳ ɹʢServerless൛ʣ

Slide 56

Slide 56 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ "1*(BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI

Slide 57

Slide 57 text

Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηε෼ੳ

Slide 58

Slide 58 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 -BNCEB $MPVE8BUDI -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI-PHTͷઃఆʹΑΓࣗಈੜ੒ ιʔείʔυ΋"84͕ఏڙ

Slide 59

Slide 59 text

ෆਖ਼ΞΫηε෼ੳ • IAMϢʔβʔ΁ͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ࢖༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ௒͑ͨՄࢹԽ

Slide 60

Slide 60 text

Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹ௕ظؒอଘɺ౷߹෼ੳ

Slide 61

Slide 61 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱ΋ʹ ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$

Slide 62

Slide 62 text

CloudWatchϝτϦΫε • 2िؒΛ௒͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ  ΤϦΞνϟʔτͳͲͷՄࢹԽ • ਺஋Ҏ֎ͷѻ͍ • ॊೈͳ৚݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ

Slide 63

Slide 63 text

Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ

Slide 64

Slide 64 text

͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4΁ग़ྗ &-#ͱಉ͡Α͏ʹ࣮૷Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮૷Մ

Slide 65

Slide 65 text

CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδ෼ੳ • Ωϟογϡώοτ෼ੳ • ΞΫηε஍Ҭ෼ੳ • ར༻ऀͷ୺຤/OS෼ੳ • ෆਖ਼ΞΫηε෼ੳ ʢࢀߟURLʣ  http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/

Slide 66

Slide 66 text

·ͱΊ • AWSͷαʔϏεͷଟ͘͸ϩάΛग़ྗ͢Δ • ϩάͷ໨త͸อ؅Ͱ͸ͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ • Amazon Elasticsearch ServiceͰ͋Ε͹ɺ  ໿$20/month͔Β࢝ΊΒΕ·͢