やらかした話 - ログ集約編

AWS Startup Tech Meetup Online#1

Transcript

1. ΍Β͔ͨ͠࿩ - ϩάू໿ฤ Tomohiro Urakawa @ AWS Startup Tech Meetup Online#1
2. @corn3rl0ve • ΫϥελʔגࣜձࣾͰαʔόʔαΠυΤϯδχΞΛ΍͍ͬͯ·͢ • golangͰΞϓϦΛॻ͍ͨΓɼTerraformͰAWS౳ʑͷϦιʔε؅ཧͯͨ͠Γɼ 
 ࠷ۙ͸ϞχλϦϯά·ΘΓʹڵຯ͕͋Γ·͢
3. cluster ͱ͸ • PC/VR/εϚϗ͔ΒࢀՃͰ͖Δόʔνϟϧ SNS • ษڧձ΍ϥΠϒͳͲͷΠϕϯτ͕Ͱ͖ͨ Γɼࣗ࡞ήʔϜΛ࡞ͬͯެ։Ͱ͖ͨΓ • https://resources.awscloud.com/aws- summit-online-japan-2020-on-demand- industry-4-10523/cus-64-aws-summit- online-2020-cluster-inc ͰͥΜͿ͠Ό΂Γ· ͨ͠
4. clusterͷαʔόʔαΠυ֓ཁ ग़ల: https://resources.awscloud.com/aws-summit-online-japan-2020-on-demand-industry-3-12034/cus-64-aws-summit-online-2020-cluster-inc
5. clusterͷαʔόʔαΠυ֓ཁ • ECS • HTTP API Server • ઀ଓதͷϢʔβʔΛ؅ཧ͢ΔͨΊͷdaemonతͳαʔϏε • ఆ࣌ͷόονδϣϒ (scheduled task) • EC2 • ۭؒ಺ͷମݧΛఏڙ͢ΔͨΊͷMQTTαʔόʔ • ۭؒ಺Ͱඞཁͳ৘ใΛੜ੒ʗૹड৴͢ΔͨΊͷಠࣗαʔϏε • Lambda
6. clusterͷαʔόʔαΠυ֓ཁ • ECS • HTTP API Server • ઀ଓதͷϢʔβʔΛ؅ཧ͢ΔͨΊͷdaemonతͳαʔϏε • ఆ࣌ͷόονδϣϒ (scheduled task) • EC2 • room಺ͷମݧΛఏڙ͢ΔͨΊͷMQTTαʔόʔ • room಺Ͱඞཁͳ৘ใΛੜ੒ʗૹड৴͢ΔͨΊͷಠࣗαʔϏε • Lambda CloudWatch Logs
7. clusterͷαʔόʔαΠυ֓ཁ • ECS • HTTP API Server • ઀ଓதͷϢʔβʔΛ؅ཧ͢ΔͨΊͷdaemonతͳαʔϏε • ఆ࣌ͷόονδϣϒ (scheduled task) • EC2 • room಺ͷମݧΛఏڙ͢ΔͨΊͷMQTTαʔόʔ • room಺Ͱඞཁͳ৘ใΛੜ੒ʗૹड৴͢ΔͨΊͷಠࣗαʔϏε • Lambda CloudWatch Logs ΄΅ಉ͡ߏ੒ͰEFWTUHQSE
   ؀ڥ "84ΞΧ΢ϯτ ଘࡏ
8. αʔόʔͷϩάΛؾܰʹݟΕΔͱௐࠪ ͕ḿͬͯศར͚ͩͲɽɽɽ
9. CloudWatch Logs೉͍͠໰୊ • ·ͣAWSͷϢʔβʔΞΧ΢ϯτ͕ඞཁ • ؀ڥ͝ͱʹΞΧ΢ϯτผͳͷͰswitch-role͕ඞཁ • log-group, log-stream ͱ͸ɽɽɽ? • f i lter͕೉͍͠ (׳Ε͕ඞཁ • Insights͕೉͍͠ (׳Ε͕ඞཁ
10. ϩάΛݟ͍͚ͨͩͳͷʹ ߟ͑Δ͜ͱ͕ଟ͗͢Δ
11. Elasticsearch ServiceʹೖΕͨ • Kibana͕ศར • dev͔Β͸͡Ίͯstg/prd΁ల։ CloudWatch Logs Elasticsearch Service Subscription f i lter
12. ҆௚͗ͨ͢ • ར༻͢Δͷ͸99%prd༻ɼ໰୊͕ى͜Δͷ͸99%dev༻ • prd༻Ҏ֎͸ීஈ͋·Γར༻͠ͳ͍͕ɼ໰୊͕ى͖ͨΒରԠ͕ඞཁ • ઃఆมߋ͕͋Ε͹શ෦ʹద༻͠ͳ͍ͱ͍͚ͳ͍ • ܽଛͨ͠ͱ͖ͷิ׬͕೉͍͠ • ϩά͕ݟ͔ͭΒͳ͍? ͱࢥͬͨΒผ؀ڥͷKibanaΛݟ͍ͯͨ
13. Elasticsearch ServiceΛ1ͭʹ͠Α͏
14. 1ͭͷElasticsearch Serviceʹू໿ͨ͠ • ్தʹS3ΛڬΉ͜ͱͰηʔϒϙΠϯτ͕Ͱ͖ͨ • Lambda͸ࣦഊ࣌ʹϦτϥΠͯ͘͠ΕΔͷͰ΄΅ख์͠ʹͳͬͨ dev Elasticsearch Service Kinesis Streams Kinesis Firehose S3 Output Lambda Event Noti f i cation Bulk API stg prd Subscription f i lter
15. ΊͰͨ͠
16. ͱ͸͍͔ͳ͔ͬͨ
17. ͠͹Βͨͬͨ͋͘Δ೔ɽɽɽ • Domain status͕ Upgrade processing ͷ··Կ΋Ͱ͖ͳ͍ঢ়ଶʹ • • clusterͷstatus͸ਅͬ੺ (※ը૾ࡱΓ๨Ε) • Lambda͸શ෦ࣦഊͯ͠ϦτϥΠ͍ͯͨ͠ͷͰɼݺग़ճ਺ʗॲཧ࣌ؒʗ ϩάग़ྗྔ͕͍͢͜͝ͱʹ • →݁ہ͜ͷDomain͸࡟আ͢Δ͜ͱʹͳͬͯ͠·͍·ͨ͠
18. ഊҼ • Elasticsearch ServiceͷෛՙΛແବʹ૿େ͍ͤͯͨ͞ • ର৅ͷlog-groupʹsubscription f i lterΛઃఆ͢Δ͚ͩɼͱ͍͏ঢ়ଶͩͬͨͷ Ͱɼ͋ΒΏΔlog-groupʹઃఆΛ͍Εͯ͠·ͬͨ • ͋·ΓॏཁͰ͸ͳ͍ (ௐࠪʹ΋ར༻͠ʹ͍͘) ϩά΋େྔʹElasticsearch Serviceʹอଘ͞ΕΔ͜ͱʹͳͬͯ͠·ͬͨ • ϦτϥΠͷཞ༻ • ੒ޭ͢ΔݟࠐΈ͕ͳ͍ͷʹϦτϥΠ͠ଓ͚Δঢ়ଶͩͬͨ
19. ·ͱΊ • αʔϏεͷಛੑʗѻ͏σʔλʹ͍͋ͬͯΔ͔ • CloudWatch LogsΑΓElasticsearch Service͕ྑ͍ɼͱ͍͏࿩Ͱ͸ͳ ͍ • উखʹϦτϥΠͯ͘͠ΕͯศརʙɼͰ͸͍͚ͳ͍ • ໨త͕ζϨͯ͠·ͬͯͳ͍͔ • ʮௐࠪΛ΍Γ΍͍ͨ͘͢͠ʯˠʮू໿͍ͨ͠ʯʹมΘͬͯ͠·ͬͯ ͍ͨ
20. ͓ΘΓ
21. https://www.wantedly.com/projects/562252