AWS上に構築する メンテ容易なElasticsearch System / Maintainable Elasticsearch system on AWS

Kyoto.なんか #5 (https://kyoto-nanka.connpass.com/event/141982/) の資料です

Transcript

1. AWS্ʹߏங͢Δ ϝϯς༰қͳElasticsearch System Kyoto.ͳΜ͔ #5 @moznion
2. @moznion Software engineer
3. એ఻
5. Ṗࣗ࡞ΨδΣοτలࣔ͠·͢
7. ESӡ༻ࡶײ - αʔόͷ໘౗Λݟͨ͘ͳ͍……
8. ESӡ༻ࡶײ - ໘౗ʂʂʂʂʂʂʂʂʂ - όʔδϣϯΞοϓ (͍ͩͿྑ͘ͳΓ·͕ͨ͠……) - ҆શͳϊʔυ਺ͷ૿ݮ - ҆શͳεέʔϧΞοϓ - ࣗಈతͳόοΫΞοϓͷߏங - ͳͲͳͲ……
9. Full Managed Elasticsearch
10. Full Managed ES - Elastic Cloud (ΊͬͪΌ͍͍) - Amazon Elasticsearch Service - Alibaba Cloud Elasticsearch
11. Full Managed ES - Elastic Cloud (ΊͬͪΌ͍͍) - Amazon Elasticsearch Service - Alibaba Cloud Elasticsearch
12. Amazon Elasticsearch Service
13. Amazon ES service - ߏங؆୯ - εέʔϧΞοϓɾεέʔϧΞ΢τ͠์୊ - ఆظతͳindexόοΫΞοϓ΋औͬͯ͘ΕΔ - όʔδϣϯΞοϓ͕νϣʔ؆୯ʂʂ - ESࣗମͷϝτϦΫεʹԠͯ͡AlartΛ্͛Δͱ͔΋Ͱ͖Δ - Error Log΍Slow Query Log·Ͱ͋Δ (͜Ε͸CloudWatch Logsʹग़Δ)
14. Amazon ES service - Kibana΋࢖͑Δ - CognitoͱͷΞΧ΢ϯτ࿈ܞ - ࠷ۙElasticsearchΛ࢖ͬͨAlart͕αϙʔτ͞Εͨ
15. Amazon ES service - AWSαʔϏεͱͷγφδʔ - CloudWatch Logs - Kinesis - SQS - Lambda - etc...
16. ΞʔΩςΫνϟྫ
17. Kinesis FirehoseΛར༻ Client Firehose Elasticsearch S3 index backup + failed record
18. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ΫϥΠΞϯτ͸Firehoseʹ౤͛ࠐΉ͚ͩ - ༷ʑͳΫϥΠΞϯτιϑτ΢ΣΞ͕͋Δ - kinesis-agent - td-agent + plugin - ֤ݴޠͷϥΠϒϥϦ
19. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ESʹindexͭͭ͠S3ʹΦϦδφϧΛtee͢Δ͜ͱ͕Մೳ - S3ʹ͸߃ٱతʹ࢒͢ - ES͔Β͸ݹ͍σʔλΛফ͢ - Έ͍ͨͳӡ༻͕Ͱ͖Δ - ES΁ͷindex͕ࣦഊͨ͠ΤϯτϦ΋S3ʹόοΫΞοϓͯ͠
 ͘ΕΔ
20. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ࣌ؒ୯Ґ (e.g. σΠϦʔ) ͰͷESͷΠϯσΫε ෼ׂΛࣗಈͰ΍ͬͯ͘ΕΔʂʂ - S3΋AthenaͰΫΤϦՄೳͳܗࣜͰ
 σΟϨΫτϦ (?) Λ෼ׂͯ͘͠ΕΔ - ໌Β͔ʹϩά޲͖
21. Kinesis Firehoseͱͷ૊Έ߹Θͤ - Kinesis FirehoseΛσϑΥϧτͷঢ়ଶͰ࢖͏ͱྲྀྔ੍ݶ͕͋Δ - > For EU (Paris), Asia Pacific (Mumbai), US East (Ohio), EU (Frankfurt), South America (São Paulo), Asia Pacific (Seoul), EU (London), Asia Pacific (Tokyo), US West (N. California), Asia Pacific (Singapore), Asia Pacific (Sydney), and Canada (Central): 1,000 records/second, 1,000 transactions/second, and 1 MiB/second. - Ҿ͔͔ͬͬͨ৔߹͸Kinesis StreamΛผ్༻ҙ͠·͠ΐ͏
22. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ͦΕ͸ͦ͏ͱ - S3ͷόοΫΞοϓͷܗ͕ࣜҧ͏໰୊ - Τϥʔܥ (վߦ۠੾Γෳ਺JSON (Θ͔Δ)) - ਖ਼ৗܥ (۠੾Γจࣈແ͠ෳ਺JSON (Θ͔ΒΜ))
23. Τϥʔܥ (վߦ۠੾Γෳ਺JSON)
24. ਖ਼ৗܥ (۠੾Γจࣈແ͠ෳ਺JSON) ʂʁ
25. DynamoDB StreamsΛར༻ Client DynamoDB Elasticsearch Lambda S3 DynamoDB Streams backup re-index index
26. DynamoDB Streamsͱͷ૊Έ߹Θͤ - ϓϥΠϚϦετϨʔδ (DynamoDB) ͱElasticsearchͷ
 ಉظΛͱΔͷ͕νϣʔָ - DynamoDBͷόοΫΞοϓ͔ΒͷESͷ෮ݩ΋νϣʔָ - ͳΜΒ͔೚ҙͷσʔλݕࡧΛఏڙ͢Δ༻్޲͖ - ͜͜Ͱ࿩ͨ͠
 https://www.slideshare.net/SORACOM/soracom-ug-tokyo-9-2018iot-lt- 87355107
27. ӡ༻ͷͨΊͷLambda
28. Index template updater Elasticsearch Lambda S3 put index template json forward apply
29. Index template updater - Χλ͘ESΛ࢖͓͏ͱ͢Δͱindex template͕΄΅ඞਢ - αʔϏε͝ͱͷindex templateͷద༻Λ͍͍ͪͪ୭͔ʹ
 ґཔ͍ͯͯ͠͸໘౗ - ʮ͓ئ͍͠·͢೴ͷڪා™ʯ - S3ʹtemplate jsonஔ͚͹ࣗಈͰద༻͞ΕΔͱศར͡ΌΜʁ - ࣮ࡍศར
30. ͦͷଞͷӡ༻ͷͨΊͷLambda - Re-index from S3 - Outdated index sweeper - ES stats reporter - ͳͲͳͲ - ੜ׆͕๛͔ʹ
31. ࠷ߴʂɹศརʂʂ
32. ࠔΓ͝ͱ
33. ࠔΓ͝ͱ - ࣗಈͰϦγϟʔσΟϯάͯ͘͠Εͳ͍ - 1 nodeͰ΋Disk FullʹͳΔͱॻ͖ࠐΈ͕ϒϩοΫ͢Δ - ʮଞͷnode͸disk͕५୔ʹ͋Δͷʹ……ʯ - ϦγϟʔσΟϯά͸ҎԼʹΑͬͯىͤ͜Δ - εέʔϧΞοϓɾεέʔϧΞ΢τ - ESͷόʔδϣϯΞοϓ - Diskෆ଍ݕग़࣍ୈࣗಈͰεέʔϧΞ΢τͯ͠εέʔϧΠϯ͢Δ
 Lambdaͳ͍͠͸Step FunctionsͰղܾ (๫ྗ)
34. ࠔΓ͝ͱ - ϨϓϦέʔγϣϯͷ࢓૊Έ͕ͳ͍ - ͜Ε͸ͦ΋ͦ΋ES͕ͦ͏ - ͔͠͠internal༻్ͷࢀর͚ͩΛ޲͚ΔESΫϥελ͕
 ཉ͍͠Έ͍ͨͳͷ͸͋Γ͕ͪ……
35. ࠔΓ͝ͱ - ϨϓϦέʔγϣϯͷ࢓૊Έ͕ͳ͍ - ϦΞϧλΠϜੑ͕ඞཁͳ͚Ε͹ - S3ʹεφοϓγϣοτΛஔ͍ͯఆظతʹsync͢Δʁ - ϦΞϧλΠϜੑ͕ඞཁͰ͋Ε͹ - μϒϧϥΠτ - nginxͷshadow proxyΛ࢖͏ͱ͍͏ͷ͸Ͳ͏͔ʁʢຊؾ͔ʣ
36. ඇ·ͱΊ
37. ඇ·ͱΊ - ࠓ·Ͱ͸ESͷΦϖϨʔγϣϯʹ͸ઐ໳తͳ஌͕ࣝཁٻ͞Ε͍ͯͨ - => ؆୯ʹESͷߏஙɾӡ༻͕Ͱ͖ΔΑ͏ʹͳͬͨ
 ɹϑϧϚωδϝϯτ࠷ߴʂʂʂ - AWSͷΤίγεςϜΛ࢖ͬͯӡ༻ίετΛԼ͛Δ - ఆظతʹ΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸΋ͪΖΜग़ͯ͘ΔͷͰ: - ୭Ͱ΋Ͱ͖ΔΑ͏ʹ͢Δ - ؆୯ʹͰ͖ΔΑ͏ʹ͢Δ - (ࠓճ͸ओʹLambdaͰ)
38. ඇ·ͱΊ - ྑ͔ͬͨͰ͢Ͷ - ͔͠͠ESͷઐ໳తͳ஌ؙ͕ࣝͬͱෆཁʹͳͬͨΘ͚
 Ͱ͸ͳ͍ͷͰҾ͖ଓ͖ؤுΓ·͠ΐ͏
39. Q?