AWS上に構築する メンテ容易なElasticsearch System / Maintainable Elasticsearch system on AWS

672ee2598bea695496f11022a9a9c4c8?s=47 moznion
August 24, 2019

AWS上に構築する メンテ容易なElasticsearch System / Maintainable Elasticsearch system on AWS

Kyoto.なんか #5 (https://kyoto-nanka.connpass.com/event/141982/) の資料です

672ee2598bea695496f11022a9a9c4c8?s=128

moznion

August 24, 2019
Tweet

Transcript

  1. AWS্ʹߏங͢Δ ϝϯς༰қͳElasticsearch System Kyoto.ͳΜ͔ #5 @moznion

  2. @moznion Software engineer

  3. એ఻

  4. None
  5. Ṗࣗ࡞ΨδΣοτలࣔ͠·͢

  6. None
  7. ESӡ༻ࡶײ - αʔόͷ໘౗Λݟͨ͘ͳ͍……

  8. ESӡ༻ࡶײ - ໘౗ʂʂʂʂʂʂʂʂʂ - όʔδϣϯΞοϓ (͍ͩͿྑ͘ͳΓ·͕ͨ͠……) - ҆શͳϊʔυ਺ͷ૿ݮ - ҆શͳεέʔϧΞοϓ

    - ࣗಈతͳόοΫΞοϓͷߏங - ͳͲͳͲ……
  9. Full Managed Elasticsearch

  10. Full Managed ES - Elastic Cloud (ΊͬͪΌ͍͍) - Amazon Elasticsearch

    Service - Alibaba Cloud Elasticsearch
  11. Full Managed ES - Elastic Cloud (ΊͬͪΌ͍͍) - Amazon Elasticsearch

    Service - Alibaba Cloud Elasticsearch
  12. Amazon Elasticsearch Service

  13. Amazon ES service - ߏங؆୯ - εέʔϧΞοϓɾεέʔϧΞ΢τ͠์୊ - ఆظతͳindexόοΫΞοϓ΋औͬͯ͘ΕΔ -

    όʔδϣϯΞοϓ͕νϣʔ؆୯ʂʂ - ESࣗମͷϝτϦΫεʹԠͯ͡AlartΛ্͛Δͱ͔΋Ͱ͖Δ - Error Log΍Slow Query Log·Ͱ͋Δ (͜Ε͸CloudWatch Logsʹग़Δ)
  14. Amazon ES service - Kibana΋࢖͑Δ - CognitoͱͷΞΧ΢ϯτ࿈ܞ - ࠷ۙElasticsearchΛ࢖ͬͨAlart͕αϙʔτ͞Εͨ

  15. Amazon ES service - AWSαʔϏεͱͷγφδʔ - CloudWatch Logs - Kinesis

    - SQS - Lambda - etc...
  16. ΞʔΩςΫνϟྫ

  17. Kinesis FirehoseΛར༻ Client Firehose Elasticsearch S3 index backup + failed

    record
  18. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ΫϥΠΞϯτ͸Firehoseʹ౤͛ࠐΉ͚ͩ - ༷ʑͳΫϥΠΞϯτιϑτ΢ΣΞ͕͋Δ - kinesis-agent - td-agent

    + plugin - ֤ݴޠͷϥΠϒϥϦ
  19. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ESʹindexͭͭ͠S3ʹΦϦδφϧΛtee͢Δ͜ͱ͕Մೳ - S3ʹ͸߃ٱతʹ࢒͢ - ES͔Β͸ݹ͍σʔλΛফ͢ - Έ͍ͨͳӡ༻͕Ͱ͖Δ

    - ES΁ͷindex͕ࣦഊͨ͠ΤϯτϦ΋S3ʹόοΫΞοϓͯ͠
 ͘ΕΔ
  20. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ࣌ؒ୯Ґ (e.g. σΠϦʔ) ͰͷESͷΠϯσΫε ෼ׂΛࣗಈͰ΍ͬͯ͘ΕΔʂʂ - S3΋AthenaͰΫΤϦՄೳͳܗࣜͰ


    σΟϨΫτϦ (?) Λ෼ׂͯ͘͠ΕΔ - ໌Β͔ʹϩά޲͖
  21. Kinesis Firehoseͱͷ૊Έ߹Θͤ - Kinesis FirehoseΛσϑΥϧτͷঢ়ଶͰ࢖͏ͱྲྀྔ੍ݶ͕͋Δ - > For EU (Paris),

    Asia Pacific (Mumbai), US East (Ohio), EU (Frankfurt), South America (São Paulo), Asia Pacific (Seoul), EU (London), Asia Pacific (Tokyo), US West (N. California), Asia Pacific (Singapore), Asia Pacific (Sydney), and Canada (Central): 1,000 records/second, 1,000 transactions/second, and 1 MiB/second. - Ҿ͔͔ͬͬͨ৔߹͸Kinesis StreamΛผ్༻ҙ͠·͠ΐ͏
  22. Kinesis Firehoseͱͷ૊Έ߹Θͤ - ͦΕ͸ͦ͏ͱ - S3ͷόοΫΞοϓͷܗ͕ࣜҧ͏໰୊ - Τϥʔܥ (վߦ۠੾Γෳ਺JSON (Θ͔Δ))

    - ਖ਼ৗܥ (۠੾Γจࣈແ͠ෳ਺JSON (Θ͔ΒΜ))
  23. Τϥʔܥ (վߦ۠੾Γෳ਺JSON)

  24. ਖ਼ৗܥ (۠੾Γจࣈແ͠ෳ਺JSON) ʂʁ

  25. DynamoDB StreamsΛར༻ Client DynamoDB Elasticsearch Lambda S3 DynamoDB Streams backup

    re-index index
  26. DynamoDB Streamsͱͷ૊Έ߹Θͤ - ϓϥΠϚϦετϨʔδ (DynamoDB) ͱElasticsearchͷ
 ಉظΛͱΔͷ͕νϣʔָ - DynamoDBͷόοΫΞοϓ͔ΒͷESͷ෮ݩ΋νϣʔָ -

    ͳΜΒ͔೚ҙͷσʔλݕࡧΛఏڙ͢Δ༻్޲͖ - ͜͜Ͱ࿩ͨ͠
 https://www.slideshare.net/SORACOM/soracom-ug-tokyo-9-2018iot-lt- 87355107
  27. ӡ༻ͷͨΊͷLambda

  28. Index template updater Elasticsearch Lambda S3 put index template json

    forward apply
  29. Index template updater - Χλ͘ESΛ࢖͓͏ͱ͢Δͱindex template͕΄΅ඞਢ - αʔϏε͝ͱͷindex templateͷద༻Λ͍͍ͪͪ୭͔ʹ
 ґཔ͍ͯͯ͠͸໘౗

    - ʮ͓ئ͍͠·͢೴ͷڪා™ʯ - S3ʹtemplate jsonஔ͚͹ࣗಈͰద༻͞ΕΔͱศར͡ΌΜʁ - ࣮ࡍศར
  30. ͦͷଞͷӡ༻ͷͨΊͷLambda - Re-index from S3 - Outdated index sweeper -

    ES stats reporter - ͳͲͳͲ - ੜ׆͕๛͔ʹ
  31. ࠷ߴʂɹศརʂʂ

  32. ࠔΓ͝ͱ

  33. ࠔΓ͝ͱ - ࣗಈͰϦγϟʔσΟϯάͯ͘͠Εͳ͍ - 1 nodeͰ΋Disk FullʹͳΔͱॻ͖ࠐΈ͕ϒϩοΫ͢Δ - ʮଞͷnode͸disk͕५୔ʹ͋Δͷʹ……ʯ -

    ϦγϟʔσΟϯά͸ҎԼʹΑͬͯىͤ͜Δ - εέʔϧΞοϓɾεέʔϧΞ΢τ - ESͷόʔδϣϯΞοϓ - Diskෆ଍ݕग़࣍ୈࣗಈͰεέʔϧΞ΢τͯ͠εέʔϧΠϯ͢Δ
 Lambdaͳ͍͠͸Step FunctionsͰղܾ (๫ྗ)
  34. ࠔΓ͝ͱ - ϨϓϦέʔγϣϯͷ࢓૊Έ͕ͳ͍ - ͜Ε͸ͦ΋ͦ΋ES͕ͦ͏ - ͔͠͠internal༻్ͷࢀর͚ͩΛ޲͚ΔESΫϥελ͕
 ཉ͍͠Έ͍ͨͳͷ͸͋Γ͕ͪ……

  35. ࠔΓ͝ͱ - ϨϓϦέʔγϣϯͷ࢓૊Έ͕ͳ͍ - ϦΞϧλΠϜੑ͕ඞཁͳ͚Ε͹ - S3ʹεφοϓγϣοτΛஔ͍ͯఆظతʹsync͢Δʁ - ϦΞϧλΠϜੑ͕ඞཁͰ͋Ε͹ -

    μϒϧϥΠτ - nginxͷshadow proxyΛ࢖͏ͱ͍͏ͷ͸Ͳ͏͔ʁʢຊؾ͔ʣ
  36. ඇ·ͱΊ

  37. ඇ·ͱΊ - ࠓ·Ͱ͸ESͷΦϖϨʔγϣϯʹ͸ઐ໳తͳ஌͕ࣝཁٻ͞Ε͍ͯͨ - => ؆୯ʹESͷߏஙɾӡ༻͕Ͱ͖ΔΑ͏ʹͳͬͨ
 ɹϑϧϚωδϝϯτ࠷ߴʂʂʂ - AWSͷΤίγεςϜΛ࢖ͬͯӡ༻ίετΛԼ͛Δ -

    ఆظతʹ΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸΋ͪΖΜग़ͯ͘ΔͷͰ: - ୭Ͱ΋Ͱ͖ΔΑ͏ʹ͢Δ - ؆୯ʹͰ͖ΔΑ͏ʹ͢Δ - (ࠓճ͸ओʹLambdaͰ)
  38. ඇ·ͱΊ - ྑ͔ͬͨͰ͢Ͷ - ͔͠͠ESͷઐ໳తͳ஌ؙ͕ࣝͬͱෆཁʹͳͬͨΘ͚
 Ͱ͸ͳ͍ͷͰҾ͖ଓ͖ؤுΓ·͠ΐ͏

  39. Q?