Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch 7 クラスタ on GCP 事始め - Containers on GCE と GKE - #elasticsearchjp

Elasticsearch 7 クラスタ on GCP 事始め - Containers on GCE と GKE - #elasticsearchjp

Elasticsearch勉強会の発表資料です。
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/270800745/

8261d04bf57a042c8eab6757c386f7b2?s=128

Tomoko Uchida

June 18, 2020
Tweet

Transcript

  1. Elasticsearch 7 Ϋϥελ on GCP ࣄ࢝Ί - Containers on GCE

    ͱ GKE - ୈ36ճElasticsearchษڧձ (2020/6/18) @moco_beta
  2. ࣗݾ঺հ • ଧాஐࢠ @moco_beta • גࣜձࣾLegalForce R&DηΫγϣϯ • ڵຯ: ݕࡧɼࣗવݴޠॲཧ

    ͳͲ • Elasticsearch ӡ༻ܦݧ 3 ೥ɼGCP ܦݧ ̍ϲ݄ʢֶशதʣ
  3. ձࣾ঺հ: LegalForce • 2017೥૑ۀͷϦʔΨϧςοΫاۀ • اۀϛογϣϯʮશͯͷܖ໿ϦεΫΛ੍ޚՄೳʹ͢Δʯ • ఏڙαʔϏε • LegalForce:

    ܖ໿ॻϨϏϡʔࢧԉαʔϏε • Marshall: [క݁൛] ܖ໿ॻ؅ཧαʔϏε
  4. جװػೳͰ͋ΔAIϨϏϡʔʹՃ͑ɼ اۀ๏຿޲͚φϨοδϚωδϝϯτʹ΋஫ྗ • ܖ໿ॻόʔδϣϯ؅ཧ • ܖ໿ॻͷࠩ෼ൺֱ • ৚จ୯ҐͰͷશจݕࡧ • ྨࣅܖ໿ॻͷਪન

    • ৚จ୯ҐͷΞϥΠϯϝϯτ ྫɿ৚จݕࡧ ੺ࣈͷػೳΛR&DͰݚڀ։ൃ ʢϚΠΫϩαʔϏεͱͯ͠੡඼։ൃ νʔϜʹػೳఏڙʣ
  5. ໨࣍ • GCP Ͱ Elasticsearch ΫϥελΛߏங͢Δ࣌ͷબ୒ࢶ [2020 Ver.] • Containers

    on GCE ্ʹཱͯΔ • GKE (Kubernetes) ্ʹཱͯΔ • ·ͱΊʢॴײʣ
  6. GCP Ͱ Elasticsearch Ϋϥελߏங ԿΛ࢖͏ʁ [2020 Ver.] • Elasticsearch Service

    (managed service) • GKE (Kubernetes) • Containers on GCE • GCE VM ʹ௚઀Πϯετʔϧ
  7. Elasticsearch Service • (ϓϨε) Elasticsearch ServiceɺGoogle Cloud PlatformʢGCPʣ೔ ຊͰαʔϏε։࢝ •

    ϚωʔδυαʔϏε • ࣗલͰΠϯϑϥΛߏஙɾӡ༻͢Δڧ͍ཧ༝͕ͳ͚Ε͹ɼ࠷༗ ྗͷબ୒ࢶʹͳΔͱࢥ͏
  8. GKE (Kubernetes) • Google Kubernetes Engine • Kubernetes ʹ໌Δ͍ΠϯϑϥΤϯδχΞ/SRE ͕νʔϜ಺ʹ͍

    Ε͹ʢ·ͨ͸Ұ͔Βษڧ͢Δे෼ͳ͕࣌ؒ͋Ε͹ʣɼબ୒ࢶ ʹͳΔ • ֶशίετ͸·͊·͊ߴΊͱࢥ͏
  9. Containers on GCE • Compute Engine: VM ͱ MIG ΁ͷίϯςφͷσϓϩΠ

    • k8s ͸஌Βͳ͍͕ίϯςφͷӡ༻ʢAWS ECS ౳ʣʹ͸׳Εͯ ͍Δɼͱ͍͏৔߹͸ͱ͖ͬͭ΍͍͢
  10. GCE VM ʹ௚઀Πϯετʔϧ • ૉ๿ͳ΍Γํ͕ͩɼΦϯϓϨͰӡ༻͍ͯ͠ΔγεςϜ͸Ҡߦ͠΍͢ ͍ • Bitnami ͷ Elasticsearch

    Cluster ςϯϓϨʔτ (࠷৽ 7.7.0 ରԠ) Λద༻ ͯ͠Έͨͱ͜Ζɼ ਺෼Ͱ3୆ߏ੒ͷΫϥελ͕ىಈ • Πϯϑϥʹ͋Ε͜ΕखؒΛ͔͚ͣʹɼͬ͞ͱ্ཱ͍ͪ͛ͨ࣌͸ྑ͞ ͦ͏
  11. Πϯϑϥબఆํ਑ • Elasticsearch Service ͸ɼࣄۀͷσʔλ؅ཧϙϦγʔ্࢖͑ͣ • Ϣʔεέʔεɿ௕ظతʹӡ༻༧ఆͷ৽نαʔϏε • কདྷతͳӡ༻Λߟ͑ΔͱɼίϯςφԽ͓͖͍ͯͨ͠ ➡

    GKE or Containers on GCE
  12. Ͳ͕͍͍ͬͪͷɾɾɾʁ • GKE : ʢϚΠΫϩαʔϏεͰ͸࢖͍ͬͯΔ͕...ʣϛυϧ΢ΣΞӡ༻ ܦݧɼϊ΢ϋ΢͕ࣾ಺ʹͳ͍ • GCE : ίϯςφରԠঢ়گ͕Α͘Θ͔Βͳ͍

    • ྆ํ΍ͬͯΈͨ • SREͷਓʹॿ͚ͯ΋Βͬͯ2ਓͰ෼୲
  13. ݕূͷϙΠϯτ • ΠϯσοΫεσʔλΛอଘ͢ΔετϨʔδͷѻ͍ • Elasticsearch ͷ Important System Configuration ΛຬͨͤΔ͔

    • ϊʔυো֐࣌ͷॖୀӡస΍ࣗಈϦΧόϦ • ӡ༻ੑʢॳظߏங΍εέʔϧΞ΢τɾΠϯ͸༰қ͔ɼίϯςφ ͷઃఆมߋɾΞοϓάϨʔυ͸ॊೈʹͰ͖Δ͔ɼͳͲʣ
  14. Containers on GCE ✖

  15. ݕূ༻ΫϥελͷΞʔΩςΫνϟ • ߏ੒: master x 3, data x 3 •

    1 GCE Πϯελϯε͋ͨ Γ 1 ίϯςφ • ΠϯσοΫεσʔλ͸ boot disk ʹอଘʢGCE Π ϯελϯεͱಉ͡ϥΠϑ αΠΫϧʣ
  16. ߏஙखॱ 1. Docker ΠϝʔδΛϏϧυͯ͠ GCR ʹ Push 2. 1. ͔ΒΠϯελϯεςϯϓϨʔτʢ਽ܗʣΛ࡞੒

    3. 2. ͔ΒΠϯελϯεάϧʔϓΛىಈ 4. 3. ʹϔϧενΣοΫΛొ࿥ʢΠϯελϯεͷࣗಈ෮چʣ
  17. ߏஙखॱʢৄࡉʣ • Լهͷهࣄͷखॱ௨Γʹ࣮ߦ͢Δͱɼݕূ༻Ϋϥελ͕ग़དྷ ্͕Γ·͢ • Elasticsearch 7 ΫϥελΛ Containers on

    GCE Ͱ্ཱͪ͛Δ: ͦͷ2 (Managed Instance Group ฤ)
  18. ݕূͷϙΠϯτ • ΠϯσοΫεσʔλΛอଘ͢ΔετϨʔδͷѻ͍ • Persistent Disk Λ࢖͏৔߹͸ΠϯελϯεςϯϓϨʔτ͝ͱʹҰҙͳ disk name ͕ඞཁ...

    • Elasticsearch ͷ Important System Configuration ΛຬͨͤΔ͔ • ϋϚΓͲ͜Ζ͸͋Δ͕ɼճආͰ͖Δ • ϊʔυো֐࣌ͷॖୀӡస΍ࣗಈϦΧόϦ • ظ଴Ͳ͓Γ • ӡ༻ੑ • AWS ECS ʹൺ΂Δͱ෺଍Γͳ͍ͱ͜Ζ͸͋Δɻ͕ɼಛੑΛ஌Ε͹େৎ෉ɻ
  19. GKE (Kubernetes)

  20. ݕূ༻ΫϥελͷΞʔΩςΫνϟ • ߏ੒: master x 3, data x 3 •

    1 Node ͋ͨΓ 1 ϨϓϦΧ • StatefulSet ͰɼΠϯσοΫ εσʔλ͸ӬଓσΟεΫ ʹอଘʢPod ͱ͸ಠཱ͠ ͨϥΠϑαΠΫϧΛ࣋ ͭʣ
  21. ߏஙखॱ 1. Docker ΠϝʔδΛϏϧυͯ͠ GCR ʹ Push 2. GKE ΫϥελΛ࡞੒

    (Web ίϯιʔϧ or gcloud ίϚϯυ) 3. k8s ͷϚχϑΣετ (YAML) Λ࡞੒ 4. 2. ʹ 3. Λద༻ (kubectl)
  22. k8s ϚχϑΣετʹ͍ͭͯิ଍ • ެࣜͷ Elastic Cloud on Kubernetes (All-in-one yaml)

    ͕͋Δ • Elasticsearch, Kibana ͳͲશ෦ೖΓˍઃఆͣΈύοέʔδ • ʢৄࡉ͸֬ೝதͷͨΊɼʣࠓճͷݕূͰ͸ࣗ෼ͨͪͰॻ͖ى ͨ͜͠ϚχϑΣετΛ࢖༻
  23. ߏஙखॱʢৄࡉʣ • Լهͷهࣄͷखॱ௨Γʹ࣮ߦ͢Δͱɼݕূ༻Ϋϥελ͕ग़དྷ ্͕Γ·͢ • Elasticsearch 7 ΫϥελΛ GKE Ͱ্ཱͪ͛Δ

  24. ݕূͷϙΠϯτ • ΠϯσοΫεσʔλΛอଘ͢ΔετϨʔδͷѻ͍ • Persistent Volume (w/ StatefulSet), Ephemeral Volume

    (w/ Deployment) ྆ํ͍͚Δ • Elasticsearch ͷ Important System Configuration ΛຬͨͤΔ͔ • ϋϚΓͲ͜Ζ͸͋Δ͕ɼճආͰ͖Δ • ϊʔυো֐࣌ͷॖୀӡస΍ࣗಈϦΧόϦ • ظ଴Ͳ͓Γ • ӡ༻ੑ • k8s ͷΤίγεςϜʹৄ͘͠ͳΕ͹৭ʑͰ͖Δɾɾɾ͸ͣ
  25. GKE DEMO

  26. GKE DEMO

  27. GKE DEMO

  28. GKE DEMO

  29. ·ͱΊ • Elasticsearch 7 ΫϥελΛ Containers on GCE ͱ GKE

    Ͱߏங͠ɼ࡞ۀ खॱ΍Մ༻ੑͷݕূΛߦͬͨ • ͲͪΒ΋ʢӡ༻खॱΛ੔උͯ͠ɼϊ΢ϋ΢ΛͨΊ͍͚ͯ͹ʣ҆શ ʹϓϩμΫγϣϯӡ༻Ͱ͖Δ͸ͣ • ͲͪΒʹ͢Δ͔͸ɼνʔϜମ੍΍ϝϯόʔͷεΩϧηοτͱ૬ஊ͠ ܾͯΊΔͷ͕ྑ͍
  30. ࠓ೔࿩͞ͳ͔ͬͨ͜ͱ • ϩʔυόϥϯγϯά • NWΞΫηείϯτϩʔϧ • ΠϯσοΫεσʔλͷ Snapshot / Restore

    • ίϯςφͷ Rolling Update • ΫϥελͷεέʔϧΞ΢τɾεέʔϧΠϯ • ϦιʔεɼϩάͷϞχλϦϯά
  31. Thank You (๑•ŷ•)و✧ • LegalForce Ͱ͸ɼAWS / GCP ʹ໌Δ͍ΠϯϑϥΤϯδχΞɼ SRE

    ΛٻΊ͍ͯ·͢ʂڵຯͷ͋Δํ͸͓੠͕͚͍ͩ͘͞ • https://www.wantedly.com/projects/442926 • https://www.wantedly.com/projects/364756