Elasticsearch on EC2からAmazon Elasticsearch Serviceに 移行してだいぶ楽になった話

46d57e2c568a65ade212f56539687bc2?s=47 ikuwow
September 15, 2016

Elasticsearch on EC2からAmazon Elasticsearch Serviceに 移行してだいぶ楽になった話

46d57e2c568a65ade212f56539687bc2?s=128

ikuwow

September 15, 2016
Tweet

Transcript

  1. Elasticsearch on EC2͔Β Amazon Elasticsearch Serviceʹ Ҡߦ͍ͯͩ͠Ϳָʹͳͬͨ࿩ @ikuwow 2016/09/15 ू·ͬtail

    #6
  2. ࣗݾ঺հ • όοΫΤϯυͷPHPͰͷ։ ൃɺAWS΍GCPͷॾʑͷΠ ϯϑϥ·ΘΓத৺ʹ͍ΖΜ ͳ͜ͱΛ΍͍ͬͯ·͢ • SoftwareDesignʹʮΞϓϦ ΤϯδχΞͷͨΊͷΠϯϑ ϥೖ໳ʯͱ͍͏࿈ࡌΛ࢝Ί

    ·ͨ͠ @ikuwow ग़઒ ز෉
  3. teratailͰͷ෼ੳ Elasticsearch + Kibana Redshift + Tableau

  4. Elastic search + Kibanaͱ͸ • Elasticsearch: จॻͷશจ ݕࡧ΍ϩάͷ෼ੳʹར༻ ͞ΕΔݕࡧΤϯδϯʢε ΩʔϚϨεʣ

    • Kibana: Elasticsearchͷϑ ϩϯτΤϯυͱͯ͠௚ײ తͳૢ࡞ͰσʔλΛ؆୯ ʹूܭ΍ՄࢹԽ͕ߦ͑Δ
  5. গ͠લͷߏ੒ node.master: false node.data: false node.master: true node.data: true node.master:

    false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ
  6. Կ͕ى͔ͬͨ͜ • ʮ҆ఆՔಇͦ͠͏ͩ͠ɺΠϯελϯεҰݸ૿΍ ͯ͠replica૿΍ͤΔΑ͏ʹͯ͠ΈΑͬͱ̇ʯ

  7. Loadingͷ··ฦͬͯ͜ͳ͍ʁʂ

  8. Πϯελϯε͕ɾɾɾ • CPU࢖༻཰͕100%ʹ ுΓ෇͍͍ͯΔ

  9. cluster_health status: red { "cluster_name" : "elasticsearch", "status" : "

    red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 1061, "active_shards" : 1061, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 3488, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 23.32380743020444 } ͔͠΋Ϋϥελ͕෼཭ͯ͠Δ
  10. ͭΒ͍ • Ϋϥελ؅ཧָͪΜͩͱࢥͬͨΒͦΜͳ͜ͱແ͔ͬͨʂ • ؾָʹϊʔυ૿΍͢ͱམͪΔ • ͍͘ΒεϖοΫ্͛ͯ΋ͨΓͳ͍ • ΘΓͱ୔ࢁͷΠϯελϯεΛ؅ཧ •

    ΠϯλʔϑΣʔεͱ͔΋ؚΊ • EBSͷ্ʹݐͯͨ΋Μ͔ͩΒॏ͍
  11. ͦΜͳ͋Δ೔ɾɾɾ

  12. None
  13. ࢖͏͔͠ͳ͍

  14. Amazon Elasticsearch Service • AWSͷElasticsearchͷ ϚωʔδυαʔϏε • ؆୯ʹσϓϩΠɺૢ࡞ɺ εέʔϧ

  15. ࠓͷߏ੒ teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ Amazon Redshift Amazon S3 KPIΛਂ͘ՄࢹԽ

  16. εέʔϧΞ΢τ͕༰қ • ϊʔυ਺ 5 => 6

  17. ઃఆ͸؆୯ ϊʔυ਺ܾΊͯ access policy ܾΊΔ͚ͩ

  18. ͻͨ͢Βָʂ

  19. ͦͷଞ͍͍ͱࢥͬͨͱ͜Ζ • ͻͨ͢Βָ • ΠϯελϯεҰͭҰͭ؅ཧ͠ͳ͍͍ͯ͘ • མͪͳ͍҆৺ײ • ͨ·ʹhung up͢Δ͜ͱ͸͋ΔΈ͍͚ͨͩͲ

    • Instance Storage΋ར༻Մೳ • AWSΤίγεςϜʹ͋ΔͷͰखؒͳ͠
  20. ࢖ͬͯΈͯײͨܽ͡఺ • ΞΫηεݩ੍ݶ͕໘౗ • άϩʔόϧͳΤϯυϙΠϯτ͚͕ͩҰ͕ͭ͋ͯΘΕΔ • VPCະରԠͰɺΞΫηεϙϦγʔͰIPΞυϨεɺIAMͰͷ੍ݶ͕ओ • AWS Lambda͔ΒϦΫΤετૹΔͷ͕ඇৗʹ໘౗…

    • ϓϥάΠϯͷ௥Ճ͕Ͱ͖ͳ͍ • Timelion, Sense, Elasticsearch-head, ElastAlert, … • ࢖͑ͳ͍APIΤϯυϙΠϯτ͕͋Δ • /_reindex, /_reroute etc.
  21. ·ͱΊ • ୯७ʹ؅ཧର৅΍खָ͕ؒʹͳͬͨʂʂ • ؾܰʹΦʔϓϯͳElasticsearchΛѻ͍͍ͨͳΒ ͍͕ͭ͜Φεεϝ • VPCରԠͱElasticsearch 5.0ରԠ͕଴ͨΕ Δɾɾɾ

    ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
  22. @ikuwow