Elasticsearch on EC2からAmazon Elasticsearch Serviceに 移行してだいぶ楽になった話

Transcript

1. Elasticsearch on EC2͔Β Amazon Elasticsearch Serviceʹ Ҡߦ͍ͯͩ͠Ϳָʹͳͬͨ࿩ @ikuwow 2016/09/15 ू·ͬtail #6
2. ࣗݾ঺հ • όοΫΤϯυͷPHPͰͷ։ ൃɺAWS΍GCPͷॾʑͷΠ ϯϑϥ·ΘΓத৺ʹ͍ΖΜ ͳ͜ͱΛ΍͍ͬͯ·͢ • SoftwareDesignʹʮΞϓϦ ΤϯδχΞͷͨΊͷΠϯϑ ϥೖ໳ʯͱ͍͏࿈ࡌΛ࢝Ί ·ͨ͠ @ikuwow ग़઒ ز෉
3. teratailͰͷ෼ੳ Elasticsearch + Kibana Redshift + Tableau
4. Elastic search + Kibanaͱ͸ • Elasticsearch: จॻͷશจ ݕࡧ΍ϩάͷ෼ੳʹར༻ ͞ΕΔݕࡧΤϯδϯʢε ΩʔϚϨεʣ • Kibana: Elasticsearchͷϑ ϩϯτΤϯυͱͯ͠௚ײ తͳૢ࡞ͰσʔλΛ؆୯ ʹूܭ΍ՄࢹԽ͕ߦ͑Δ
5. গ͠લͷߏ੒ node.master: false node.data: false node.master: true node.data: true node.master: false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ
6. Կ͕ى͔ͬͨ͜ • ʮ҆ఆՔಇͦ͠͏ͩ͠ɺΠϯελϯεҰݸ૿΍ ͯ͠replica૿΍ͤΔΑ͏ʹͯ͠ΈΑͬͱ̇ʯ
7. Loadingͷ··ฦͬͯ͜ͳ͍ʁʂ
8. Πϯελϯε͕ɾɾɾ • CPU࢖༻཰͕100%ʹ ுΓ෇͍͍ͯΔ
9. cluster_health status: red { "cluster_name" : "elasticsearch", "status" : " red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 1061, "active_shards" : 1061, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 3488, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 23.32380743020444 } ͔͠΋Ϋϥελ͕෼཭ͯ͠Δ
10. ͭΒ͍ • Ϋϥελ؅ཧָͪΜͩͱࢥͬͨΒͦΜͳ͜ͱແ͔ͬͨʂ • ؾָʹϊʔυ૿΍͢ͱམͪΔ • ͍͘ΒεϖοΫ্͛ͯ΋ͨΓͳ͍ • ΘΓͱ୔ࢁͷΠϯελϯεΛ؅ཧ • ΠϯλʔϑΣʔεͱ͔΋ؚΊ • EBSͷ্ʹݐͯͨ΋Μ͔ͩΒॏ͍
11. ͦΜͳ͋Δ೔ɾɾɾ
13. ࢖͏͔͠ͳ͍
14. Amazon Elasticsearch Service • AWSͷElasticsearchͷ ϚωʔδυαʔϏε • ؆୯ʹσϓϩΠɺૢ࡞ɺ εέʔϧ
15. ࠓͷߏ੒ teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ Amazon Redshift Amazon S3 KPIΛਂ͘ՄࢹԽ
16. εέʔϧΞ΢τ͕༰қ • ϊʔυ਺ 5 => 6
17. ઃఆ͸؆୯ ϊʔυ਺ܾΊͯ access policy ܾΊΔ͚ͩ
18. ͻͨ͢Βָʂ
19. ͦͷଞ͍͍ͱࢥͬͨͱ͜Ζ • ͻͨ͢Βָ • ΠϯελϯεҰͭҰͭ؅ཧ͠ͳ͍͍ͯ͘ • མͪͳ͍҆৺ײ • ͨ·ʹhung up͢Δ͜ͱ͸͋ΔΈ͍͚ͨͩͲ • Instance Storage΋ར༻Մೳ • AWSΤίγεςϜʹ͋ΔͷͰखؒͳ͠
20. ࢖ͬͯΈͯײͨܽ͡఺ • ΞΫηεݩ੍ݶ͕໘౗ • άϩʔόϧͳΤϯυϙΠϯτ͚͕ͩҰ͕ͭ͋ͯΘΕΔ • VPCະରԠͰɺΞΫηεϙϦγʔͰIPΞυϨεɺIAMͰͷ੍ݶ͕ओ • AWS Lambda͔ΒϦΫΤετૹΔͷ͕ඇৗʹ໘౗… • ϓϥάΠϯͷ௥Ճ͕Ͱ͖ͳ͍ • Timelion, Sense, Elasticsearch-head, ElastAlert, … • ࢖͑ͳ͍APIΤϯυϙΠϯτ͕͋Δ • /_reindex, /_reroute etc.
21. ·ͱΊ • ୯७ʹ؅ཧର৅΍खָ͕ؒʹͳͬͨʂʂ • ؾܰʹΦʔϓϯͳElasticsearchΛѻ͍͍ͨͳΒ ͍͕ͭ͜Φεεϝ • VPCରԠͱElasticsearch 5.0ରԠ͕଴ͨΕ Δɾɾɾ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
22. @ikuwow