Elasticsearch on EC2からAmazon Elasticsearch Serviceに移行してだいぶ楽になった話

by ikuwow

Slide 1

Slide 1 text

Elasticsearch on EC2͔Β Amazon Elasticsearch Serviceʹ Ҡߦ͍ͯͩ͠Ϳָʹͳͬͨ࿩ @ikuwow 2016/09/15 ू·ͬtail #6

Slide 2

Slide 2 text

ࣗݾ঺հ • όοΫΤϯυͷPHPͰͷ։ ൃɺAWS΍GCPͷॾʑͷΠ ϯϑϥ·ΘΓத৺ʹ͍ΖΜ ͳ͜ͱΛ΍͍ͬͯ·͢ • SoftwareDesignʹʮΞϓϦ ΤϯδχΞͷͨΊͷΠϯϑ ϥೖ໳ʯͱ͍͏࿈ࡌΛ࢝Ί ·ͨ͠ @ikuwow ग़઒ ز෉

Slide 3

Slide 3 text

teratailͰͷ෼ੳ Elasticsearch + Kibana Redshift + Tableau

Slide 4

Slide 4 text

Elastic search + Kibanaͱ͸ • Elasticsearch: จॻͷશจ ݕࡧ΍ϩάͷ෼ੳʹར༻ ͞ΕΔݕࡧΤϯδϯʢε ΩʔϚϨεʣ • Kibana: Elasticsearchͷϑ ϩϯτΤϯυͱͯ͠௚ײ తͳૢ࡞ͰσʔλΛ؆୯ ʹूܭ΍ՄࢹԽ͕ߦ͑Δ

Slide 5

Slide 5 text

গ͠લͷߏ੒ node.master: false node.data: false node.master: true node.data: true node.master: false node.data: false node.master: true node.data: true Amazon Redshift Amazon S3 teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ KPIΛਂ͘ՄࢹԽ όονॲཧ

Slide 6

Slide 6 text

Կ͕ى͔ͬͨ͜ • ʮ҆ఆՔಇͦ͠͏ͩ͠ɺΠϯελϯεҰݸ૿΍ ͯ͠replica૿΍ͤΔΑ͏ʹͯ͠ΈΑͬͱ̇ʯ

Slide 7

Slide 7 text

Loadingͷ··ฦͬͯ͜ͳ͍ʁʂ

Slide 8

Slide 8 text

Πϯελϯε͕ɾɾɾ • CPU࢖༻཰͕100%ʹ ுΓ෇͍͍ͯΔ

Slide 9

Slide 9 text

cluster_health status: red { "cluster_name" : "elasticsearch", "status" : " red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 1061, "active_shards" : 1061, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 3488, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 23.32380743020444 } ͔͠΋Ϋϥελ͕෼཭ͯ͠Δ

Slide 10

Slide 10 text

ͭΒ͍ • Ϋϥελ؅ཧָͪΜͩͱࢥͬͨΒͦΜͳ͜ͱແ͔ͬͨʂ • ؾָʹϊʔυ૿΍͢ͱམͪΔ • ͍͘ΒεϖοΫ্͛ͯ΋ͨΓͳ͍ • ΘΓͱ୔ࢁͷΠϯελϯεΛ؅ཧ • ΠϯλʔϑΣʔεͱ͔΋ؚΊ • EBSͷ্ʹݐͯͨ΋Μ͔ͩΒॏ͍

Slide 11

Slide 11 text

ͦΜͳ͋Δ೔ɾɾɾ

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

࢖͏͔͠ͳ͍

Slide 14

Slide 14 text

Amazon Elasticsearch Service • AWSͷElasticsearchͷ ϚωʔδυαʔϏε • ؆୯ʹσϓϩΠɺૢ࡞ɺ εέʔϧ

Slide 15

Slide 15 text

ࠓͷߏ੒ teratailͷதͷਓ ४ϦΞϧλΠϜՄࢹԽ Amazon Redshift Amazon S3 KPIΛਂ͘ՄࢹԽ

Slide 16

Slide 16 text

εέʔϧΞ΢τ͕༰қ • ϊʔυ਺ 5 => 6

Slide 17

Slide 17 text

ઃఆ͸؆୯ ϊʔυ਺ܾΊͯ access policy ܾΊΔ͚ͩ

Slide 18

Slide 18 text

ͻͨ͢Βָʂ

Slide 19

Slide 19 text

ͦͷଞ͍͍ͱࢥͬͨͱ͜Ζ • ͻͨ͢Βָ • ΠϯελϯεҰͭҰͭ؅ཧ͠ͳ͍͍ͯ͘ • མͪͳ͍҆৺ײ • ͨ·ʹhung up͢Δ͜ͱ͸͋ΔΈ͍͚ͨͩͲ • Instance Storage΋ར༻Մೳ • AWSΤίγεςϜʹ͋ΔͷͰखؒͳ͠

Slide 20

Slide 20 text

࢖ͬͯΈͯײͨܽ͡఺ • ΞΫηεݩ੍ݶ͕໘౗ • άϩʔόϧͳΤϯυϙΠϯτ͚͕ͩҰ͕ͭ͋ͯΘΕΔ • VPCະରԠͰɺΞΫηεϙϦγʔͰIPΞυϨεɺIAMͰͷ੍ݶ͕ओ • AWS Lambda͔ΒϦΫΤετૹΔͷ͕ඇৗʹ໘౗… • ϓϥάΠϯͷ௥Ճ͕Ͱ͖ͳ͍ • Timelion, Sense, Elasticsearch-head, ElastAlert, … • ࢖͑ͳ͍APIΤϯυϙΠϯτ͕͋Δ • /_reindex, /_reroute etc.

Slide 21

Slide 21 text

·ͱΊ • ୯७ʹ؅ཧର৅΍खָ͕ؒʹͳͬͨʂʂ • ؾܰʹΦʔϓϯͳElasticsearchΛѻ͍͍ͨͳΒ ͍͕ͭ͜Φεεϝ • VPCରԠͱElasticsearch 5.0ରԠ͕଴ͨΕ Δɾɾɾ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

Slide 22

Slide 22 text

@ikuwow