Elasticsearch on EC2からAmazon Elasticsearch Serviceに 移行してだいぶ楽になった話

Transcript

1. Elasticsearch on EC2͔Β
   Amazon Elasticsearch Serviceʹ
   Ҡߦ͍ͯͩ͠Ϳָʹͳͬͨ࿩
   @ikuwow
   2016/09/15 ू·ͬtail #6
   

   View Slide

2. ࣗݾ঺հ
   • όοΫΤϯυͷPHPͰͷ։
   ൃɺAWS΍GCPͷॾʑͷΠ
   ϯϑϥ·ΘΓத৺ʹ͍ΖΜ
   ͳ͜ͱΛ΍͍ͬͯ·͢
   • SoftwareDesignʹʮΞϓϦ
   ΤϯδχΞͷͨΊͷΠϯϑ
   ϥೖ໳ʯͱ͍͏࿈ࡌΛ࢝Ί
   ·ͨ͠
   @ikuwow
   ग़઒ ز෉
   

   View Slide

3. teratailͰͷ෼ੳ
   Elasticsearch + Kibana Redshift + Tableau
   

   View Slide

4. Elastic search + Kibanaͱ͸
   • Elasticsearch: จॻͷશจ
   ݕࡧ΍ϩάͷ෼ੳʹར༻
   ͞ΕΔݕࡧΤϯδϯʢε
   ΩʔϚϨεʣ
   • Kibana: Elasticsearchͷϑ
   ϩϯτΤϯυͱͯ͠௚ײ
   తͳૢ࡞ͰσʔλΛ؆୯
   ʹूܭ΍ՄࢹԽ͕ߦ͑Δ
   

   View Slide

5. গ͠લͷߏ੒
   node.master: false
   node.data: false
   node.master: true
   node.data: true
   node.master: false
   node.data: false
   node.master: true
   node.data: true
   Amazon Redshift
   Amazon S3
   teratailͷதͷਓ
   ४ϦΞϧλΠϜՄࢹԽ
   KPIΛਂ͘ՄࢹԽ
   όονॲཧ
   

   View Slide

6. Կ͕ى͔ͬͨ͜
   • ʮ҆ఆՔಇͦ͠͏ͩ͠ɺΠϯελϯεҰݸ૿΍
   ͯ͠replica૿΍ͤΔΑ͏ʹͯ͠ΈΑͬͱ̇ʯ
   

   View Slide

7. Loadingͷ··ฦͬͯ͜ͳ͍ʁʂ
   

   View Slide

8. Πϯελϯε͕ɾɾɾ
   • CPU࢖༻཰͕100%ʹ
   ுΓ෇͍͍ͯΔ
   

   View Slide

9. cluster_health status: red
   {
   "cluster_name" : "elasticsearch",
   "status" : "
   red",
   "timed_out" : false,
   "number_of_nodes" : 1,
   "number_of_data_nodes" : 1,
   "active_primary_shards" : 1061,
   "active_shards" : 1061,
   "relocating_shards" : 0,
   "initializing_shards" : 0,
   "unassigned_shards" : 3488,
   "delayed_unassigned_shards" : 0,
   "number_of_pending_tasks" : 0,
   "number_of_in_flight_fetch" : 0,
   "task_max_waiting_in_queue_millis" : 0,
   "active_shards_percent_as_number" : 23.32380743020444
   }
   ͔͠΋Ϋϥελ͕෼཭ͯ͠Δ
   

   View Slide

10. ͭΒ͍
    • Ϋϥελ؅ཧָͪΜͩͱࢥͬͨΒͦΜͳ͜ͱແ͔ͬͨʂ
    • ؾָʹϊʔυ૿΍͢ͱམͪΔ
    • ͍͘ΒεϖοΫ্͛ͯ΋ͨΓͳ͍
    • ΘΓͱ୔ࢁͷΠϯελϯεΛ؅ཧ
    • ΠϯλʔϑΣʔεͱ͔΋ؚΊ
    • EBSͷ্ʹݐͯͨ΋Μ͔ͩΒॏ͍
    

    View Slide

11. ͦΜͳ͋Δ೔ɾɾɾ
    

    View Slide

12. View Slide

13. ࢖͏͔͠ͳ͍
    

    View Slide

14. Amazon Elasticsearch Service
    • AWSͷElasticsearchͷ
    ϚωʔδυαʔϏε
    • ؆୯ʹσϓϩΠɺૢ࡞ɺ
    εέʔϧ
    

    View Slide

15. ࠓͷߏ੒
    teratailͷதͷਓ
    ४ϦΞϧλΠϜՄࢹԽ
    Amazon Redshift
    Amazon S3
    KPIΛਂ͘ՄࢹԽ
    

    View Slide

16. εέʔϧΞ΢τ͕༰қ
    • ϊʔυ਺ 5 => 6
    

    View Slide

17. ઃఆ͸؆୯
    ϊʔυ਺ܾΊͯ access policy ܾΊΔ͚ͩ
    

    View Slide

18. ͻͨ͢Βָʂ
    

    View Slide

19. ͦͷଞ͍͍ͱࢥͬͨͱ͜Ζ
    • ͻͨ͢Βָ
    • ΠϯελϯεҰͭҰͭ؅ཧ͠ͳ͍͍ͯ͘
    • མͪͳ͍҆৺ײ
    • ͨ·ʹhung up͢Δ͜ͱ͸͋ΔΈ͍͚ͨͩͲ
    • Instance Storage΋ར༻Մೳ
    • AWSΤίγεςϜʹ͋ΔͷͰखؒͳ͠
    

    View Slide

20. ࢖ͬͯΈͯײͨܽ͡఺
    • ΞΫηεݩ੍ݶ͕໘౗
    • άϩʔόϧͳΤϯυϙΠϯτ͚͕ͩҰ͕ͭ͋ͯΘΕΔ
    • VPCະରԠͰɺΞΫηεϙϦγʔͰIPΞυϨεɺIAMͰͷ੍ݶ͕ओ
    • AWS Lambda͔ΒϦΫΤετૹΔͷ͕ඇৗʹ໘౗…
    • ϓϥάΠϯͷ௥Ճ͕Ͱ͖ͳ͍
    • Timelion, Sense, Elasticsearch-head, ElastAlert, …
    • ࢖͑ͳ͍APIΤϯυϙΠϯτ͕͋Δ
    • /_reindex, /_reroute etc.
    

    View Slide

21. ·ͱΊ
    • ୯७ʹ؅ཧର৅΍खָ͕ؒʹͳͬͨʂʂ
    • ؾܰʹΦʔϓϯͳElasticsearchΛѻ͍͍ͨͳΒ
    ͍͕ͭ͜Φεεϝ
    • VPCରԠͱElasticsearch 5.0ରԠ͕଴ͨΕ
    Δɾɾɾ
    ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide

22. @ikuwow
    

    View Slide