Elasticsearch x AutoScaling

第10回elasticsearch勉強会で使用した資料です。

1. Amazon Web Searvice Ͱ࣮ݱ͢Δ elasticsearchͷେن໛ӡ༻ × גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔɹদా࿨थ
2. ࣗݾ঺հ দా ࿨थ גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔʢࣾһ਺໊̕ʣ ։ൃຊ෦ • ύϒϦοΫDMPʮAudienceSearchʯͷ։ൃ • ओʹΠϯϑϥपΓΛ୲౰
3. ΞδΣϯμ • IMʹ͓͚Δelasticsearchͷ࢖ΘΕํ • elasticsearchͷߏ੒ • ߏ੒ͷܾΊํ • AutoScalingͷϙΠϯτ • ͦͷଞઃఆͷϙΠϯτ
4. elasticsearchͷ࢖ΘΕํ ໿̐ԯͷϢχʔΫIDʹର༷ͯ͠ʑͳଐੑ৘ใΛ෇༩ • Ӿཡཤྺʹجͮ͘Ωʔϫʔυ • ֤ϝσΟΞΑΓఏڙ͞ΕΔηάϝϯτ • User Agent • ΞΫηεݩIPʹجͮ͘ॅॴ΍اۀ৘ใ ೚ҙͷ৚݅ͰIDΛߜΓࠐΉͨΊʹɺɹɹɹɹɹɹɹɹɹɹɹɹΛ ར༻͍ͯ͠·͢ɻ  
   
5. AudienceSearchʹ͍ͭͯ  
   ϝσΟΞདྷ๚ऀ ωοτϦαʔν ޿ࠂ഑৴πʔϧ ˞ɹਤ͸ҰྫͰ͢ ཱྀߦʹڵຯ ೥ऩສ ԣ඿ࢢ໾ॴ ୅ঁੑ
6. શମߏ੒ • શͯAWS؀ڥ • Cluster: 1, Type: 1, Node: 86 • master: 3 (master:true, data: false) • data: 80 (master:false, data:true) • searcher: 2 (master:false, data: false) • indexer: 1 (master:false, data: false)ɹ˞όονॲཧ • version: 1.3.8 → 1.5.2 (5/26Ξοϓσʔτ)
7. શମߏ੒ master data searcher x2 indexer indexerҎ֎͸ɺશ୆AutoScalingͰӡ༻ x3 x80 ELB
8. ߏ੒ܾఆʹ͍ͭͯ
9. Cluster Size • Shard਺ = Node਺ ʹݻఆͯ͠ݕূ • 10 ʙ 200 ͱ୆਺Λมߋ͠ɺԠ౴࣌ؒΛൺֱ • ʙ100୆Ґ·Ͱ͸୆਺ʹԠͯ͡ύϑΥʔϚϯε͕޲্ • 200୆ن໛Ͱ͸AggregationͷύϑΥʔϚϯε͕ྼԽ →ɹूܭ݁ՌͷϚʔδίετͷ૿Ճʁ • σʔλ༰ྔΛՃຯ͠ɺ80୆Ͱελʔτ
10. Shard • Node਺ = 80 ʹݻఆͯ͠ݕূʢc3.largeʣ (1) shard: 40, replica: 1 ʢshard/node = 0.5ʣ (2) shard: 80, reprica: 1 ʢshard/node = 1ʣ (3) shard: 160, replica:1 ʢshard/node = 2ʣ • shard/node ͕̍Λ௒͑Δͱܹ͘͠ύϑΥʔϚϯε͕ྼԽ ※ shard/nodeͰ͸ͳ͘ɺshard/cpuͷํ͕ॏཁ͔΋͠Εͳ͍ɾɾ
11. Instance Size • ΠϯσΫγϯάߴ଎ԽͷͨΊɺϩʔΧϧSSDͷੑೳΛॏࢹ →ɹEphemeral DiskΛ̎ͭੵΊΔc3ϑΝϛϦʔ͔Βબ୒ • σʔλྔ͕ଟ͍৔߹͸xlargeҎ্ͷํ͕CPʹ༏ΕΔ →ɹجຊతʹՁ͕֨ഒʹͳΔͱɺ༰ྔ΋ഒʹͳΔ͕ɺ ɹɹlarge→xlargeͷ࣌ͷΈ30GB→80GB • JVM΁ͷϝϞϦͷׂ౰ྔ͸શମͷ̎ʙׂ̏ʢdoc_valueʣ
12. AutoScallingͷϙΠϯτ
13. Why AutoScaling ? AutoScaling͍ͤͯ͞Δཧ༝ • ̍̌̌୆ن໛ͷαʔόΛ̍୆̍୆؅ཧͯ͠ӡ༻͢Δ͜ͱ͕ɺ ਓ਺తʹݫ͔ͬͨ͠ • terminate͢Ε͹ݩͷ؀ڥ͕࠶ݱՄೳ • εϙοτΠϯελϯε͕ར༻ՄೳʹͳΔ • ύϥϝʔλΛ͍͡Δ͚ͩͰεέʔϧΞ΢τՄೳ
14. recoveryͷνϡʔχϯά • LifeCycleʹΑΓఆظతʹΠϯελϯε͕ೖΕସΘΔ͜ͱ͕͋Δ • ଳҬ΍ಉ࣌ҠಈՄೳͳγϟʔυ਺Λௐ੔͠ɺrecoveryʹ͔͔Δ ࣌ؒΛ୹ॖ͍ͯ͠Δ • ϨϓϦΧ਺͸̍Ͱӡ༻ʢ0Ͱ΍Δͱ࣌ʑσʔλ͕ແ͘ͳΓ·͢ʣ indices: recovery: max_bytes_per_sec: 500mb concurrent_streams: 4
15. recoveryͷνϡʔχϯά • LifeCycleʹΑΓ௥Ճ͞ΕͨΠϯελϯεͷෛՙ • σʔλྔ: 40GB Ͱ10෼ఔ౓ʢc3.xlargeʣ
16. ΫϥελϦϯάͷߴ଎Խ • elasticsearch-cloud-awsϓϥάΠϯΛར༻ • λάࢦఆͰdiscoveryΛߦ͍ɺmaster nodeʹͷΈλάΛઃఆ͢Δ • SGࢦఆͰ͸ର৅͕ଟ͕͔͔͗ͯ࣌ؒ͢Δ discovery: type: ec2 ec2: tag: es_cluster: production
17. ΫϥελϦϯάͷߴ଎Խ • master nodeͷΦʔτεέʔϧͷઃఆ࣌ʹλάΛࢦఆ • ʮaws:autoscaling:groupNameʯλά΋ར༻Մೳʁ
18. ELBʹΑΔ؂ࢹ • ௚઀ݕࡧΛड͚ͳ͍ϊʔυʹؔͯ͠΋ɺࢮ׆؂ࢹ༻ʹELBͱͷ ඥ෇͚Λઃఆʢmaster, dataʣ • ؂ࢹର৅͸9200൪ϙʔτʢ9300ͷํ͕͍͍͔΋ʣ
19. ͦͷଞͷϙΠϯτ
20. ৑௕Խ • Multi-AZߏ੒Ͱߏஙʢ˞3ڌ఺Ҏ্ͷ৔߹͸ෆ໌ʣ • ԼهઃఆͰɺࣗϊʔυͷอ࣋͢ΔshardͷreplicaΛɺҟͳΔAZ ͷϊʔυ্ʹ഑ஔ͢Δ cluster: routing: allocation: awareness: attributes: aws_availability_zone
21. ৑௕Խ • ”aws_availability_zone”ʹࣗϊʔυͷ഑ஔ͞Ε͍ͯΔAZ͕ύϥ ϝʔλͱͯ͠༩͑ΒΕΔʢelasticsearch-cloud-awsʣ • Ұ౓ɺ൒յͨ͠ࡍ΋ແࣄͰͨ͠ { "attributes" : { "aws_availability_zone" : "ap-northeast-1c", "master" : "false" } }
22. ϊʔυ໊Λhostnameʹ͢Δ • ࣗಈͰઃఆ͞ΕΔhostname(ip-10-0-0-1)ʹɺES্ͷϊʔυ໊Λ ߹ΘͤΔ • ๨ΕͯAMIʹݻΊͨΓ͢Δͱɺશϊʔυಉ໊͡લʹɾɾ node: name: ${HOSTNAME} export HOSTNAME=$(hostname -s) /etc/sysconfig/elasticsearch /etc/elasticsearch/elasticsearch.yml
23. ؂ࢹ ؂ࢹͷ̏ຊபʢΦʔτεέʔϧͱͷ૬ੑॏࢹʣ • ELBͷϔϧενΣοΫ • MackerelʢϦιʔεɾϓϩηε؂ࢹʣ • Papertrailʢϩά؂ࢹʣ • ʢىಈ/ఀࢭ࣌ͷSlack΁ͷwebhookʣ
24. ʢࢀߟʣPapertrail master data searcher x2 x3 x80 αʔόʹSSHෆཁͰɺର৅ͷϩάΛtailՄೳ syslog
25. ·ͱΊ • εέʔϧΞ΢τͷ͠қ͞ͰɺAWSͱES͸૬ੑ͕ྑ͍ • ઃఆ࣍ୈͰ͸͋Δ͕ɺAutoScalling͸ຊ൪Ͱͷӡ༻ʹ଱͑͏Δ • Cluster Size͸Ұఆ਺Ҏ্͕ඞཁ • ϊʔυͷೖΕସΘΓΛߟྀ͢Δ͜ͱ͕ඞཁ • 1ͭͷindex͕ԯ݅ϨϕϧͰ͋ͬͯ΋ɺelasticsearch͸࣮༻ʹ଱ ͑ΒΕΔ
26. ΤϯδχΞืू ࠷৽ͷٕज़ʹڵຯͷ͋Δ৽ϝϯόʔΛืूதͰ͢ʂ ࠾༻ٕज़ ։ൃج൫
27. ͋Γ͕ͱ͏͍͟͝·ͨ͠