Elasticsearch x AutoScaling

Slide 1

Slide 1 text

Amazon Web Searvice Ͱ࣮ݱ͢Δ elasticsearchͷେن໛ӡ༻ × גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔɹদా࿨थ

Slide 2

Slide 2 text

ࣗݾ঺հ দా ࿨थ גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔʢࣾһ਺໊̕ʣ ։ൃຊ෦ • ύϒϦοΫDMPʮAudienceSearchʯͷ։ൃ • ओʹΠϯϑϥपΓΛ୲౰

Slide 3

Slide 3 text

ΞδΣϯμ • IMʹ͓͚Δelasticsearchͷ࢖ΘΕํ • elasticsearchͷߏ੒ • ߏ੒ͷܾΊํ • AutoScalingͷϙΠϯτ • ͦͷଞઃఆͷϙΠϯτ

Slide 4

Slide 4 text

elasticsearchͷ࢖ΘΕํ ໿̐ԯͷϢχʔΫIDʹର༷ͯ͠ʑͳଐੑ৘ใΛ෇༩ • Ӿཡཤྺʹجͮ͘Ωʔϫʔυ • ֤ϝσΟΞΑΓఏڙ͞ΕΔηάϝϯτ • User Agent • ΞΫηεݩIPʹجͮ͘ॅॴ΍اۀ৘ใ ೚ҙͷ৚݅ͰIDΛߜΓࠐΉͨΊʹɺɹɹɹɹɹɹɹɹɹɹɹɹΛ ར༻͍ͯ͠·͢ɻ

Slide 5

Slide 5 text

AudienceSearchʹ͍ͭͯ ϝσΟΞདྷ๚ऀ ωοτϦαʔν ޿ࠂ഑৴πʔϧ ˞ɹਤ͸ҰྫͰ͢ ཱྀߦʹڵຯ ೥ऩສ ԣ඿ࢢ໾ॴ ୅ঁੑ

Slide 6

Slide 6 text

શମߏ੒ • શͯAWS؀ڥ • Cluster: 1, Type: 1, Node: 86 • master: 3 (master:true, data: false) • data: 80 (master:false, data:true) • searcher: 2 (master:false, data: false) • indexer: 1 (master:false, data: false)ɹ˞όονॲཧ • version: 1.3.8 → 1.5.2 (5/26Ξοϓσʔτ)

Slide 7

Slide 7 text

શମߏ੒ master data searcher x2 indexer indexerҎ֎͸ɺશ୆AutoScalingͰӡ༻ x3 x80 ELB

Slide 8

Slide 8 text

ߏ੒ܾఆʹ͍ͭͯ

Slide 9

Slide 9 text

Cluster Size • Shard਺ = Node਺ ʹݻఆͯ͠ݕূ • 10 ʙ 200 ͱ୆਺Λมߋ͠ɺԠ౴࣌ؒΛൺֱ • ʙ100୆Ґ·Ͱ͸୆਺ʹԠͯ͡ύϑΥʔϚϯε͕޲্ • 200୆ن໛Ͱ͸AggregationͷύϑΥʔϚϯε͕ྼԽ →ɹूܭ݁ՌͷϚʔδίετͷ૿Ճʁ • σʔλ༰ྔΛՃຯ͠ɺ80୆Ͱελʔτ

Slide 10

Slide 10 text

Shard • Node਺ = 80 ʹݻఆͯ͠ݕূʢc3.largeʣ (1) shard: 40, replica: 1 ʢshard/node = 0.5ʣ (2) shard: 80, reprica: 1 ʢshard/node = 1ʣ (3) shard: 160, replica:1 ʢshard/node = 2ʣ • shard/node ͕̍Λ௒͑Δͱܹ͘͠ύϑΥʔϚϯε͕ྼԽ ※ shard/nodeͰ͸ͳ͘ɺshard/cpuͷํ͕ॏཁ͔΋͠Εͳ͍ɾɾ

Slide 11

Slide 11 text

Instance Size • ΠϯσΫγϯάߴ଎ԽͷͨΊɺϩʔΧϧSSDͷੑೳΛॏࢹ →ɹEphemeral DiskΛ̎ͭੵΊΔc3ϑΝϛϦʔ͔Βબ୒ • σʔλྔ͕ଟ͍৔߹͸xlargeҎ্ͷํ͕CPʹ༏ΕΔ →ɹجຊతʹՁ͕֨ഒʹͳΔͱɺ༰ྔ΋ഒʹͳΔ͕ɺ ɹɹlarge→xlargeͷ࣌ͷΈ30GB→80GB • JVM΁ͷϝϞϦͷׂ౰ྔ͸શମͷ̎ʙׂ̏ʢdoc_valueʣ

Slide 12

Slide 12 text

AutoScallingͷϙΠϯτ

Slide 13

Slide 13 text

Why AutoScaling ? AutoScaling͍ͤͯ͞Δཧ༝ • ̍̌̌୆ن໛ͷαʔόΛ̍୆̍୆؅ཧͯ͠ӡ༻͢Δ͜ͱ͕ɺ ਓ਺తʹݫ͔ͬͨ͠ • terminate͢Ε͹ݩͷ؀ڥ͕࠶ݱՄೳ • εϙοτΠϯελϯε͕ར༻ՄೳʹͳΔ • ύϥϝʔλΛ͍͡Δ͚ͩͰεέʔϧΞ΢τՄೳ

Slide 14

Slide 14 text

recoveryͷνϡʔχϯά • LifeCycleʹΑΓఆظతʹΠϯελϯε͕ೖΕସΘΔ͜ͱ͕͋Δ • ଳҬ΍ಉ࣌ҠಈՄೳͳγϟʔυ਺Λௐ੔͠ɺrecoveryʹ͔͔Δ ࣌ؒΛ୹ॖ͍ͯ͠Δ • ϨϓϦΧ਺͸̍Ͱӡ༻ʢ0Ͱ΍Δͱ࣌ʑσʔλ͕ແ͘ͳΓ·͢ʣ indices: recovery: max_bytes_per_sec: 500mb concurrent_streams: 4

Slide 15

Slide 15 text

recoveryͷνϡʔχϯά • LifeCycleʹΑΓ௥Ճ͞ΕͨΠϯελϯεͷෛՙ • σʔλྔ: 40GB Ͱ10෼ఔ౓ʢc3.xlargeʣ

Slide 16

Slide 16 text

ΫϥελϦϯάͷߴ଎Խ • elasticsearch-cloud-awsϓϥάΠϯΛར༻ • λάࢦఆͰdiscoveryΛߦ͍ɺmaster nodeʹͷΈλάΛઃఆ͢Δ • SGࢦఆͰ͸ର৅͕ଟ͕͔͔͗ͯ࣌ؒ͢Δ discovery: type: ec2 ec2: tag: es_cluster: production

Slide 17

Slide 17 text

ΫϥελϦϯάͷߴ଎Խ • master nodeͷΦʔτεέʔϧͷઃఆ࣌ʹλάΛࢦఆ • ʮaws:autoscaling:groupNameʯλά΋ར༻Մೳʁ

Slide 18

Slide 18 text

ELBʹΑΔ؂ࢹ • ௚઀ݕࡧΛड͚ͳ͍ϊʔυʹؔͯ͠΋ɺࢮ׆؂ࢹ༻ʹELBͱͷ ඥ෇͚Λઃఆʢmaster, dataʣ • ؂ࢹର৅͸9200൪ϙʔτʢ9300ͷํ͕͍͍͔΋ʣ

Slide 19

Slide 19 text

ͦͷଞͷϙΠϯτ

Slide 20

Slide 20 text

৑௕Խ • Multi-AZߏ੒Ͱߏஙʢ˞3ڌ఺Ҏ্ͷ৔߹͸ෆ໌ʣ • ԼهઃఆͰɺࣗϊʔυͷอ࣋͢ΔshardͷreplicaΛɺҟͳΔAZ ͷϊʔυ্ʹ഑ஔ͢Δ cluster: routing: allocation: awareness: attributes: aws_availability_zone

Slide 21

Slide 21 text

৑௕Խ • ”aws_availability_zone”ʹࣗϊʔυͷ഑ஔ͞Ε͍ͯΔAZ͕ύϥ ϝʔλͱͯ͠༩͑ΒΕΔʢelasticsearch-cloud-awsʣ • Ұ౓ɺ൒յͨ͠ࡍ΋ແࣄͰͨ͠ { "attributes" : { "aws_availability_zone" : "ap-northeast-1c", "master" : "false" } }

Slide 22

Slide 22 text

ϊʔυ໊Λhostnameʹ͢Δ • ࣗಈͰઃఆ͞ΕΔhostname(ip-10-0-0-1)ʹɺES্ͷϊʔυ໊Λ ߹ΘͤΔ • ๨ΕͯAMIʹݻΊͨΓ͢Δͱɺશϊʔυಉ໊͡લʹɾɾ node: name: ${HOSTNAME} export HOSTNAME=$(hostname -s) /etc/sysconﬁg/elasticsearch /etc/elasticsearch/elasticsearch.yml

Slide 23

Slide 23 text

؂ࢹ ؂ࢹͷ̏ຊபʢΦʔτεέʔϧͱͷ૬ੑॏࢹʣ • ELBͷϔϧενΣοΫ • MackerelʢϦιʔεɾϓϩηε؂ࢹʣ • Papertrailʢϩά؂ࢹʣ • ʢىಈ/ఀࢭ࣌ͷSlack΁ͷwebhookʣ

Slide 24

Slide 24 text

ʢࢀߟʣPapertrail master data searcher x2 x3 x80 αʔόʹSSHෆཁͰɺର৅ͷϩάΛtailՄೳ syslog

Slide 25

Slide 25 text

·ͱΊ • εέʔϧΞ΢τͷ͠қ͞ͰɺAWSͱES͸૬ੑ͕ྑ͍ • ઃఆ࣍ୈͰ͸͋Δ͕ɺAutoScalling͸ຊ൪Ͱͷӡ༻ʹ଱͑͏Δ • Cluster Size͸Ұఆ਺Ҏ্͕ඞཁ • ϊʔυͷೖΕସΘΓΛߟྀ͢Δ͜ͱ͕ඞཁ • 1ͭͷindex͕ԯ݅ϨϕϧͰ͋ͬͯ΋ɺelasticsearch͸࣮༻ʹ଱ ͑ΒΕΔ

Slide 26

Slide 26 text

ΤϯδχΞืू ࠷৽ͷٕज़ʹڵຯͷ͋Δ৽ϝϯόʔΛืूதͰ͢ʂ ࠾༻ٕज़ ։ൃج൫

Slide 27

Slide 27 text

͋Γ͕ͱ͏͍͟͝·ͨ͠