Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearch x AutoScaling
Search
mats
June 02, 2015
Technology
0
52
Elasticsearch x AutoScaling
第10回elasticsearch勉強会で使用した資料です。
mats
June 02, 2015
Tweet
Share
More Decks by mats
See All by mats
「生成系AI」と「ソフトウェアライセンス」の今 / Generative AI and OSS License
mats16
3
1.1k
🦜️🔗LangChain入門 / LangChain 101
mats16
1
480
Supabase - AWS DevDay 2022
mats16
2
1.4k
AWS スタートアップ支援プログラム / AWS Activate
mats16
0
940
AWS Startup ゼミ 2021 秋期講習 / AWS Startup Seminar 2021 Autumn class - AWS Dev Day
mats16
4
2.3k
Unicorns run on AWS
mats16
0
130
AWS Startup tech Meetup Online 6
mats16
0
810
シードスタートアップに知っておいてほしいこと / What seed startups need to know
mats16
0
300
Introduction to AWS App Runner
mats16
0
890
Other Decks in Technology
See All in Technology
20240321_生成AI時代のDevOps
kzkmaeda
2
620
Getting started with controlling LEGO using Swift
hcrane
0
140
長文から長文を生成するLLMツールをオープンソースで作ってみた。
tomohisa
2
150
技術広報経験0のEMがエンジニアブランディングをはじめてみた
coconala_engineer
1
140
AMLD 2024 - Build Your Own GPT
donlelef
1
260
KubeCon EU: Unlocking new Platform Experiences with Open Interfaces
salaboy
1
380
大規模言語モデル開発のための日本語 Instruction データセット作成の取り組み
kunishou
1
180
依存ライブラリはどこに?
takesection
0
120
Why do you get AWS certificates
hirosys
0
120
Terraform v1.7のTest mocking機能の紹介 / Introducing the Test mocking feature of Terraform v1.7
yayoi_dd
1
100
中央集権体制からDataOpsへの転換 / centralized-to-dataops-transformation
pei0804
7
1.6k
バッチ処理のSLOをどう設計するか
rynsuke
7
590
Featured
See All Featured
Dealing with People You Can't Stand - Big Design 2015
cassininazir
355
22k
What's new in Ruby 2.0
geeforr
335
31k
Fireside Chat
paigeccino
19
2.6k
Navigating Team Friction
lara
177
13k
Typedesign – Prime Four
hannesfritz
36
2k
Pencils Down: Stop Designing & Start Developing
hursman
115
11k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
111
35k
How To Stay Up To Date on Web Technology
chriscoyier
781
250k
The Cult of Friendly URLs
andyhume
73
5.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
5
1.4k
Rails Girls Zürich Keynote
gr2m
91
13k
How to name files
jennybc
62
92k
Transcript
Amazon Web Searvice Ͱ࣮ݱ͢Δ elasticsearchͷେنӡ༻ × גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔɹদాथ
ࣗݾհ দా थ גࣜձࣾΠϯςΟϝʔτɾϚʔδϟʔʢࣾһ໊̕ʣ ։ൃຊ෦ • ύϒϦοΫDMPʮAudienceSearchʯͷ։ൃ • ओʹΠϯϑϥपΓΛ୲
ΞδΣϯμ • IMʹ͓͚ΔelasticsearchͷΘΕํ • elasticsearchͷߏ • ߏͷܾΊํ • AutoScalingͷϙΠϯτ •
ͦͷଞઃఆͷϙΠϯτ
elasticsearchͷΘΕํ ̐ԯͷϢχʔΫIDʹର༷ͯ͠ʑͳଐੑใΛ༩ • Ӿཡཤྺʹجͮ͘Ωʔϫʔυ • ֤ϝσΟΞΑΓఏڙ͞ΕΔηάϝϯτ • User Agent •
ΞΫηεݩIPʹجͮ͘ॅॴاۀใ ҙͷ݅ͰIDΛߜΓࠐΉͨΊʹɺɹɹɹɹɹɹɹɹɹɹɹɹΛ ར༻͍ͯ͠·͢ɻ
AudienceSearchʹ͍ͭͯ ϝσΟΞདྷ๚ऀ ωοτϦαʔν ࠂ৴πʔϧ ˞ɹਤҰྫͰ͢ ཱྀߦʹڵຯ ऩສ ԣࢢॴ
ঁੑ
શମߏ • શͯAWSڥ • Cluster: 1, Type: 1, Node: 86
• master: 3 (master:true, data: false) • data: 80 (master:false, data:true) • searcher: 2 (master:false, data: false) • indexer: 1 (master:false, data: false)ɹ˞όονॲཧ • version: 1.3.8 → 1.5.2 (5/26Ξοϓσʔτ)
શମߏ master data searcher x2 indexer indexerҎ֎ɺશAutoScalingͰӡ༻ x3 x80 ELB
ߏܾఆʹ͍ͭͯ
Cluster Size • Shard = Node ʹݻఆͯ͠ݕূ • 10 ʙ
200 ͱΛมߋ͠ɺԠ࣌ؒΛൺֱ • ʙ100Ґ·ͰʹԠͯ͡ύϑΥʔϚϯε্͕ • 200نͰAggregationͷύϑΥʔϚϯε͕ྼԽ →ɹूܭ݁ՌͷϚʔδίετͷ૿Ճʁ • σʔλ༰ྔΛՃຯ͠ɺ80Ͱελʔτ
Shard • Node = 80 ʹݻఆͯ͠ݕূʢc3.largeʣ (1) shard: 40, replica:
1 ʢshard/node = 0.5ʣ (2) shard: 80, reprica: 1 ʢshard/node = 1ʣ (3) shard: 160, replica:1 ʢshard/node = 2ʣ • shard/node ͕̍Λ͑Δͱܹ͘͠ύϑΥʔϚϯε͕ྼԽ ※ shard/nodeͰͳ͘ɺshard/cpuͷํ͕ॏཁ͔͠Εͳ͍ɾɾ
Instance Size • ΠϯσΫγϯάߴԽͷͨΊɺϩʔΧϧSSDͷੑೳΛॏࢹ →ɹEphemeral DiskΛ̎ͭੵΊΔc3ϑΝϛϦʔ͔Βબ • σʔλྔ͕ଟ͍߹xlargeҎ্ͷํ͕CPʹ༏ΕΔ →ɹجຊతʹՁ͕֨ഒʹͳΔͱɺ༰ྔഒʹͳΔ͕ɺ ɹɹlarge→xlargeͷ࣌ͷΈ30GB→80GB
• JVMͷϝϞϦͷׂྔશମͷ̎ʙׂ̏ʢdoc_valueʣ
AutoScallingͷϙΠϯτ
Why AutoScaling ? AutoScaling͍ͤͯ͞Δཧ༝ • ̍̌̌نͷαʔόΛ̍̍ཧͯ͠ӡ༻͢Δ͜ͱ͕ɺ ਓతʹݫ͔ͬͨ͠ • terminate͢Εݩͷڥ͕࠶ݱՄೳ •
εϙοτΠϯελϯε͕ར༻ՄೳʹͳΔ • ύϥϝʔλΛ͍͡Δ͚ͩͰεέʔϧΞτՄೳ
recoveryͷνϡʔχϯά • LifeCycleʹΑΓఆظతʹΠϯελϯε͕ೖΕସΘΔ͜ͱ͕͋Δ • ଳҬಉ࣌ҠಈՄೳͳγϟʔυΛௐ͠ɺrecoveryʹ͔͔Δ ࣌ؒΛॖ͍ͯ͠Δ • ϨϓϦΧ̍Ͱӡ༻ʢ0ͰΔͱ࣌ʑσʔλ͕ແ͘ͳΓ·͢ʣ indices: recovery:
max_bytes_per_sec: 500mb concurrent_streams: 4
recoveryͷνϡʔχϯά • LifeCycleʹΑΓՃ͞ΕͨΠϯελϯεͷෛՙ • σʔλྔ: 40GB Ͱ10ఔʢc3.xlargeʣ
ΫϥελϦϯάͷߴԽ • elasticsearch-cloud-awsϓϥάΠϯΛར༻ • λάࢦఆͰdiscoveryΛߦ͍ɺmaster nodeʹͷΈλάΛઃఆ͢Δ • SGࢦఆͰର͕ଟ͕͔͔͗ͯ࣌ؒ͢Δ discovery: type:
ec2 ec2: tag: es_cluster: production
ΫϥελϦϯάͷߴԽ • master nodeͷΦʔτεέʔϧͷઃఆ࣌ʹλάΛࢦఆ • ʮaws:autoscaling:groupNameʯλάར༻Մೳʁ
ELBʹΑΔࢹ • ݕࡧΛड͚ͳ͍ϊʔυʹؔͯ͠ɺࢮ׆ࢹ༻ʹELBͱͷ ඥ͚Λઃఆʢmaster, dataʣ • ࢹର9200൪ϙʔτʢ9300ͷํ͕͍͍͔ʣ
ͦͷଞͷϙΠϯτ
Խ • Multi-AZߏͰߏஙʢ˞3ڌҎ্ͷ߹ෆ໌ʣ • ԼهઃఆͰɺࣗϊʔυͷอ࣋͢ΔshardͷreplicaΛɺҟͳΔAZ ͷϊʔυ্ʹஔ͢Δ cluster: routing: allocation: awareness:
attributes: aws_availability_zone
Խ • ”aws_availability_zone”ʹࣗϊʔυͷஔ͞Ε͍ͯΔAZ͕ύϥ ϝʔλͱͯ͠༩͑ΒΕΔʢelasticsearch-cloud-awsʣ • Ұɺյͨ͠ࡍແࣄͰͨ͠ { "attributes" : {
"aws_availability_zone" : "ap-northeast-1c", "master" : "false" } }
ϊʔυ໊Λhostnameʹ͢Δ • ࣗಈͰઃఆ͞ΕΔhostname(ip-10-0-0-1)ʹɺES্ͷϊʔυ໊Λ ߹ΘͤΔ • ΕͯAMIʹݻΊͨΓ͢Δͱɺશϊʔυಉ໊͡લʹɾɾ node: name: ${HOSTNAME} export
HOSTNAME=$(hostname -s) /etc/sysconfig/elasticsearch /etc/elasticsearch/elasticsearch.yml
ࢹ ࢹͷ̏ຊபʢΦʔτεέʔϧͱͷ૬ੑॏࢹʣ • ELBͷϔϧενΣοΫ • MackerelʢϦιʔεɾϓϩηεࢹʣ • Papertrailʢϩάࢹʣ • ʢىಈ/ఀࢭ࣌ͷSlackͷwebhookʣ
ʢࢀߟʣPapertrail master data searcher x2 x3 x80 αʔόʹSSHෆཁͰɺରͷϩάΛtailՄೳ syslog
·ͱΊ • εέʔϧΞτͷ͠қ͞ͰɺAWSͱES૬ੑ͕ྑ͍ • ઃఆ࣍ୈͰ͋Δ͕ɺAutoScallingຊ൪Ͱͷӡ༻ʹ͑͏Δ • Cluster SizeҰఆҎ্͕ඞཁ • ϊʔυͷೖΕସΘΓΛߟྀ͢Δ͜ͱ͕ඞཁ
• 1ͭͷindex͕ԯ݅ϨϕϧͰ͋ͬͯɺelasticsearch࣮༻ʹ ͑ΒΕΔ
ΤϯδχΞืू ࠷৽ͷٕज़ʹڵຯͷ͋Δ৽ϝϯόʔΛืूதͰ͢ʂ ࠾༻ٕज़ ։ൃج൫
͋Γ͕ͱ͏͍͟͝·ͨ͠