Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchに入門してチューニングや監視をした話 #elasticsearchj...
Search
Manabu Matsuzaki
October 06, 2017
Technology
1
1.3k
Elasticsearchに入門してチューニングや監視をした話 #elasticsearchjp /elasticsearch-fukuoka-2017
Elasticsearch勉強会 in 福岡 2017の発表資料です
Manabu Matsuzaki
October 06, 2017
Tweet
Share
More Decks by Manabu Matsuzaki
See All by Manabu Matsuzaki
Spring BootユーザのためのArmeria入門 #jsug / Introduce to Armeria for Spring users
matsumana
0
2.8k
Canary Release with Argo Rollouts #ふくばねてす / canary-release-with-argo-rollouts
matsumana
1
1.2k
Getting started Central Dogma with Golang #fukuokago #umedago / getting-started-central-dogma-with-golang
matsumana
0
890
Micrometer入門 #javaq / introduce-to-micrometer
matsumana
1
2.9k
ArmeriaとCentral Dogmaから学ぶ、マイクロサービスに必要な機能 #edayfuk / lean-from-armeria-and-central-dogma
matsumana
0
4.3k
SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report
matsumana
0
880
SRE入門 & チームで取り組んでいるSRE #srefukuoka / introduce-to-sre
matsumana
0
1.3k
Introduce to Armeria and Central Dogma #GWD_Nulab / introduce-to-armeria-and-central-dogma
matsumana
0
560
Connector/JでMaster/Slave Replication構成のMySQLに接続する #mysql_casual_fukuoka /connector-j-master-slave-replication
matsumana
0
1.5k
Other Decks in Technology
See All in Technology
Expertise as a Service via MCP
yodakeisuke
0
130
データエンジニアリング 4年前と変わったこと、 4年前と変わらないこと
tanakarian
2
330
PHPでResult型やってみよう
higaki_program
0
180
Shadow DOM & Security - Exploring the boundary between light and shadow
masatokinugawa
0
640
Autify Company Deck
autifyhq
2
44k
なぜAI時代に 「イベント」を中心に考えるのか? / Why focus on "events" in the age of AI?
ytake
0
140
振り返りTransit Gateway ~VPCをいい感じでつなげるために~
masakiokuda
4
230
AIコードアシスタントとiOS開発
jollyjoester
1
220
ゼロから始めるSREの事業貢献 - 生成AI時代のSRE成長戦略と実践 / Starting SRE from Day One
shinyorke
PRO
0
220
"Découvrir le Liberland"
rlifchitz
0
130
低レイヤソフトウェア技術者が YouTuberとして食っていこうとした話
sat
PRO
7
5.8k
そもそも AWS FIS について。なぜ今 FIS のハンズオンなのか?などなど
kazzpapa3
2
110
Featured
See All Featured
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Music & Morning Musume
bryan
46
6.7k
Thoughts on Productivity
jonyablonski
69
4.7k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
RailsConf 2023
tenderlove
30
1.2k
It's Worth the Effort
3n
185
28k
What's in a price? How to price your products and services
michaelherold
246
12k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
The Pragmatic Product Professional
lauravandoore
35
6.8k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Navigating Team Friction
lara
187
15k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Transcript
Elasticsearchʹೖͯ͠ νϡʔχϯάࢹΛͨ͠ Elasticsearchษڧձ in Ԭ 2017 2017/10/06 @matsumana
ࣗݾհ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ •
Twitterɿ @matsumana
ΞδΣϯμ • ೖ͢Δ࣌ʹؾΛ͚ͭͨࣄ • ηοτΞοϓ • νϡʔχϯά • όοΫΞοϓͱϦΧόϦ •
ϞχλϦϯά • ·ͱΊ
લఏ • ͜ͷεϥΠυͷ༰Elastic stack 5.4ʙ5.6࣌ͷใͰ͢ • LogstashͱBeatsࠓͷͱ͜Ζ͏༧ఆͳ͠ FluentdΛ͍ͬͯΔ • X-PackແঈͰ͑ΔMonitoring͚͍ͩͬͯΔ
• αʔϏεͷࢹΛطʹPrometheusͰߦ͍ͬͯΔͷͰ elasticsearch_exporterΛಋೖͯ͠PrometheusͰࢹ͢Δ
લఏ • ࠓճElasticsearchΛಋೖͨ͠ϓϩδΣΫτͷϢʔεέʔεҎԼ • શจݕࡧ༻్Ͱ͍ͬͯΔ • indexing1ʹճɺόονʹΑΔbulk indexingͷΈ • Ϣʔβsearch͔͠ߦΘͳ͍
• ͦͷϓϩδΣΫτͷαʔόαΠυͷݴޠJava
ೖ͢Δ࣌ʹؾΛ͚ͭͨࣄ • ެࣜυΩϡϝϯτɺެࣜϒϩάΛಡΉ • άάͬͯग़͖ͯͨݸਓͷϒϩάͳͲग़དྷΔ͚ͩಡ·ͳ͍ ʢݹ͍όʔδϣϯΛݩʹॻ͔Εͨهࣄʹҙʣ • ࡉ͔͍ڍಈΛΓ͍ͨ࣌ιʔεΛಡΉ
ެࣜυΩϡϝϯτ ϘϦϡʔϜ͕ଟ͍ͷͰ ಛʹେࣄͦ͏ͳՕॴ͚ͩಡΜͩ https://matsumana.info/blog/2017/06/17/elasticsearch-getting-started-docs/ Ҏલϒϩάʹ·ͱΊ͍ͯ·͢
ηοτΞοϓ • ެࣜͷAnsible playbookͰηοτΞοϓ ʢࢼͨ͠ࣄͳ͍͚ͲɺPuppet൛ɺChef൛͋ΔΈ͍ͨʣ • ศརɻ͓͢͢Ί • 2017/10/03ݱࡏͩͱɺ5.5.1ͱ͍͏tagΛ͏ͷ͕ྑ͍
ެࣜplaybookͷॏཁͦ͏ͳม es_version es_instance_name es_heap_size es_config.cluster.name es_config.node.name → "${HOSTNAME}" ͕ྑ͍ es_config.discovery.zen.ping.unicast.hosts
es_config.node.master es_config.node.data es_config.bootstrap.memory_lock → true ͕ྑ͍ es_config.network.host → 0.0.0.0 es_jvm_custom_parameters → Ճ͍ͨ͠JVM OptionΛࢦఆ es_plugins → analysis-icuanalysis-kuromojiͳͲΛࢦఆՄೳ es_enable_xpack → X-PackΛ͏߹true es_xpack_features → monitoringͳͲΛΠϯετʔϧՄೳ https://github.com/elastic/ansible-elasticsearch/blob/master/README.md
༻͍ͯ͠Δόʔδϣϯ • CentOS 7.3 (ެࣜͷAnsible playbookCentOS 6Λαϙʔτ͍ͯ͠ͳ͍) • Oracle JDK
1.8.0 • Elastic stack 5.5.1 • Elasticsearch • Kibana (X-Pack MonitoringͷμογϡϘʔυΛݟΔͨΊʹಋೖ) • X-Pack Monitoring ※Slideͷ࠷ޙʹัଊ͋Γ (ӡ༻ʹཱͭใ͕͋Δ͔ݟͯΈ͔ͨͬͨͷͰಋೖ) • Cerebro 0.6.6
νϡʔχϯά ʢαʔόαΠυʣ
• APIͰಈతʹઃఆͰ͖Δύϥϝʔλͱग़དྷͳ͍ύϥϝʔλ͕͋Δ • ಈతʹมߋͰ͖ͳ͍߲elasticsearch.ymlʹॻ͘ ʢөʹElasticsearchͷ࠶ىಈ͕ඞཁʣ • ಈతʹมߋͰ͖Δ߲APIͰมߋ͢Δ ʢElasticsearchΛ࠶ىಈ͠ͳͯ͘ө͞ΕΔʣ • elasticsearch.ymlͱAPIͷ྆ํͰಉ߲͡Λมߋͨ͠߹ɺ
APIͰͷมߋ͕༏ઌ͞ΕΔ ҙ
ઃఆͨ͠ओͳύϥϝʔλ
• Cacheؔ࿈ • indices.fielddata.cache.size • indices.queries.cache.size • indices.requests.cache.size • Thread
poolؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • thread_pool.index.queue_size • thread_pool.bulk.queue_size elasticsearch.ymlͰߦ͏Ϋϥελશମͷઃఆ ʢಈతʹมߋͰ͖ͳ͍ʣ
• Circuit Breakerؔ࿈ • indices.breaker.total.limit • indices.breaker.fielddata.limit • indices.breaker.request.limit •
Segments and Mergingؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • indices.store.throttle.max_bytes_per_sec σϑΥϧτHDD͚ʹͳ͍ͬͯΔͷͰɺ SSDΛ͏߹มߋ͢ΔΑ͏ʹDefinitive Guideʹॻ͍ͯ͋Δ APIͰߦ͏Ϋϥελશମͷઃఆ ʢಈతʹมߋͰ͖Δʣ
• number_of_shards • number_of_replicas bulk indexingத0ʹઃఆɻऴΘͬͨΒ͢ • _allΛແޮԽ • dynamic:
strict • index.indexing.slowlogؔ࿈ • index.search.slowlogؔ࿈ APIͰߦ͏index/index templateຖͷઃఆ ʢಈతʹมߋͰ͖Δʣ
νϡʔχϯά ʢΫϥΠΞϯταΠυʣ
• ࠷ۙɺElasticsearchࣾREST Clientਪ͠ͳงғؾ • Transport ClientͱҧͬͯαʔόͱΫϥΠΞϯτϥΠϒϥϦͷ όʔδϣϯΛҰகͤ͞ͳͯ͘ྑ͍ • ύϑΥʔϚϯεTransport Clientͱಉ͘͡Β͍Β͍͠
• Elasticsearch 5.6ͰJava High Level REST Client ϦϦʔε͞Εͨ REST Client
• Transport ClientͰbulkॲཧ͢Δ߹ɺ BulkProcessorΛ͏ͱྑ͍ɻҎԼͷύϥϝʔλ͕ઃఆՄೳ • bulkActions • bulkSize • flushInterval
• concurrentRequests • backoffPolicy https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html Transport Client
όοΫΞοϓͱϦΧόϦ • ࠓճͷϓϩδΣΫτͰɺݩͷσʔλશͯMySQLʹอଘ͞Ε͍ͯͯ ̍ʹճόονͰElasticsearchʹશ݅builk indexing͍ͯ͠Δ ElasticsearchͷόοΫΞοϓऔ͍ͬͯͳ͍ • ElasticsearchΫϥελશମ͕յΕΔͱ͍͏࠷ѱͷέʔεͷ߹ɺ ElasticsearchΫϥελΛAnsibleͰ࠶ߏஙͯ͠ɺ͍ͭͷఆظόονͰ શ݅builk
indexing͢Δఆ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
ϞχλϦϯά • ͍ͬͯΔPrometheus exporter • node_exporter • jmx_exporter • elasticsearch_exporter
• άάΔͱ͍͔ͭ͘ݟ͔ͭΔ͕ɺݱࡏ͍ͬͯΔͷ͜Ε https://github.com/justwatchcom/elasticsearch_exporter • authorͱcontributorୡ͕ඞཁͳϝτϦΫε͔͠export͞Εͯ ͍ͳ͍ͷͰɺ͕ࣗඞཁͳϝτϦΫε͕ແ͚ΕPR͠Α͏
• Load Avg • CPU༻ • Disk I/O • network
trafic node_exporterͰऩूͰ͖Δ OSʹؔ͢ΔϝτϦΫεͷҰྫ
• Heap༻ • GCճ • GCॴ༻࣌ؒ jmx_exporterͰऔಘͰ͖Δ JVMʹؔ͢ΔϝτϦΫεͷҰྫ
• cluster health • Document count(primary shardͷΈ) • Document size(primary
shardͷΈ) • Thread pool queue count • Thread pool rejected count • Circuit Breaker tripped count elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ
• Fielddata cache size • Query cache size • Query
cache ώοτ (hit / total) • Request cache size elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ
elasticsearch_exporterҎ֎Ͱ औಘ͍ͯ͠Δ Elasticsearchؔ࿈ͷϝτϦΫε
• ϦΫΤετճ • ྫ֎ൃੜճ • Search latency percentile (0.5, 0.75,
0.98, 0.99, 0.999) prometheus_clientΛαʔόαΠυΞϓϦʹΈࠐΜͰ DropWizard MetricsͰऩू͍ͯ͠ΔϝτϦΫε
• Slow logൃੜճ • NorikraͰूܭ • Raw slow log •
Elasticsearchʹอଘ • Elasticsearchʹอଘͨ͠σʔλKibanaͰݟΔͷ͕Ұൠత ͩͱࢥ͏͕ɺ͍׳Ε͍ͯΔGrafanaͰҰ୴ݟΔࣄʹͨ͠ slow logΛFluentdͰऩू
slow log ՄࢹԽͷ֓ཁਤ
·ͱΊ
• ެࣜυΩϡϝϯτΛಡ͏ • ެࣜAnsible playbook͓͢͢Ί • ύϥϝʔλͷઃఆมߋɺAPIͰಈతʹग़དྷΔͷͱग़དྷͳ͍ ͷ͕͋Δ • elasticsearch_exporter͚ͩͰऩूͰ͖ͳ͍ϝτϦΫεผͷ
ํ๏Ͱɻ • αʔόαΠυΞϓϦʹDropWizard MetricsΛΈࠐΜͰܭଌ • slow logΛऩूͯ͠NorikraͰूܭ • Raw slow logผͷElasticsearchʹอଘ
ิ • X-Pack MonitoringΛΠϯετʔϧ͢ΔͱɺϝτϦΫεσʔλ Λอଘ͢Δઐ༻ͷΠϯσοΫε͕ग़དྷΔɻΠϯσοΫεͷอ ଘઌผͷElasticsearchΫϥελʹ͠·͠ΐ͏ • Thread poolؔ࿈ͷύϥϝʔλνϡʔχϯάɺqueue_size͚ͩ ૿ͯ͠ύϑΥʔϚϯε্͕ΒͣΩϡʔ͕ͪ૿͑Δ͚ͩ
ͳͷͰɺsize߹Θͤͯνϡʔχϯά͠·͠ΐ͏ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html • indices.store.throttle.max_bytes_per_sec Elasticsearch 6.0 Ͱഇࢭ͞Ε·͢ https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_settings_changes.html
Thank you :)