Elasticsearchに入門してチューニングや監視をした話 #elasticsearchjp /elasticsearch-fukuoka-2017

Transcript

1. Elasticsearchʹೖ໳ͯ͠ νϡʔχϯά΍؂ࢹΛͨ͠࿩ Elasticsearchษڧձ in ෱Ԭ 2017 2017/10/06 @matsumana

2. ࣗݾ঺հ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ •

   Twitterɿ @matsumana

3. ΞδΣϯμ • ೖ໳͢Δ࣌ʹؾΛ͚ͭͨࣄ • ηοτΞοϓ • νϡʔχϯά • όοΫΞοϓͱϦΧόϦ •

   ϞχλϦϯά • ·ͱΊ

4. લఏ • ͜ͷεϥΠυͷ಺༰͸Elastic stack 5.4ʙ5.6࣌఺ͷ৘ใͰ͢ • LogstashͱBeats͸ࠓͷͱ͜Ζ࢖͏༧ఆͳ͠
 FluentdΛ࢖͍ͬͯΔ • X-Pack͸ແঈͰ࢖͑ΔMonitoring͚ͩ࢖͍ͬͯΔ

   • αʔϏεͷ؂ࢹΛطʹPrometheusͰߦ͍ͬͯΔͷͰ
 elasticsearch_exporterΛಋೖͯ͠PrometheusͰ؂ࢹ͢Δ

5. લఏ • ࠓճElasticsearchΛಋೖͨ͠ϓϩδΣΫτͷϢʔεέʔε͸ҎԼ • શจݕࡧ༻్Ͱ࢖͍ͬͯΔ • indexing͸1೔ʹ਺ճɺόονʹΑΔbulk indexingͷΈ • Ϣʔβ͸search͔͠ߦΘͳ͍

   • ͦͷϓϩδΣΫτͷαʔόαΠυͷݴޠ͸Java

6. ೖ໳͢Δ࣌ʹؾΛ͚ͭͨࣄ • ެࣜυΩϡϝϯτɺެࣜϒϩάΛಡΉ • άάͬͯग़͖ͯͨݸਓͷϒϩάͳͲ͸ग़དྷΔ͚ͩಡ·ͳ͍
 ʢݹ͍όʔδϣϯΛݩʹॻ͔Εͨهࣄʹ஫ҙʣ • ࡉ͔͍ڍಈΛ஌Γ͍ͨ࣌͸ιʔεΛಡΉ

7. ެࣜυΩϡϝϯτ͸
 ϘϦϡʔϜ͕ଟ͍ͷͰ ಛʹେࣄͦ͏ͳՕॴ͚ͩಡΜͩ https://matsumana.info/blog/2017/06/17/elasticsearch-getting-started-docs/ Ҏલϒϩάʹ·ͱΊ͍ͯ·͢

8. ηοτΞοϓ • ެࣜͷAnsible playbookͰηοτΞοϓ
 ʢࢼͨ͠ࣄ͸ͳ͍͚ͲɺPuppet൛ɺChef൛΋͋ΔΈ͍ͨʣ • ศརɻ͓͢͢Ί • 2017/10/03ݱࡏͩͱɺ5.5.1ͱ͍͏tagΛ࢖͏ͷ͕ྑ͍

9. ެࣜplaybook಺ͷॏཁͦ͏ͳม਺ es_version es_instance_name es_heap_size es_config.cluster.name es_config.node.name → "${HOSTNAME}" ͕ྑ͍ es_config.discovery.zen.ping.unicast.hosts

   es_config.node.master es_config.node.data es_config.bootstrap.memory_lock → true ͕ྑ͍ es_config.network.host → 0.0.0.0 es_jvm_custom_parameters → ௥Ճ͍ͨ͠JVM OptionΛࢦఆ es_plugins → analysis-icu΍analysis-kuromojiͳͲΛࢦఆՄೳ es_enable_xpack → X-PackΛ࢖͏৔߹͸true es_xpack_features → monitoringͳͲΛΠϯετʔϧՄೳ https://github.com/elastic/ansible-elasticsearch/blob/master/README.md

10. ࢖༻͍ͯ͠Δόʔδϣϯ • CentOS 7.3
 (ެࣜͷAnsible playbook͸CentOS 6Λαϙʔτ͍ͯ͠ͳ͍) • Oracle JDK

    1.8.0 • Elastic stack 5.5.1 • Elasticsearch • Kibana
 (X-Pack MonitoringͷμογϡϘʔυΛݟΔͨΊʹಋೖ) • X-Pack Monitoring ※Slideͷ࠷ޙʹัଊ͋Γ
 (ӡ༻ʹ໾ཱͭ৘ใ͕͋Δ͔ݟͯΈ͔ͨͬͨͷͰಋೖ) • Cerebro 0.6.6

11. νϡʔχϯά ʢαʔόαΠυʣ

12. • APIͰಈతʹઃఆͰ͖Δύϥϝʔλͱग़དྷͳ͍ύϥϝʔλ͕͋Δ • ಈతʹมߋͰ͖ͳ͍߲໨͸elasticsearch.ymlʹॻ͘
 ʢ൓өʹ͸Elasticsearchͷ࠶ىಈ͕ඞཁʣ • ಈతʹมߋͰ͖Δ߲໨͸APIͰมߋ͢Δ
 ʢElasticsearchΛ࠶ىಈ͠ͳͯ͘΋൓ө͞ΕΔʣ • elasticsearch.ymlͱAPIͷ྆ํͰಉ߲͡໨Λมߋͨ͠৔߹͸ɺ

    APIͰͷมߋ͕༏ઌ͞ΕΔ ஫ҙ఺

13. ઃఆͨ͠ओͳύϥϝʔλ

14. • Cacheؔ࿈ • indices.fielddata.cache.size • indices.queries.cache.size • indices.requests.cache.size • Thread

    poolؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • thread_pool.index.queue_size • thread_pool.bulk.queue_size elasticsearch.ymlͰߦ͏Ϋϥελશମͷઃఆ ʢಈతʹมߋͰ͖ͳ͍ʣ

15. • Circuit Breakerؔ࿈ • indices.breaker.total.limit • indices.breaker.fielddata.limit • indices.breaker.request.limit •

    Segments and Mergingؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • indices.store.throttle.max_bytes_per_sec
 σϑΥϧτ஋͸HDD޲͚ʹͳ͍ͬͯΔͷͰɺ
 SSDΛ࢖͏৔߹͸มߋ͢ΔΑ͏ʹDefinitive Guideʹॻ͍ͯ͋Δ APIͰߦ͏Ϋϥελશମͷઃఆ
 ʢಈతʹมߋͰ͖Δʣ

16. • number_of_shards • number_of_replicas
 bulk indexingத͸0ʹઃఆɻऴΘͬͨΒ໭͢ • _allΛແޮԽ • dynamic:

    strict • index.indexing.slowlogؔ࿈ • index.search.slowlogؔ࿈ APIͰߦ͏index/index templateຖͷઃఆ ʢಈతʹมߋͰ͖Δʣ

17. νϡʔχϯά ʢΫϥΠΞϯταΠυʣ

18. • ࠷ۙɺElasticsearchࣾ͸REST Clientਪ͠ͳงғؾ • Transport ClientͱҧͬͯαʔόͱΫϥΠΞϯτϥΠϒϥϦͷ όʔδϣϯΛҰகͤ͞ͳͯ͘ྑ͍ • ύϑΥʔϚϯε΋Transport Clientͱಉ͘͡Β͍Β͍͠

    • Elasticsearch 5.6Ͱ͸Java High Level REST Client΋
 ϦϦʔε͞Εͨ REST Client

19. • Transport ClientͰbulkॲཧ͢Δ৔߹͸ɺ
 BulkProcessorΛ࢖͏ͱྑ͍ɻҎԼͷύϥϝʔλ͕ઃఆՄೳ • bulkActions • bulkSize • flushInterval

    • concurrentRequests • backoffPolicy https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html Transport Client

20. όοΫΞοϓͱϦΧόϦ • ࠓճͷϓϩδΣΫτͰ͸ɺݩͷσʔλ͸શͯMySQLʹอଘ͞Ε͍ͯͯ
 ̍೔ʹ਺ճόονͰElasticsearchʹશ݅builk indexing͍ͯ͠Δ
 ElasticsearchͷόοΫΞοϓ͸औ͍ͬͯͳ͍ • ElasticsearchΫϥελશମ͕յΕΔͱ͍͏࠷ѱͷέʔεͷ৔߹ɺ
 ElasticsearchΫϥελΛAnsibleͰ࠶ߏஙͯ͠ɺ͍ͭ΋ͷఆظόονͰ
 શ݅builk

    indexing͢Δ૝ఆ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

21. ϞχλϦϯά • ࢖͍ͬͯΔPrometheus exporter • node_exporter • jmx_exporter • elasticsearch_exporter

    • άάΔͱ͍͔ͭ͘ݟ͔ͭΔ͕ɺݱࡏ࢖͍ͬͯΔͷ͸͜Ε
 https://github.com/justwatchcom/elasticsearch_exporter • authorͱcontributorୡ͕ඞཁͳϝτϦΫε͔͠export͞Εͯ ͍ͳ͍ͷͰɺࣗ෼͕ඞཁͳϝτϦΫε͕ແ͚Ε͹PR͠Α͏

22. • Load Avg • CPU࢖༻཰ • Disk I/O • network

    trafic node_exporterͰऩूͰ͖Δ OSʹؔ͢ΔϝτϦΫεͷҰྫ

23. • Heap࢖༻཰ • GCճ਺ • GCॴ༻࣌ؒ jmx_exporterͰऔಘͰ͖Δ JVMʹؔ͢ΔϝτϦΫεͷҰྫ

24. • cluster health • Document count(primary shardͷΈ) • Document size(primary

    shardͷΈ) • Thread pool queue count • Thread pool rejected count • Circuit Breaker tripped count elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ

25. • Fielddata cache size • Query cache size • Query

    cache ώοτ཰ (hit / total) • Request cache size elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ

26. elasticsearch_exporterҎ֎Ͱ औಘ͍ͯ͠Δ Elasticsearchؔ࿈ͷϝτϦΫε

27. • ϦΫΤετճ਺ • ྫ֎ൃੜճ਺ • Search latency percentile (0.5, 0.75,

    0.98, 0.99, 0.999) prometheus_clientΛαʔόαΠυΞϓϦʹ૊ΈࠐΜͰ DropWizard MetricsͰऩू͍ͯ͠ΔϝτϦΫε

28. • Slow logൃੜճ਺ • NorikraͰूܭ • Raw slow log •

    Elasticsearchʹอଘ • Elasticsearchʹอଘͨ͠σʔλ͸KibanaͰݟΔͷ͕Ұൠత ͩͱࢥ͏͕ɺ࢖͍׳Ε͍ͯΔGrafanaͰҰ୴ݟΔࣄʹͨ͠ slow logΛFluentdͰऩू

29. slow log ՄࢹԽͷ֓ཁਤ

30. ·ͱΊ

31. • ެࣜυΩϡϝϯτΛಡ΋͏ • ެࣜAnsible playbook͓͢͢Ί • ύϥϝʔλͷઃఆมߋ͸ɺAPIͰಈతʹग़དྷΔ΋ͷͱग़དྷͳ͍ ΋ͷ͕͋Δ • elasticsearch_exporter͚ͩͰ͸ऩूͰ͖ͳ͍ϝτϦΫε͸ผͷ

    ํ๏Ͱɻ • αʔόαΠυΞϓϦʹDropWizard MetricsΛ૊ΈࠐΜͰܭଌ • slow logΛऩूͯ͠NorikraͰूܭ • Raw slow log͸ผͷElasticsearchʹอଘ

32. ิ଍ • X-Pack MonitoringΛΠϯετʔϧ͢ΔͱɺϝτϦΫεσʔλ Λอଘ͢Δઐ༻ͷΠϯσοΫε͕ग़དྷΔɻΠϯσοΫεͷอ ଘઌ͸ผͷElasticsearchΫϥελʹ͠·͠ΐ͏ • Thread poolؔ࿈ͷύϥϝʔλνϡʔχϯάɺqueue_size͚ͩ ૿΍ͯ͠΋ύϑΥʔϚϯε͸্͕ΒͣΩϡʔ଴͕ͪ૿͑Δ͚ͩ

    ͳͷͰɺsize΋߹Θͤͯνϡʔχϯά͠·͠ΐ͏
 https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html • indices.store.throttle.max_bytes_per_sec ͸Elasticsearch 6.0 Ͱഇࢭ͞Ε·͢
 https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_settings_changes.html

33. Thank you :)