Monitoring Solr

Monitoring Apache Solr and its index with Prometheus and Grafana.

Transcript

1. Monitoring Solr
2. େਢլ ູ (͓͓͔͢ ΈͷΔ) • Apache ιϑτ΢ΣΞࡒஂ ίϛολʔ
 https://www.apache.org • θοτϥϘגࣜձࣾ / Ϡϑʔגࣜձࣾ
 https://zlab.co.jp / https://www.yahoo.co.jp • [վగୈ3൛] Apache Solrೖ໳ - ΦʔϓϯιʔεશจݕࡧΤϯδϯ ڞචऀ
 http://gihyo.jp/book/2017/978-4-7741-8930-7
 http://gihyo.jp/book/2014/978-4-7741-6163-1 ࣗݾ঺հ
3. Z Lab ͷ Solr Ϣʔεέʔε ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϩάͷอଘ LOG LOG LOG Monitoring
 Alerting Visualize LOG LOG LOG etc... Log store Banana Visualize
4. Solr ͷϞχλϦϯά
5. γεςϜμ΢ϯ !
6. Solr ϞχλϦϯάʹ࢖͑Δ API • Ping - https://lucene.apache.org/solr/guide/ping.html • MBean Request Handler - https://lucene.apache.org/solr/guide/mbean-request-handler.html • CoreAdmin API - https://lucene.apache.org/solr/guide/coreadmin-api.html • Collections API - https://lucene.apache.org/solr/guide/collections-api.html • Metrics API - https://lucene.apache.org/solr/guide/metrics-reporting.html • etc...
7. ෆศ !
8. ϝτϦΫεϞχλϦϯά
9. Solr ͷαϙʔτ͢ΔϝτϦΫεϞχλϦϯάιϑτ΢ΣΞ • jconsole - https://docs.oracle.com/javase/9/tools/jconsole.html • Graphite - http://graphiteapp.org • Ganglia - http://ganglia.info • etc... Reporter Λ࢖ͬͯ Solr ͷϝτϦΫεΛ GraphiteɺGanglia ͳͲ΁ૹ৴Մೳ • Reporters - https://lucene.apache.org/solr/guide/metrics-reporting.html#reporters
10. Z Lab ͰͷϝτϦΫεϞχλϦϯά Monitoring
 Alerting Visualize etc... node_exporter jmx_exporter mysqld_exporter ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϝτϦΫεΛ
 Prometheus ΁อଘ
11. ϩάϞχλϦϯά
12. Z Lab ͰͷϩάϞχλϦϯά ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϩάΛ Solr ΁อଘ LOG LOG LOG LOG LOG LOG etc... Log store Banana Visualization ՄࢹԽͷΈ !
13. ϩάͷϞχλϦϯά͸ʁ LOG LOG LOG LOG LOG LOG Log store Log collector ʁ etc... Visualization ӡ༻͢Δ؂ࢹγεςϜ૿ ! ϊ΢ϋ΢ͷ͋Δ Solr ͰͳΜͱ͔ !
14. Z Lab ͰͷϩάϞχλϦϯά طଘͷ؂ࢹγεςϜͱ Solr Λ
 ͏·͘࿈ܞͤ͞Δ LOG LOG LOG Log / Metrics
 Monitoring,
 Alerting Metrics Visualization LOG LOG LOG Log store Banana Log Visualization etc...
15. CONTRIBUTED !
16. Prometheus ͱ Grafana Λ࢖ͬͨ Solr ͷϞχλϦϯά • Solr 7.3.0 ͔Β Solr ͷϝτϦΫεΛ Prometheus ʹެ։͢ΔͨΊͷ Exporter Λ௥Ճ - Solr ͷҰൠతͳϝτϦΫεΛσϑΥϧτͰऩू - Grafana ͷαϯϓϧμογϡϘʔυΛఏڙ • GitHub
 ɹɹhttps://github.com/apache/lucene-solr/tree/master/solr/contrib/prometheus-exporter • Solr Ref Guide
 ɹɹhttp://lucene.apache.org/solr/guide/monitoring-solr-with-prometheus-and-grafana.html

17. Prometheus • ࣌ܥྻσʔλϕʔεΛ಺แͨ͠Φʔϓϯιʔεͷ
 Pull ܕͷαʔϏε؂ࢹγεςϜ • ಠࣗͷΫΤϦݴޠΛ࣋ͭ • ݩ Google ͷΤϯδχΞ͕ Google ࣾ಺γεςϜ (Borg) ͷ
 ؂ࢹγεςϜ (Borgmon) ʹΠϯεύΠΞ͞Εͯ։ൃ • Exporter Λ࡞੒͢Δ͜ͱͰɺϝτϦΫεΛ Prometheus ͕ऩूͰ͖Δ https://prometheus.io
18. Grafana • Kibana ͔ΒϑΥʔΫͨ͠Φʔϓϯιʔεͷ
 σʔλՄࢹԽπʔϧ • ࣌ܥྻσʔλͷՄࢹԽʹϑΥʔΧε • ՄࢹԽ͚ͩͰͳ͘ɺΞϥʔςΟϯά΋ߦ͑ɺϓϥάΠϯʹΑΔ
 μογϡϘʔυͷ֦ு͕Մೳ • Prometheus ΍ InfluxDB ͳͲɺଟ͘ͷ࣌ܥྻσʔλϕʔεʹରԠ https://grafana.com
19. solr-exporter • Prometheus ͕ऩूͰ͖ΔϑΥʔϚοτͰ Solr ͷ
 ϝτϦΫεΛऩूɾެ։͢Δখ͞ͳ HTTP αʔό • Java Ͱ։ൃ (SolrJ Λར༻) • ελϯυΞϩʔϯͷ Solr ͚ͩͰͳ͘ SolrCloud ʹ΋ରԠ • Solr ͷݕࡧ݁Ռ΋ϝτϦΫεͱͯ͠ѻ͏͜ͱ͕Ͱ͖Δ
20. solr-exporter Query Pull Metrics Push alerts Notify Query Visualize Find targets Service Discovery Find nodes Register
21. solr-exporter ͷىಈ • Standalone mode (Linux) • SolrCloud mode (Linux) • ϝτϦΫεެ։ URL
 ɹɹhttp://localhost:9854/metrics $ cd contrib/prometheus-exporter $ ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8 $ cd contrib/prometheus-exporter $ ./bin/solr-exporter -p 9854 -z localhost:2181/solr -f ./conf/solr-exporter-config.xml -n 16
22. solr-exporter ͷઃఆ • solr-exporter Λىಈ͢Δ͚ͩͰɺ͙͢ʹओཁͳϝτϦΫεͷऩू͕Մೳ • Solr ͷϨεϙϯεͷ JSON Λύʔε͢Δͷʹ jq γϯλοΫεΛ࠾༻ • ඞཁʹԠͯ͡ solr-exporter-config.xml Λฤू͠ɺϝτϦΫεͷऩूํ๏ͳͲมߋՄೳ - ऩू͢ΔϝτϦΫεͷ௥Ճɾ࡟আ - ϝτϦΫε໊ͷมߋ - etc
23. solr-exporter ͷઃఆ • solr-exporter ͷઃఆϑΝΠϧ͸γϯϓϧͰɺ࣍ͷΑ͏ͳλάͰߏ੒ • rules λά಺ʹ࣍ͷϦΫΤετΛؚΉ͜ͱ͕Մೳ <config> <rules> </rules> </config> <ping /> Ping ϦΫΤετϋϯυϥͷϨεϙϯεͷεΫϨΠϐϯά <metrics /> Metrics API ͷϨεϙϯεͷεΫϨΠϐϯά <collections /> Collections API ͷϨεϙϯεͷεΫϨΠϐϯά <search /> ݕࡧϦΫΤετͷϨεϙϯεͷεΫϨΠϐϯά
24. <ping /> <ping> <lst name="request"> <lst name="query"> <str name="path">/admin/ping</str> </lst> <arr name="jsonQueries"> <str> . as $object | $object | (if $object.status == "OK" then 1.0 else 0.0 end) as $value | { name : "solr_ping", type : "GAUGE", help : "See following URL: https://lucene.apache.org/solr/guide/ping.html", label_names : [], label_values : [], value : $value } </str> </arr> </lst> </ping>
25. <metrics /> <metrics> <lst name="request"> <lst name="query"> <str name="path">/admin/metrics</str> <lst name="params"> <str name="group">all</str> <str name="type">all</str> </lst> </lst> <arr name="jsonQueries"> <str> .metrics["solr.jetty"] | to_entries | .[] |
 select(.key | startswith("org.eclipse.jetty.server.handler.DefaultHandler")) | 
 select(.key | endswith("xx-responses")) as $object | $object.key | split(".") | last | split("-") | first as $status | $object.value.count as $value | { name : "solr_metrics_jetty_response_total", type : "COUNTER", help : "See following URL: https://lucene.apache.org/solr/guide/metrics-reporting.html", label_names : ["status"], label_values : [$status], value : $value } </str> </arr> </lst> </metrics>
26. <collections /> <collections> <lst name="request"> <lst name="query"> <str name="path">/admin/collections</str> <lst name="params"> <str name="action">CLUSTERSTATUS</str> </lst> </lst> <arr name="jsonQueries"> <str> .cluster.live_nodes | length as $value | { name : "solr_collections_live_nodes", type : "GAUGE", help : "See following URL: https://lucene.apache.org/solr/guide/collections-api.html", label_names : [], label_values : [], value : $value } </str> </arr> </lst> </collections>
27. <search /> <search> <lst name="request"> <lst name="query"> <str name="core">collection1</str> <str name="path">/select</str> <lst name="params"> <str name="q">*:*</str> <str name="json.facet"> { category: { type: terms, field: cat } } </str> </lst> </lst> <arr name="jsonQueries"> <str> .facets.category.buckets[] as $object | $object.val as $term | $object.count as $value | { name : "solr_facets_category", type : "GAUGE", help : "Category facets", label_names : ["term"], label_values : [$term], value : $value } </str> </arr> </lst> </search>
28. ϝτϦΫεެ։ϑΥʔϚοτ • PrometheusͷϝτϦΫεެ։ϑΥʔϚοτ • # TYPE <name> <type> # HELP <name> <help> <name>{<label_names[0]>=<label_values[0]>,<label_names[1]>=<label_values[1]>,...} <value> name ϝτϦοΫ໊ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ type ϝτϦοΫܕ (COUNTERɺGAUGEɺSUMMARYɺHISTOGRAMɺUNTYPED)
 - ৄࡉ͸ https://prometheus.io/docs/concepts/metric_types/ help ϝτϦοΫͷઆ໌ label_names ϝτϦοΫͷϥϕϧ໊ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ label_values ϝτϦοΫͷϥϕϧ஋ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ value ϝτϦοΫ஋ (Double ܕ)
29. ϝτϦΫεެ։ϑΥʔϚοτ • JSON (ม׵લ) • Prometheus Exposition Format (ม׵ޙ) { "name": "solr_ping", "type": "GAUGE", "help": "See following URL: https://lucene.apache.org/solr/guide/ping.html", "label_names": ["base_url","core"], "label_values": ["http://localhost:8983/solr","collection1"], "value": 1.0 } # TYPE solr_ping gauge # HELP solr_ping See following URL: https://lucene.apache.org/solr/guide/ping.html solr_ping{base_url="http://localhost:8983/solr",core="collection1"} 1.0
30. ϝτϦΫεެ։ϑΥʔϚοτ $ curl -s http://localhost:9894/metrics # HELP solr_metrics_jetty_response_total See following URL: https://lucene.apache.org/solr/... # TYPE solr_metrics_jetty_response_total counter solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="1xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="2xx",} 516.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="3xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="4xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="5xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="1xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="2xx",} 517.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="3xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="4xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="5xx",} 0.0 ... [தུ] ...
31. Prometheus ͷઃఆ • prometheus.yml ͷ scrape_configs ʹ solr-exporter ͷઃఆΛ௥Ճ scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: 'solr' static_configs: - targets: ['localhost:9854']
32. • lucene-solr/solr/contrib/prometheus-exporter/conf ʹαϯϓϧͷ
 μογϡϘʔυΛಉࠝ - grafana-solr-dashboard.json • Grafana ʹμογϡϘʔυΛొ࿥͙ͯ͢͠ར༻։࢝Ͱ͖Δ • ඞཁʹԠͯ͡μογϡϘʔυΛฤूͯ͠ΧελϚΠζ Grafana μογϡϘʔυ
34. solr-exporter Λ࢖ͬͯͰ͖ΔΑ͏ʹͳΔ͜ͱ • Solr ͷ༷ʑͳϝτϦΫεΛऩूՄೳ • Solr ͷΠϯσοΫεͷ಺༰ΛϝτϦΫεͱͯ͠ऩूՄೳ - ΞΫηεϩά΍ΞϓϦέʔγϣϯϩάΛΠϯσοΫε͍ͯͨ͠Β... - ΞΫηε݅਺ͷՄࢹԽ΍ɺΞϓϦέʔγϣϯͷΤϥʔͷݕ஌ - EC αʔϏεͰ঎඼৘ใΛΠϯσοΫε͍ͯͨ͠Β... - ঎඼ΧςΰϦʔຖͷ঎඼఺਺ͷਪҠ΍ɺҧ๏঎඼ͷݕ஌ - SNS ͳͲͷϢʔβ౤ߘܕͷσʔλΛΠϯσοΫε͍ͯͨ͠Β... - ౤ߘ͞ΕΔίϯςϯπʹؚ·ΕΔΩʔϫʔυͷਪҠɺެংྑଏʹ൓͢Δίϯςϯπͷݕ஌ - etc..
35. ·ͱΊ • Solr ͷՔಇঢ়گ͚ͩͰͳ͘ɺΠϯσοΫε΋ϞχλϦϯά - CPU ΍ετϨʔδͷར༻཰ͳͲͷϝτϦΫε͚ͩͰͳ͘ɺ
 อଘ͍ͯ͠Δσʔλ͕݈શͳঢ়ଶ͔΋νΣοΫͰ͖Δ • ͦΕͧΕͷ༻్ʹ߹Θͤͯιϑτ΢ΣΞΛબͿ - సஔΠϯσοΫεΛ࢖ͬͨσʔλͷݕࡧ͸ݕࡧΤϯδϯ - ࣌ܥྻσʔλͷूܭ΍ղੳͳͲ͸࣌ܥྻσʔλϕʔε
36. Monitoring Solr