Tweet
Tweet

Slide 1

Slide 1 text

Monitoring Solr

Slide 2

Slide 2 text

େਢլ ູ (͓͓͔͢ ΈͷΔ) • Apache ιϑτ΢ΣΞࡒஂ ίϛολʔ
 https://www.apache.org • θοτϥϘגࣜձࣾ / Ϡϑʔגࣜձࣾ
 https://zlab.co.jp / https://www.yahoo.co.jp • [վగୈ3൛] Apache Solrೖ໳ - ΦʔϓϯιʔεશจݕࡧΤϯδϯ ڞචऀ
 http://gihyo.jp/book/2017/978-4-7741-8930-7
 http://gihyo.jp/book/2014/978-4-7741-6163-1 ࣗݾ঺հ

Slide 3

Slide 3 text

Z Lab ͷ Solr Ϣʔεέʔε ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϩάͷอଘ LOG LOG LOG Monitoring
 Alerting Visualize LOG LOG LOG etc... Log store Banana Visualize

Slide 4

Slide 4 text

Solr ͷϞχλϦϯά

Slide 5

Slide 5 text

γεςϜμ΢ϯ !

Slide 6

Slide 6 text

Solr ϞχλϦϯάʹ࢖͑Δ API • Ping - https://lucene.apache.org/solr/guide/ping.html • MBean Request Handler - https://lucene.apache.org/solr/guide/mbean-request-handler.html • CoreAdmin API - https://lucene.apache.org/solr/guide/coreadmin-api.html • Collections API - https://lucene.apache.org/solr/guide/collections-api.html • Metrics API - https://lucene.apache.org/solr/guide/metrics-reporting.html • etc...

Slide 7

Slide 7 text

ෆศ !

Slide 8

Slide 8 text

ϝτϦΫεϞχλϦϯά

Slide 9

Slide 9 text

Solr ͷαϙʔτ͢ΔϝτϦΫεϞχλϦϯάιϑτ΢ΣΞ • jconsole - https://docs.oracle.com/javase/9/tools/jconsole.html • Graphite - http://graphiteapp.org • Ganglia - http://ganglia.info • etc... Reporter Λ࢖ͬͯ Solr ͷϝτϦΫεΛ GraphiteɺGanglia ͳͲ΁ૹ৴Մೳ • Reporters - https://lucene.apache.org/solr/guide/metrics-reporting.html#reporters

Slide 10

Slide 10 text

Z Lab ͰͷϝτϦΫεϞχλϦϯά Monitoring
 Alerting Visualize etc... node_exporter jmx_exporter mysqld_exporter ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϝτϦΫεΛ
 Prometheus ΁อଘ

Slide 11

Slide 11 text

ϩάϞχλϦϯά

Slide 12

Slide 12 text

Z Lab ͰͷϩάϞχλϦϯά ΞϓϦέʔγϣϯ΍ Containerɺ
 Virtual Machine ͷϩάΛ Solr ΁อଘ LOG LOG LOG LOG LOG LOG etc... Log store Banana Visualization ՄࢹԽͷΈ !

Slide 13

Slide 13 text

ϩάͷϞχλϦϯά͸ʁ LOG LOG LOG LOG LOG LOG Log store Log collector ʁ etc... Visualization ӡ༻͢Δ؂ࢹγεςϜ૿ ! ϊ΢ϋ΢ͷ͋Δ Solr ͰͳΜͱ͔ !

Slide 14

Slide 14 text

Z Lab ͰͷϩάϞχλϦϯά طଘͷ؂ࢹγεςϜͱ Solr Λ
 ͏·͘࿈ܞͤ͞Δ LOG LOG LOG Log / Metrics
 Monitoring,
 Alerting Metrics Visualization LOG LOG LOG Log store Banana Log Visualization etc...

Slide 15

Slide 15 text

CONTRIBUTED !

Slide 16

Slide 16 text

Prometheus ͱ Grafana Λ࢖ͬͨ Solr ͷϞχλϦϯά • Solr 7.3.0 ͔Β Solr ͷϝτϦΫεΛ Prometheus ʹެ։͢ΔͨΊͷ Exporter Λ௥Ճ - Solr ͷҰൠతͳϝτϦΫεΛσϑΥϧτͰऩू - Grafana ͷαϯϓϧμογϡϘʔυΛఏڙ • GitHub
 ɹɹhttps://github.com/apache/lucene-solr/tree/master/solr/contrib/prometheus-exporter • Solr Ref Guide
 ɹɹhttp://lucene.apache.org/solr/guide/monitoring-solr-with-prometheus-and-grafana.html


Slide 17

Slide 17 text

Prometheus • ࣌ܥྻσʔλϕʔεΛ಺แͨ͠Φʔϓϯιʔεͷ
 Pull ܕͷαʔϏε؂ࢹγεςϜ • ಠࣗͷΫΤϦݴޠΛ࣋ͭ • ݩ Google ͷΤϯδχΞ͕ Google ࣾ಺γεςϜ (Borg) ͷ
 ؂ࢹγεςϜ (Borgmon) ʹΠϯεύΠΞ͞Εͯ։ൃ • Exporter Λ࡞੒͢Δ͜ͱͰɺϝτϦΫεΛ Prometheus ͕ऩूͰ͖Δ https://prometheus.io

Slide 18

Slide 18 text

Grafana • Kibana ͔ΒϑΥʔΫͨ͠Φʔϓϯιʔεͷ
 σʔλՄࢹԽπʔϧ • ࣌ܥྻσʔλͷՄࢹԽʹϑΥʔΧε • ՄࢹԽ͚ͩͰͳ͘ɺΞϥʔςΟϯά΋ߦ͑ɺϓϥάΠϯʹΑΔ
 μογϡϘʔυͷ֦ு͕Մೳ • Prometheus ΍ InfluxDB ͳͲɺଟ͘ͷ࣌ܥྻσʔλϕʔεʹରԠ https://grafana.com

Slide 19

Slide 19 text

solr-exporter • Prometheus ͕ऩूͰ͖ΔϑΥʔϚοτͰ Solr ͷ
 ϝτϦΫεΛऩूɾެ։͢Δখ͞ͳ HTTP αʔό • Java Ͱ։ൃ (SolrJ Λར༻) • ελϯυΞϩʔϯͷ Solr ͚ͩͰͳ͘ SolrCloud ʹ΋ରԠ • Solr ͷݕࡧ݁Ռ΋ϝτϦΫεͱͯ͠ѻ͏͜ͱ͕Ͱ͖Δ

Slide 20

Slide 20 text

solr-exporter Query Pull Metrics Push alerts Notify Query Visualize Find targets Service Discovery Find nodes Register

Slide 21

Slide 21 text

solr-exporter ͷىಈ • Standalone mode (Linux) • SolrCloud mode (Linux) • ϝτϦΫεެ։ URL
 ɹɹhttp://localhost:9854/metrics $ cd contrib/prometheus-exporter $ ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -n 8 $ cd contrib/prometheus-exporter $ ./bin/solr-exporter -p 9854 -z localhost:2181/solr -f ./conf/solr-exporter-config.xml -n 16

Slide 22

Slide 22 text

solr-exporter ͷઃఆ • solr-exporter Λىಈ͢Δ͚ͩͰɺ͙͢ʹओཁͳϝτϦΫεͷऩू͕Մೳ • Solr ͷϨεϙϯεͷ JSON Λύʔε͢Δͷʹ jq γϯλοΫεΛ࠾༻ • ඞཁʹԠͯ͡ solr-exporter-config.xml Λฤू͠ɺϝτϦΫεͷऩूํ๏ͳͲมߋՄೳ - ऩू͢ΔϝτϦΫεͷ௥Ճɾ࡟আ - ϝτϦΫε໊ͷมߋ - etc

Slide 23

Slide 23 text

solr-exporter ͷઃఆ • solr-exporter ͷઃఆϑΝΠϧ͸γϯϓϧͰɺ࣍ͷΑ͏ͳλάͰߏ੒ • rules λά಺ʹ࣍ͷϦΫΤετΛؚΉ͜ͱ͕Մೳ Ping ϦΫΤετϋϯυϥͷϨεϙϯεͷεΫϨΠϐϯά Metrics API ͷϨεϙϯεͷεΫϨΠϐϯά Collections API ͷϨεϙϯεͷεΫϨΠϐϯά ݕࡧϦΫΤετͷϨεϙϯεͷεΫϨΠϐϯά

Slide 24

Slide 24 text

/admin/ping . as $object | $object | (if $object.status == "OK" then 1.0 else 0.0 end) as $value | { name : "solr_ping", type : "GAUGE", help : "See following URL: https://lucene.apache.org/solr/guide/ping.html", label_names : [], label_values : [], value : $value }

Slide 25

Slide 25 text

/admin/metrics all all .metrics["solr.jetty"] | to_entries | .[] |
 select(.key | startswith("org.eclipse.jetty.server.handler.DefaultHandler")) | 
 select(.key | endswith("xx-responses")) as $object | $object.key | split(".") | last | split("-") | first as $status | $object.value.count as $value | { name : "solr_metrics_jetty_response_total", type : "COUNTER", help : "See following URL: https://lucene.apache.org/solr/guide/metrics-reporting.html", label_names : ["status"], label_values : [$status], value : $value }

Slide 26

Slide 26 text

/admin/collections CLUSTERSTATUS .cluster.live_nodes | length as $value | { name : "solr_collections_live_nodes", type : "GAUGE", help : "See following URL: https://lucene.apache.org/solr/guide/collections-api.html", label_names : [], label_values : [], value : $value }

Slide 27

Slide 27 text

collection1 /select *:* { category: { type: terms, field: cat } } .facets.category.buckets[] as $object | $object.val as $term | $object.count as $value | { name : "solr_facets_category", type : "GAUGE", help : "Category facets", label_names : ["term"], label_values : [$term], value : $value }

Slide 28

Slide 28 text

ϝτϦΫεެ։ϑΥʔϚοτ • PrometheusͷϝτϦΫεެ։ϑΥʔϚοτ • # TYPE # HELP {=,=,...} name ϝτϦοΫ໊ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ type ϝτϦοΫܕ (COUNTERɺGAUGEɺSUMMARYɺHISTOGRAMɺUNTYPED)
 - ৄࡉ͸ https://prometheus.io/docs/concepts/metric_types/ help ϝτϦοΫͷઆ໌ label_names ϝτϦοΫͷϥϕϧ໊ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ label_values ϝτϦοΫͷϥϕϧ஋ - ৄࡉ͸ https://prometheus.io/docs/practices/naming/ value ϝτϦοΫ஋ (Double ܕ)

Slide 29

Slide 29 text

ϝτϦΫεެ։ϑΥʔϚοτ • JSON (ม׵લ) • Prometheus Exposition Format (ม׵ޙ) { "name": "solr_ping", "type": "GAUGE", "help": "See following URL: https://lucene.apache.org/solr/guide/ping.html", "label_names": ["base_url","core"], "label_values": ["http://localhost:8983/solr","collection1"], "value": 1.0 } # TYPE solr_ping gauge # HELP solr_ping See following URL: https://lucene.apache.org/solr/guide/ping.html solr_ping{base_url="http://localhost:8983/solr",core="collection1"} 1.0

Slide 30

Slide 30 text

ϝτϦΫεެ։ϑΥʔϚοτ $ curl -s http://localhost:9894/metrics # HELP solr_metrics_jetty_response_total See following URL: https://lucene.apache.org/solr/... # TYPE solr_metrics_jetty_response_total counter solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="1xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="2xx",} 516.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="3xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="4xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8983/solr",status="5xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="1xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="2xx",} 517.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="3xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="4xx",} 0.0 solr_metrics_jetty_response_total{base_url="http://localhost:8984/solr",status="5xx",} 0.0 ... [தུ] ...

Slide 31

Slide 31 text

Prometheus ͷઃఆ • prometheus.yml ͷ scrape_configs ʹ solr-exporter ͷઃఆΛ௥Ճ scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: 'solr' static_configs: - targets: ['localhost:9854']

Slide 32

Slide 32 text

• lucene-solr/solr/contrib/prometheus-exporter/conf ʹαϯϓϧͷ
 μογϡϘʔυΛಉࠝ - grafana-solr-dashboard.json • Grafana ʹμογϡϘʔυΛొ࿥͙ͯ͢͠ར༻։࢝Ͱ͖Δ • ඞཁʹԠͯ͡μογϡϘʔυΛฤूͯ͠ΧελϚΠζ Grafana μογϡϘʔυ

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

solr-exporter Λ࢖ͬͯͰ͖ΔΑ͏ʹͳΔ͜ͱ • Solr ͷ༷ʑͳϝτϦΫεΛऩूՄೳ • Solr ͷΠϯσοΫεͷ಺༰ΛϝτϦΫεͱͯ͠ऩूՄೳ - ΞΫηεϩά΍ΞϓϦέʔγϣϯϩάΛΠϯσοΫε͍ͯͨ͠Β... - ΞΫηε݅਺ͷՄࢹԽ΍ɺΞϓϦέʔγϣϯͷΤϥʔͷݕ஌ - EC αʔϏεͰ঎඼৘ใΛΠϯσοΫε͍ͯͨ͠Β... - ঎඼ΧςΰϦʔຖͷ঎඼఺਺ͷਪҠ΍ɺҧ๏঎඼ͷݕ஌ - SNS ͳͲͷϢʔβ౤ߘܕͷσʔλΛΠϯσοΫε͍ͯͨ͠Β... - ౤ߘ͞ΕΔίϯςϯπʹؚ·ΕΔΩʔϫʔυͷਪҠɺެংྑଏʹ൓͢Δίϯςϯπͷݕ஌ - etc..

Slide 35

Slide 35 text

·ͱΊ • Solr ͷՔಇঢ়گ͚ͩͰͳ͘ɺΠϯσοΫε΋ϞχλϦϯά - CPU ΍ετϨʔδͷར༻཰ͳͲͷϝτϦΫε͚ͩͰͳ͘ɺ
 อଘ͍ͯ͠Δσʔλ͕݈શͳঢ়ଶ͔΋νΣοΫͰ͖Δ • ͦΕͧΕͷ༻్ʹ߹Θͤͯιϑτ΢ΣΞΛબͿ - సஔΠϯσοΫεΛ࢖ͬͨσʔλͷݕࡧ͸ݕࡧΤϯδϯ - ࣌ܥྻσʔλͷूܭ΍ղੳͳͲ͸࣌ܥྻσʔλϕʔε

Slide 36

Slide 36 text

Monitoring Solr