HOW DO YOU KNOW iT'S WORKiNG

Monitoring NGiNX.Presented during https://ngenious.website/

Transcript

1. HOW DO YOU KNOW iT’S WORKiNG MONiTORiNG
2. MiKE FiEDLER Wrangler of the Unusual Twitter: @mikefiedler Github: @miketheman OpsSchool.org Chef Community Roller Derby Referee © Alex Erde
3. WHO CARES?
4. DOES iT EVEN?
5. DOES iT EVEN? YES iT DOES!
6. ACTiViTiES - ngx_http_stub_status_module $ nginx -V 2>&1 | grep -o with-http_stub_status_module with-http_stub_status_module location /basic_status { stub_status; } $ curl http://localhost/basic_status Active connections: 1 server accepts handled requests 8 8 18 Reading: 0 Writing: 1 Waiting: 0
7. location /basic_status { allow 127.0.0.1; deny all; stub_status on; access_log off; } Active connections: 45 server accepts handled requests 1023122 1023122 4270474 Reading: 0 Writing: 1 Waiting: 44 ACTiViTiES - SECURE-ER
8. MOAR??! - ngx_http_status_module
10. FREE AS A BiRD: ❖ awstats ❖ goaccess ❖ syslog/fluentd/logstash ❖ elasticsearch (ELK/EFK) LOGS - SO MUCH DATA DOLLA DOLLA BiLLS: ❖ Logentries ❖ Loggly ❖ Logz.io ❖ Splunk
11. • Emit log line • Log line is written to file descriptor • Line is then: ◦ Read ◦ Shipped ◦ Tokenized ◦ Parsed ◦ Stored ◦ Eventually trashed • Oh, wait, the log line format changed?? FFS!! LOG PARSING iS HARD, MiKE!
12. STATSD (spec) • Push-based • UDP, connectionless • Lightweight datagram <metric name>:<value>|c[|@<sample rate>] nginx.requests:1|c EMiTTERS TO THE RESCUE!
13. GET ME AN EMiTTER! Lua-based: • https://github.com/musicglue/ngx_lua_statsd • https://github.com/simplifi/ngx_lua_datadog • https://github.com/lonelyplanet/openresty-statsd • https://github.com/knyar/nginx-lua-prometheus Native C: https://github.com/zebrafishlabs/nginx-statsd
14. DiD SOMEONE SAY C? HMMM…. • Get nginx source code • Get emitter module • Patch a couple of files • Build a package • Config & run server { ... statsd_server localhost; statsd_count "nginx.requests" 1; statsd_count "nginx.responses.$status" 1 "$status"; statsd_count "nginx.request_length" "$request_length"; statsd_count "nginx.bytes_sent" "$bytes_sent"; location / { ...
15. PROFiT!
16. INSTRUMENT all resources • CPU • Network I/O • Disk I/O & space (!!!) SUMMARY - MONiTOR FiRST ALERT on activities • Page loads • Dropped connections • Requests per second • Request timings • Error rates ESTABLiSH BASELiNES PRiOR TO CHANGE!
17. THANKS! https://github.com/miketheman/ngenious-statsd QUESTiONS?