Warsaw JUG: Listen to the sounds of your application

Transcript

1. Listen to the sounds of your application
2. !2 Krzysztof Ciesielski @kpciesielski softwaremill.com lbn.sc
3. !3 Maciej Biłas @maciejb softwaremill.com jbison.com
4. Presented at J-Day Lbn.sc Poznań JUG Szczecin JUG
5. Agenda • Monitoring, huh? • Introducing Graphite • Log analysis – the whys • Logstash architecture & use cases • Exploring logs with Kibana
6. Monitoring, huh?
7. source: codeascraft.com
8. Types of measurements Network Machine Application
9. source: codeascraft.com
10. measurement > prediction
12. measurement >> prediction
13. Our stack Server App Server App … Graphite Logstash Yammer Metrics
15. http://graphite/render? target=server.web1.load&height=800&width=600
16. echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003
17. Let’s see…
18. Graphite
19. successful.login.attempt 1 1384471287
 successful.login.attempt 1 1384471297 successful.login.attempt = 1
20. Yammer metrics private final Meter successfulLogins =
 metrics.meter(name(LoginHandler.class, "successful")); ! public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
21. Metrics Types: • Gauges • Counters • Meters • Histograms • Timers • Health Checks Reporters: • STDOUT • CSV • SLF4J • JMX • Graphite • Ganglia
22. Counters, meters… vs. Gauges
23. Aggregation app00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime …
24. source: codeascraft.com
25. You can also check out • collectd https://collectd.org • StatsD https://github.com/etsy/statsd/ • Riemman http://riemann.io/ • Twitter’s Ostrich https://github.com/twitter/ostrich • Ganglia http://ganglia.sourceforge.net/ • Dashboards:
 http://shopify.github.io/dashing/
 https://github.com/obfuscurity/descartes
 https://github.com/obfuscurity/dusk
26. Log aggregation?
27. Log aggregation Logs are data with plenty of value
28. Log aggregation Log source Log source Log source Central storage Query engine Web view
29. Logstash • Open source • Written in JRuby • Gathers logs from various inputs • Parses and extracts metadata • Writes to various outputs
30. Architecture
31. Inputs collectd drupal_dblog elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq redis relp s3 snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq
32. Grok time 2013-11-10 19:41:25.321 thread main loglevel INFO source o.a.camel.impl.DefaultCamelContext message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds Thin layer on top of a regular expression
33. Grok grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} 
 \[%{DATA:thread}\] %{LOGLEVEL:loglevel}
 %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] } 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
34. Outputs boundary circonus cloudwatch csv datadog datadog_metric s elasticsearch elasticsearch_ht tp elasticsearch_ri ver email exec file ganglia gelf gemfire google_bigquer y google_cloud_st orage graphite graphtastic hipchat http irc jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak riemann s3 sns solr_http sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq
35. Kibana
36. Checking app activity
37. Spotting anomalies
38. Regular events
39. How often is a feature used?
40. Plotting multiple event types
41. Other tools • Splunk • Graylog2 • Fluentd • log.io
42. References • Graphite
 http://graphite.readthedocs.org/en/latest/ • Yammer Metrics
 http://metrics.codahale.com/ • Logstash
 http://logstash.net/ • Surfing the event stream by Sam Newman at Geecon
 http://www.slideshare.net/spnewman/surfing-the-event-stream • Lessons from Building and Scaling LinkedIn by Jay Kreps
 http://www.infoq.com/presentations/linkedin-architecture-stack • Code as Craft
 http://codeascraft.com/ • Gibberish logger + Logstash showcase (Vagrant box)
 https://github.com/kciesielski/giblogger
44. Thank you! Questions?