Poznań JUG: Listen to the sounds of your application

Transcript

1. Listen to the sounds of your application
2. !2 Maciej Biłas @maciejb softwaremill.com jbison.com
3. Originally authored by: Maciej Biłas Krzysztof Ciesielski Presented at J-Day Lbn.sc
4. Agenda • Monitoring, huh? • Introducing Graphite • Log analysis – the whys • Logstash architecture & use cases • Exploring logs with Kibana
5. Monitoring, huh?
6. source: codeascraft.com
7. Types of measurements Network Machine Application
8. source: codeascraft.com
9. measurement > prediction
11. measurement >> prediction
12. Our stack Server App Server App … Graphite Logstash Yammer Metrics
14. http://graphite/render? target=server.web1.load&height=800&width=600
15. Graphite
16. echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003
17. successful.login.attempt 1 1384471287
 successful.login.attempt 1 1384471297 successful.login.attempt = 1
18. Yammer metrics private final Meter successfulLogins =
 metrics.meter(name(LoginHandler.class, "successful")); ! public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
19. Metrics Types: • Gauges • Counters • Meters • Histograms • Timers • Health Checks Reporters: • STDOUT • CSV • SLF4J • JMX • Graphite • Ganglia
20. Counters, meters… vs. Gauges
21. Aggregation app00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime …
22. source: codeascraft.com
23. You can also check out • collectd https://collectd.org • StatsD https://github.com/etsy/statsd/ • Riemman http://riemann.io/ • Twitter’s Ostrich https://github.com/twitter/ostrich • Ganglia http://ganglia.sourceforge.net/ • Dashboards:
 http://shopify.github.io/dashing/
 https://github.com/obfuscurity/descartes
 https://github.com/obfuscurity/dusk
24. Log aggregation?
25. Event Sourcing Tracing and storing all the events Current app state: replaying the stream Great for data mining and analysis
26. Log aggregation • Less invasive way to build an “event stream” • Logs are data with plenty of value
27. Log aggregation Log source Log source Log source Central storage Query engine Web view
28. Logstash • Open source • Written in JRuby • Gathers logs from various inputs • Parses and extracts metadata • Writes to various outputs
29. Architecture
30. Inputs collectd drupal_dblog elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc log4j lumberjack pipe rabbitmq redis relp s3 snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq
31. Grok time 2013-11-10 19:41:25.321 thread main loglevel INFO source o.a.camel.impl.DefaultCamelContext message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds Thin layer on top of a regular expression
32. Grok grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} 
 \[%{DATA:thread}\] %{LOGLEVEL:loglevel}
 %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] } 2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
33. Outputs boundary circonus cloudwatch csv datadog datadog_metric s elasticsearch elasticsearch_ht tp elasticsearch_ri ver email exec file ganglia gelf gemfire google_bigquer y google_cloud_st orage graphite graphtastic hipchat http irc jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis riak riemann s3 sns solr_http sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq
34. Kibana
35. Checking app activity
36. Spotting anomalies
37. Regular events
38. How often is a feature used?
39. Plotting multiple event types
40. Other tools • Splunk • Graylog2 • Fluentd • log.io
41. References • Graphite
 http://graphite.readthedocs.org/en/latest/ • Yammer Metrics
 http://metrics.codahale.com/ • Logstash
 http://logstash.net/ • Surfing the event stream by Sam Newman at Geecon
 http://www.slideshare.net/spnewman/surfing-the-event-stream • Lessons from Building and Scaling LinkedIn by Jay Kreps
 http://www.infoq.com/presentations/linkedin-architecture-stack • Code as Craft
 http://codeascraft.com/
43. Thank you! Questions?