Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)

Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to monitoring and graphing systems.

Throw in the sin bin the poor solutions based on log files and weakly secured web interfaces! A new generation of Open Source tooling makes it easy to graph java application metrics and integrate them to traditional monitoring systems like Nagios.

Following the logic of DevOps, we will look together how best to integrate the monitoring dimension in a project: from design to development, to QA and finally to production on both traditional deployment and in the Cloud.

Come and discover how the JmxTrans-Graphite ticket can make your life easier.

Transcript

1. Cyrille Le Clerc Open Source Monitoring for Java with Graphite Wednesday, May 15, 13
2. Speaker Cyrille Le Clerc @cyrilleleclerc Open Source CTO DevOps, Infra as Code Solution Architect Cloud Wednesday, May 15, 13
3. You ? DEV ? QA ? OPS ? Monitoring Experience ? Wednesday, May 15, 13
4. Agenda Wednesday, May 15, 13
5. Agenda • The demo application • Monitoring performance: Why ? What ? • Java webapp monitoring with JMX • Monitoring with Graphite • Conclusion Wednesday, May 15, 13
6. The Application Wednesday, May 15, 13
7. Your mission, should you decide to accept it, ... Setup technical and business monitoring e-commerce SOS Cocktails http://www.flickr.com/photos/23791504@N08/2607814349 Wednesday, May 15, 13
8. SOS Cocktail Wednesday, May 15, 13
9. Monitoring application performance Why ? What ? Wednesday, May 15, 13
10. The Indicators to Monitor Wednesday, May 15, 13
11. The Indicators to Monitor • Operating System ➔ Sysload • JVM ➔ GC duration • Tomcat ➔ activeSessions (active visitors) • Application ➔ sales revenue & items Wednesday, May 15, 13
12. Open Source Monitoring for Java Wednesday, May 15, 13
13. Open Source Monitoring for Java e-commerce web site Graphite Seyren Dashboard - wiki Nagios Email Wednesday, May 15, 13
14. JMX & Metrics Wednesday, May 15, 13
15. Why JMX ? • JMX vs. web pages vs. logs vs. ... • Simple and secured • Displayable with web pages and logs • Alternatives to JMX ? Wednesday, May 15, 13
16. JMX vs. logs vs. web pages vs. ... • Monitoring logs • Parsing GB of text files ? • Per invocation details vs. average per minute • Web pages • History ? • Clustered applications ? • Security ? Wednesday, May 15, 13
17. JMX with Spring Framework <beans  ...>      ...      <context:mbean-­‐export/>      ... </beans> @ManagedResource("cocktail:name=ShoppingCartController,type=...") class  ShoppingCartController  {          final  AtomicInteger  revenueInCentsCounter  =  new  AtomicInteger();  void  purchase(...){      ...      revenueInCentsCounter.addAndGet(priceInCents);  }  @ManagedAttribute  public  int  getRevenueInCentsCounter()  {    return  revenueInCentsCounter.get();  }         Wednesday, May 15, 13
18. JMX with VisualVM Wednesday, May 15, 13
19. JMX with JSP $  curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-­‐sales-­‐metrics.txt.jsp Epoch   SalesRevenueInCentsCounter   SalesItemsCounter   SalesOrdersCounter 1363217376   4500   7   2 curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/... {    "Epoch":1363217653,    "SalesRevenueInCentsCounter":4050,    "SalesItemsCounter":7,  ... } http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp Human readable & script friendly Watch out for security ! Wednesday, May 15, 13
20. JMX with JSP $  curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-­‐sales-­‐metrics.txt.jsp Epoch   SalesRevenueInCentsCounter   SalesItemsCounter   SalesOrdersCounter 1363217376   4500   7   2 curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/... {    "Epoch":1363217653,    "SalesRevenueInCentsCounter":4050,    "SalesItemsCounter":7,  ... } http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp Human readable & script friendly Watch out for security ! /5ADrUWr5/jmx-sales-metrics.jsp Wednesday, May 15, 13
21. JMX with JSP Wednesday, May 15, 13
22. Alternatives to JMX Wednesday, May 15, 13
23. Alternatives to JMX class  ShoppingCartController  {  @Monitor(name="revenueInCents",  type=COUNTER)  final  AtomicInteger  revenueInCentsCounter  =  new  AtomicInteger();          void  purchase(...){      ...      revenueInCentsCounter.addAndGet(priceInCents);  } Servo Wednesday, May 15, 13
24. Alternatives to JMX class  ShoppingCartController  {  @Monitor(name="revenueInCents",  type=COUNTER)  final  AtomicInteger  revenueInCentsCounter  =  new  AtomicInteger();          void  purchase(...){      ...      revenueInCentsCounter.addAndGet(priceInCents);  } Servo class  ShoppingCartController  {          final  Counter  revenueInCentsCounter  =  Metrics.newCounter(                            ShoppingCartController,  "revenue-­‐in-­‐cents");          void  purchase(...){      ...      revenueInCentsCounter.inc(priceInCents);  } Metrics Wednesday, May 15, 13
25. JMX • OS: Sysload • JVM: Garbage Collector • Tomcat: activeSessions • Application: Web Site Revenue Wednesday, May 15, 13
26. jmxtrans Wednesday, May 15, 13
27. jmxtrans - standalone ecommerce Tomcat Graphite accounting Tomcat logistics Tomcat ... jmxtrans - standalone Wednesday, May 15, 13
28. embedded-jmxtrans logistics Tomcat Graphite ecommerce Tomcat accounting Tomcat Librato ... batch embedded-jmxtrans Wednesday, May 15, 13
29. jmxtrans-agent jmxtrans-agent java -javaagent=jmxtrans-agent.jar=... Graphite ... batch ecommerce Tomcat JVM ecommerce Tomcat JVM ecommerce Tomcat JVM logistics Tomcat JVM ecommerce Tomcat JVM accounting Tomcat JVM JVM JVM Librato Wednesday, May 15, 13
30. standalone vs. embedded vs. agent standalone embedded agent Packaging Model Manually connected apps Pull Standalone apps Push Standalone middleware Push Cursor OPS -‖----------- DEV OPS ----------‖-- DEV OPS ---‖--------- DEV Use case Net unfriendly monitoring, OPS cloud, batch DEV/QA cloud, batch OPS Wednesday, May 15, 13
31. embedded-jmxtrans configuration Wednesday, May 15, 13
32. embedded-jmxtrans configuration <dependency>    <groupId>org.jmxtrans.embedded</groupId>    <artifactId>embedded-­‐jmxtrans</artifactId>    <version>1.0.7</version> </dependency> Wednesday, May 15, 13
33. embedded-jmxtrans configuration <dependency>    <groupId>org.jmxtrans.embedded</groupId>    <artifactId>embedded-­‐jmxtrans</artifactId>    <version>1.0.7</version> </dependency> <beans  ...    xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"    xsi:schemaLocation="...        http://www.jmxtrans.org/schema/embedded  ...”>    <jmxtrans:jmxtrans>        <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>        ...    </jmxtrans:jmxtrans> </beans> Wednesday, May 15, 13
34. embedded-jmxtrans configuration <dependency>    <groupId>org.jmxtrans.embedded</groupId>    <artifactId>embedded-­‐jmxtrans</artifactId>    <version>1.0.7</version> </dependency> <beans  ...    xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"    xsi:schemaLocation="...        http://www.jmxtrans.org/schema/embedded  ...”>    <jmxtrans:jmxtrans>        <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>        ...    </jmxtrans:jmxtrans> </beans> {    "queries":  [            {            "objectName":  "cocktail:name=ShoppingCartController",            "resultAlias":  "sales",            "attributes":  ["SalesRevenueInCentsCounter"]        },    ],    "outputWriters":  [        {            "@class":  "org.jmxtrans.embedded.output.GraphiteWriter",            "settings":  {                "host":  "${graphite.host:localhost}",                "port":  "${graphite.port:2003},"                "enabled":  "${graphite.enabled:true}"            }        }    ] } Wednesday, May 15, 13
35. Graphite Wednesday, May 15, 13
36. Graphite • Open Source • Time Series DataBase and graphing tool • Used by tech giants • Similar Wednesday, May 15, 13
37. Graphite • Open Source • Time Series DataBase and graphing tool • Used by tech giants • Similar Simplicity and Self Service Wednesday, May 15, 13
38. Metrics injection / Write Access • Automatic metrics creation • Socket protocols: “plain text” & Python Pickle serialization • Precision and storage duration by configuration echo  "geecon.happyCounter  1.2  1364338989"  |  nc  localhost  2003 [sales_1min_for_15days_5min_for_1year] pattern  =  ^sales\. retentions  =  60s:30d,300s:365d 1.7MB Wednesday, May 15, 13
39. • Graph Composer & URL API • Everybody can create graphs • Wide range of functions • sum, scale, derivative, timeshift, deviation, filter, ... • Various exports • png, svg, csv, json, raw text Metrics rendering / Read Access Wednesday, May 15, 13
40. The Art of Graphing Wednesday, May 15, 13
41. Monitoring on the Cloud • http://hostedgraphite.com/ • Graphite as a Service • Zero setup • Addons: Tasseo • No-OPS 1 2 3 email: admin@jmxtrans.org password: GEECON Wednesday, May 15, 13
42. Metrics Types • Examples: activeRequests, dataSource.activeConnection, ... • Information available without transformation Gauge Ever Increasing Counter • Examples: requestsCount, revenue, ... • “per minute” aggregation required Wednesday, May 15, 13
43. Graphs and Maths Ever Increasing Counter Revenue per Second ? Wednesday, May 15, 13
44. Graphs and Maths Ever Increasing Counter Revenue per Second ? derivative() Wednesday, May 15, 13
45. Graphs and Maths ? Revenue per Hour Per Second → Per Hour Wednesday, May 15, 13
46. Graphs and Maths ? Revenue per Hour Per Second → Per Hour summarize(..., “1h”) Wednesday, May 15, 13
47. Graphs and Maths 2 servers ? Total Revenue per Hour Wednesday, May 15, 13
48. Graphs and Maths 2 servers ? Total Revenue per Hour sumSeries() Wednesday, May 15, 13
49. Graphs and Maths server restart ? Ignore reset to zero Wednesday, May 15, 13
50. Graphs and Maths server restart ? Ignore reset to zero nonNegativeDerivative() Wednesday, May 15, 13
51. Graphs and Maths Compare to last week ? Wednesday, May 15, 13
52. Graphs and Maths Compare to last week ? timeShift(..., “7d”) Wednesday, May 15, 13
53. JMX • OS: Sysload • JVM: Garbage Collector • Tomcat: activeSessions • Application: Web Site Revenue Wednesday, May 15, 13
54. Graphite URL API http://localhost:8081/render/?      from=-­‐7days&      title=Revenue%20per%20Hour&      vtitle=Dollars&      lineWidth=3&      xFormat=%25a%20%25H%25p&      target=        alias(          scale(            summarize(              sumSeries(                nonNegativeDerivative(                  edu.servers.*.sales.revenueInCentsCounter)...)&        ... Wednesday, May 15, 13
55. Sharing metrics and graphs Wednesday, May 15, 13
56. Dashboards and Wikis dashboard.prod.md #  Website  Traffic <table> <tr>    <td>        <img  src="http://graphite.prod.mycompany/graphite/render/? width=400&height=300&title=Revenue&xFormat=%25a%20%25d %25p&vtitle=USD&lineWidth=3&from=-­‐5days&target=alias(scale(summarize(sumS eries(nonNegativeDerivative(edu.servers.*.sales.revenueInCentsCounter)) %2C%221h%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)...">        </td>        <td>                <img  src="http://graphite.prod.mycompany/graphite/render/?...">        </td> <tr> </table> Wiki https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Dashboard-PROD Wednesday, May 15, 13
57. Dashboards and Email Send dashboards by email Wednesday, May 15, 13
58. Alerting with Metrics Wednesday, May 15, 13
59. Graphite Integration with Alerting • Based on Graphite URL API • RawText or JSON format • Pattern /render?from=-­‐11minutes&until=-­‐1minutes&format=raw&target=**** Wednesday, May 15, 13
60. Graphite Integration with Alerting • Based on Graphite URL API • RawText or JSON format • Pattern /render?from=-­‐11minutes&until=-­‐1minutes&format=raw&target=**** $  curl  "http://graphite.example.com/render?from=-­‐11minutes&until=-­‐1minutes&format=raw&      target=keepLastValue(servers.cloudbees.jvm.os.SystemLoadAverage)"   my-­‐metric,1363225680,1363226340,60|0.03,0.01,0.1,4.0,4.0,0.9,0.7,0.8,0.4,0.5,0.5 Example Wednesday, May 15, 13
61. Graphite Alerting with Seyren • Alerting Dashboard for Graphite • Open Source • Java .war + MongoDB • Alerts: email, PagerDuty, ... http://seyren.jmxtrans.cloudbees.net/ Wednesday, May 15, 13
62. • Monitoring Infrastructure • Open Source • De facto standard • check_graphite plugin • Pierre-Yves Ritschard • Jason Dixon / obfuscurity Graphite Alerting with Nagios Wednesday, May 15, 13
63. Conclusion Wednesday, May 15, 13
64. Conclusion • Self-service monitoring changes everything • Technical and business monitoring • Monitoring integrated in the Continuous Delivery pipeline • Open Source solutions are available Wednesday, May 15, 13
65. Hosted Graphite Promo • http://hostedgraphite.com • 60 days trial with “GEECON” promo code help: cleclerc@cloudbees.com (1) Signup (2) Options / Enter Promo Code (3) Promo Code “GEECON” Wednesday, May 15, 13
66. http://demo-cocktail.jmxtrans.cloudbees.net/ http://jmxtrans.org/ Wednesday, May 15, 13