What's happening in production?

Lightning talk delivered at Agilasverige (Agile Sweden) on metrics and that you as a developer should be able to know what's happening in your production environment in real time

Approx 10 minutes.

Transcript

1. Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ marten.gustafson@gmail.com What’s happening in production Monday, May 23, 2011
2. Hi! Monday, May 23, 2011 My name is Mårten, I work as a software developer But I usually also have one foot - sometimes a whole leg - in our infrastructure
3. Monday, May 23, 2011 I work here. We provide a web site, that’s our offering. Our core.
4. Monday, May 23, 2011 Our site is, amongst other things, built from a slew of Java services We have multiple services on each server
5. Monday, May 23, 2011 We like to be availbale, thus we have multiple servers Same service, on multiple servers Multiple services on same server
6. which services are utilized? Monday, May 23, 2011
7. most? Monday, May 23, 2011
8. least? Monday, May 23, 2011
9. most memory? Monday, May 23, 2011
10. highest latency? Monday, May 23, 2011
11. does it work? Monday, May 23, 2011
12. http://www.picpop.com/ Monday, May 23, 2011
13. what’s happening in production? Monday, May 23, 2011
14. right. this. second. Monday, May 23, 2011
15. measure! Monday, May 23, 2011
16. everything! Monday, May 23, 2011
17. in real time! Monday, May 23, 2011
18. it’s cheap... Monday, May 23, 2011
19. ...and simple! Monday, May 23, 2011
20. meter = metrics.meter("results", "records", TimeUnit.SECONDS) ...calculate result... meter.mark(results.size()) Monday, May 23, 2011
21. counter = Metrics.newCounter( getClass(), "requests") ...receive request... counter.inc() Monday, May 23, 2011
22. counters Monday, May 23, 2011
23. timers Monday, May 23, 2011
24. averages Monday, May 23, 2011
25. etc... Monday, May 23, 2011
26. how? Monday, May 23, 2011
27. develop something, or... Monday, May 23, 2011
28. https://github.com /codahale/metrics ...for java & scala use: Monday, May 23, 2011 ...or at least look at this to get inspired
29. JMX Monday, May 23, 2011
30. Servlet = HTTP + JSON Monday, May 23, 2011
31. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 JSON, not always funny to look at The really interesting things are small tidbits of information “hidden” in this blob of data
32. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit s":"Ok"}]},{"name":"autocomplete","application_status":"ok","counter_tot Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 Is it ok?
33. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte 079},{"name":"memory.used","counter": 1329},{"name":"cpu","counter":8} atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How much memory do we consume?
34. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok tion_status":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageServ "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How many requests have we served?
35. data is good Monday, May 23, 2011
36. visualized data is better Monday, May 23, 2011
37. visualize Monday, May 23, 2011
38. Monday, May 23, 2011 Visualization makes it easy to distinguish worse from better
39. Monday, May 23, 2011 Visualization makes it easy to distinguish worse from better
40. Monday, May 23, 2011 Visualization makes it easy to distinguish worse from better
41. visualization = insight Monday, May 23, 2011
42. insight = knowledge Monday, May 23, 2011
43. knowledge = safety Monday, May 23, 2011
44. does it work? Monday, May 23, 2011
45. No. Monday, May 23, 2011
46. does it work? Monday, May 23, 2011
47. Yes. Monday, May 23, 2011
48. which services are utilized? Monday, May 23, 2011
49. Monday, May 23, 2011 Obviously the top one
50. available memory? Monday, May 23, 2011
51. Monday, May 23, 2011
52. 1 screen + 1 computer = visualization! Monday, May 23, 2011
53. why not more? Monday, May 23, 2011
54. think build server... Monday, May 23, 2011 Your project / team probably have a build server monitor, which is a good thing. But how much “dashboard estate” do you spend on production - where your code actually (hopefully) delivers value?
55. ...only bigger... Monday, May 23, 2011
56. ...and more Monday, May 23, 2011
57. what’s happening in production? Monday, May 23, 2011
58. Monday, May 23, 2011 The dashboard where I work - all servers, most services
59. visualizing data in real time gives insight and safety Monday, May 23, 2011
60. Monday, May 23, 2011 Thanks
61. Monday, May 23, 2011 Thanks