What's happening in production?

What's happening in production?

Lightning talk delivered at Agilasverige (Agile Sweden) on metrics and that you as a developer should be able to know what's happening in your production environment in real time

Approx 10 minutes.

A204e1fe2002bc6d087391759c3dfab0?s=128

Mårten Gustafson

May 23, 2011
Tweet

Transcript

  1. Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ marten.gustafson@gmail.com What’s happening in production Monday,

    May 23, 2011
  2. Hi! Monday, May 23, 2011 My name is Mårten, I

    work as a software developer But I usually also have one foot - sometimes a whole leg - in our infrastructure
  3. Monday, May 23, 2011 I work here. We provide a

    web site, that’s our offering. Our core.
  4. Monday, May 23, 2011 Our site is, amongst other things,

    built from a slew of Java services We have multiple services on each server
  5. Monday, May 23, 2011 We like to be availbale, thus

    we have multiple servers Same service, on multiple servers Multiple services on same server
  6. which services are utilized? Monday, May 23, 2011

  7. most? Monday, May 23, 2011

  8. least? Monday, May 23, 2011

  9. most memory? Monday, May 23, 2011

  10. highest latency? Monday, May 23, 2011

  11. does it work? Monday, May 23, 2011

  12. http://www.picpop.com/ Monday, May 23, 2011

  13. what’s happening in production? Monday, May 23, 2011

  14. right. this. second. Monday, May 23, 2011

  15. measure! Monday, May 23, 2011

  16. everything! Monday, May 23, 2011

  17. in real time! Monday, May 23, 2011

  18. it’s cheap... Monday, May 23, 2011

  19. ...and simple! Monday, May 23, 2011

  20. meter = metrics.meter("results", "records", TimeUnit.SECONDS) ...calculate result... meter.mark(results.size()) Monday, May

    23, 2011
  21. counter = Metrics.newCounter( getClass(), "requests") ...receive request... counter.inc() Monday, May

    23, 2011
  22. counters Monday, May 23, 2011

  23. timers Monday, May 23, 2011

  24. averages Monday, May 23, 2011

  25. etc... Monday, May 23, 2011

  26. how? Monday, May 23, 2011

  27. develop something, or... Monday, May 23, 2011

  28. https://github.com /codahale/metrics ...for java & scala use: Monday, May 23,

    2011 ...or at least look at this to get inspired
  29. JMX Monday, May 23, 2011

  30. Servlet = HTTP + JSON Monday, May 23, 2011

  31. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 JSON, not always funny to look at The really interesting things are small tidbits of information “hidden” in this blob of data
  32. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit s":"Ok"}]},{"name":"autocomplete","application_status":"ok","counter_tot Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 Is it ok?
  33. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte 079},{"name":"memory.used","counter": 1329},{"name":"cpu","counter":8} atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How much memory do we consume?
  34. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok tion_status":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageServ "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c

    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How many requests have we served?
  35. data is good Monday, May 23, 2011

  36. visualized data is better Monday, May 23, 2011

  37. visualize Monday, May 23, 2011

  38. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better
  39. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better
  40. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better
  41. visualization = insight Monday, May 23, 2011

  42. insight = knowledge Monday, May 23, 2011

  43. knowledge = safety Monday, May 23, 2011

  44. does it work? Monday, May 23, 2011

  45. No. Monday, May 23, 2011

  46. does it work? Monday, May 23, 2011

  47. Yes. Monday, May 23, 2011

  48. which services are utilized? Monday, May 23, 2011

  49. Monday, May 23, 2011 Obviously the top one

  50. available memory? Monday, May 23, 2011

  51. Monday, May 23, 2011

  52. 1 screen + 1 computer = visualization! Monday, May 23,

    2011
  53. why not more? Monday, May 23, 2011

  54. think build server... Monday, May 23, 2011 Your project /

    team probably have a build server monitor, which is a good thing. But how much “dashboard estate” do you spend on production - where your code actually (hopefully) delivers value?
  55. ...only bigger... Monday, May 23, 2011

  56. ...and more Monday, May 23, 2011

  57. what’s happening in production? Monday, May 23, 2011

  58. Monday, May 23, 2011 The dashboard where I work -

    all servers, most services
  59. visualizing data in real time gives insight and safety Monday,

    May 23, 2011
  60. Monday, May 23, 2011 Thanks

  61. Monday, May 23, 2011 Thanks