What's happening in production?

A204e1fe2002bc6d087391759c3dfab0?s=47 Mårten Gustafson
May 23, 2011
Programming
3
230

What's happening in production?

Lightning talk delivered at Agilasverige (Agile Sweden) on metrics and that you as a developer should be able to know what's happening in your production environment in real time

Approx 10 minutes.

A204e1fe2002bc6d087391759c3dfab0?s=128

Mårten Gustafson

May 23, 2011
Tweet

More Decks by Mårten Gustafson

See All by Mårten Gustafson
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
3
240
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
0
980
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
0
120
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
0
120
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
2
520
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
4
330
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
9
4.5k
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
2
93
A204e1fe2002bc6d087391759c3dfab0?s=48 chids
3
130

Other Decks in Programming

See All in Programming
80a3a3857a55f154d23acb705eff72cc?s=48 star_zero
3
130
E24ebbedcae292dc8e4b56186e082220?s=48 iganari
1
290
38a509643276213e62730e92ef220405?s=48 masatakamiki
0
190
A3fabd96e2ca4bee5b0333ff2a640f01?s=48 translucens
2
110
933291444e456bfb511a66a2fa9c6929?s=48 j5ik2o
7
1.1k
D9e4d5a4b36f83fbde1076815df1362f?s=48 bmack
0
180
C9ab1175a6981a2f67ce8d08aa17c15a?s=48 mpilquist
0
120
41a66760b3483d518d9e48a7da37f91b?s=48 shibadog1121
9
1.6k
37618b4d60cf3d1f43f7196253264edc?s=48 numeroanddev
0
160
0ca0c0ec6efe3a7f5220332a910e6da0?s=48 shinyoshiaki
1
510
F9e11bea0c22333596791dd0696f5d4f?s=48 daiksy
0
1.2k
817e89f88197980860f1854b74fbee25?s=48 lcdsmao
2
340

Featured

See All Featured
9952dfcd5d338f8a8e7175c8a8f65fb5?s=48 wjessup
333
15k
9128d500301ae51524e887bb680f471d?s=48 caitiem20
300
16k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
163
18k
5ce91fa49a613cbc3e20d5f96856473f?s=48 edds
55
9.2k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
281
46k
64827b31aef81498662e8af4bd6d5157?s=48 jonrohan
1020
350k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
6
370
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
144
19k
E14f55d3f939977cecbf51b64ff6f861?s=48 keithpitt
398
20k
7559f6cff1f5efc2d210965febd4d71c?s=48 bermonpainter
340
25k
E6c6e133e74c3b83f04d2861deaa1c20?s=48 searls
202
35k
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
123
21k

Transcript

  1. Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ marten.gustafson@gmail.com What’s happening in production Monday,

    May 23, 2011

  2. Hi! Monday, May 23, 2011 My name is Mårten, I

    work as a software developer But I usually also have one foot - sometimes a whole leg - in our infrastructure

  3. Monday, May 23, 2011 I work here. We provide a

    web site, that’s our offering. Our core.

  4. Monday, May 23, 2011 Our site is, amongst other things,

    built from a slew of Java services We have multiple services on each server

  5. Monday, May 23, 2011 We like to be availbale, thus

    we have multiple servers Same service, on multiple servers Multiple services on same server

  6. which services are utilized? Monday, May 23, 2011

  7. most? Monday, May 23, 2011

  8. least? Monday, May 23, 2011

  9. most memory? Monday, May 23, 2011

  10. highest latency? Monday, May 23, 2011

  11. does it work? Monday, May 23, 2011

  12. http://www.picpop.com/ Monday, May 23, 2011

  13. what’s happening in production? Monday, May 23, 2011

  14. right. this. second. Monday, May 23, 2011

  15. measure! Monday, May 23, 2011

  16. everything! Monday, May 23, 2011

  17. in real time! Monday, May 23, 2011

  18. it’s cheap... Monday, May 23, 2011

  19. ...and simple! Monday, May 23, 2011

  20. meter = metrics.meter("results", "records", TimeUnit.SECONDS) ...calculate result... meter.mark(results.size()) Monday, May

    23, 2011

  21. counter = Metrics.newCounter( getClass(), "requests") ...receive request... counter.inc() Monday, May

    23, 2011

  22. counters Monday, May 23, 2011

  23. timers Monday, May 23, 2011

  24. averages Monday, May 23, 2011

  25. etc... Monday, May 23, 2011

  26. how? Monday, May 23, 2011

  27. develop something, or... Monday, May 23, 2011

  28. https://github.com /codahale/metrics ...for java & scala use: Monday, May 23,

    2011 ...or at least look at this to get inspired

  29. JMX Monday, May 23, 2011

  30. Servlet = HTTP + JSON Monday, May 23, 2011

  31. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 JSON, not always funny to look at The really interesting things are small tidbits of information “hidden” in this blob of data

  32. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit s":"Ok"}]},{"name":"autocomplete","application_status":"ok","counter_tot Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 Is it ok?

  33. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte 079},{"name":"memory.used","counter": 1329},{"name":"cpu","counter":8} atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How much memory do we consume?

  34. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok tion_status":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageServ "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c

    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How many requests have we served?

  35. data is good Monday, May 23, 2011

  36. visualized data is better Monday, May 23, 2011

  37. visualize Monday, May 23, 2011

  38. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

  39. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

  40. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

  41. visualization = insight Monday, May 23, 2011

  42. insight = knowledge Monday, May 23, 2011

  43. knowledge = safety Monday, May 23, 2011

  44. does it work? Monday, May 23, 2011

  45. No. Monday, May 23, 2011

  46. does it work? Monday, May 23, 2011

  47. Yes. Monday, May 23, 2011

  48. which services are utilized? Monday, May 23, 2011

  49. Monday, May 23, 2011 Obviously the top one

  50. available memory? Monday, May 23, 2011

  51. Monday, May 23, 2011

  52. 1 screen + 1 computer = visualization! Monday, May 23,

    2011

  53. why not more? Monday, May 23, 2011

  54. think build server... Monday, May 23, 2011 Your project /

    team probably have a build server monitor, which is a good thing. But how much “dashboard estate” do you spend on production - where your code actually (hopefully) delivers value?

  55. ...only bigger... Monday, May 23, 2011

  56. ...and more Monday, May 23, 2011

  57. what’s happening in production? Monday, May 23, 2011

  58. Monday, May 23, 2011 The dashboard where I work -

    all servers, most services

  59. visualizing data in real time gives insight and safety Monday,

    May 23, 2011

  60. Monday, May 23, 2011 Thanks

  61. Monday, May 23, 2011 Thanks