What's happening in production?

Transcript

1. Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ marten.gustafson@gmail.com What’s happening in production Monday,

   May 23, 2011

2. Hi! Monday, May 23, 2011 My name is Mårten, I

   work as a software developer But I usually also have one foot - sometimes a whole leg - in our infrastructure

3. Monday, May 23, 2011 I work here. We provide a

   web site, that’s our offering. Our core.

4. Monday, May 23, 2011 Our site is, amongst other things,

   built from a slew of Java services We have multiple services on each server

5. Monday, May 23, 2011 We like to be availbale, thus

   we have multiple servers Same service, on multiple servers Multiple services on same server

6. which services are utilized? Monday, May 23, 2011

7. most? Monday, May 23, 2011

8. least? Monday, May 23, 2011

9. most memory? Monday, May 23, 2011

10. highest latency? Monday, May 23, 2011

11. does it work? Monday, May 23, 2011

12. http://www.picpop.com/ Monday, May 23, 2011

13. what’s happening in production? Monday, May 23, 2011

14. right. this. second. Monday, May 23, 2011

15. measure! Monday, May 23, 2011

16. everything! Monday, May 23, 2011

17. in real time! Monday, May 23, 2011

18. it’s cheap... Monday, May 23, 2011

19. ...and simple! Monday, May 23, 2011

20. meter = metrics.meter("results", "records", TimeUnit.SECONDS) ...calculate result... meter.mark(results.size()) Monday, May

    23, 2011

21. counter = Metrics.newCounter( getClass(), "requests") ...receive request... counter.inc() Monday, May

    23, 2011

22. counters Monday, May 23, 2011

23. timers Monday, May 23, 2011

24. averages Monday, May 23, 2011

25. etc... Monday, May 23, 2011

26. how? Monday, May 23, 2011

27. develop something, or... Monday, May 23, 2011

28. https://github.com /codahale/metrics ...for java & scala use: Monday, May 23,

    2011 ...or at least look at this to get inspired

29. JMX Monday, May 23, 2011

30. Servlet = HTTP + JSON Monday, May 23, 2011

31. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 JSON, not always funny to look at The really interesting things are small tidbits of information “hidden” in this blob of data

32. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit s":"Ok"}]},{"name":"autocomplete","application_status":"ok","counter_tot Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 Is it ok?

33. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ

    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte 079},{"name":"memory.used","counter": 1329},{"name":"cpu","counter":8} atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How much memory do we consume?

34. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"} tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter", r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok tion_status":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageServ "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status": calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c

    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"} n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3, Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot": ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":" nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts" tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name" nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]} Monday, May 23, 2011 How many requests have we served?

35. data is good Monday, May 23, 2011

36. visualized data is better Monday, May 23, 2011

37. visualize Monday, May 23, 2011

38. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

39. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

40. Monday, May 23, 2011 Visualization makes it easy to distinguish

    worse from better

41. visualization = insight Monday, May 23, 2011

42. insight = knowledge Monday, May 23, 2011

43. knowledge = safety Monday, May 23, 2011

44. does it work? Monday, May 23, 2011

45. No. Monday, May 23, 2011

46. does it work? Monday, May 23, 2011

47. Yes. Monday, May 23, 2011

48. which services are utilized? Monday, May 23, 2011

49. Monday, May 23, 2011 Obviously the top one

50. available memory? Monday, May 23, 2011

51. Monday, May 23, 2011

52. 1 screen + 1 computer = visualization! Monday, May 23,

    2011

53. why not more? Monday, May 23, 2011

54. think build server... Monday, May 23, 2011 Your project /

    team probably have a build server monitor, which is a good thing. But how much “dashboard estate” do you spend on production - where your code actually (hopefully) delivers value?

55. ...only bigger... Monday, May 23, 2011

56. ...and more Monday, May 23, 2011

57. what’s happening in production? Monday, May 23, 2011

58. Monday, May 23, 2011 The dashboard where I work -

    all servers, most services

59. visualizing data in real time gives insight and safety Monday,

    May 23, 2011

60. Monday, May 23, 2011 Thanks

61. Monday, May 23, 2011 Thanks