What's happening in production?

Lightning talk delivered at Agilasverige (Agile Sweden) on metrics and that you as a developer should be able to know what's happening in your production environment in real time

Approx 10 minutes.

Transcript

1. Mårten Gustafson
   @martengustafson
   http://marten.gustafson.pp.se/
   marten.gustafson@gmail.com
   What’s happening in production
   Monday, May 23, 2011
   

   View Slide

2. Hi!
   Monday, May 23, 2011
   My name is Mårten, I work as a software developer
   But I usually also have one foot - sometimes a whole leg - in our infrastructure
   

   View Slide

3. Monday, May 23, 2011
   I work here.
   We provide a web site, that’s our offering. Our core.
   

   View Slide

4. Monday, May 23, 2011
   Our site is, amongst other things, built from a slew of Java services
   We have multiple services on each server
   

   View Slide

5. Monday, May 23, 2011
   We like to be availbale, thus we have multiple servers
   Same service, on multiple servers
   Multiple services on same server
   

   View Slide

6. which services are utilized?
   Monday, May 23, 2011
   

   View Slide

7. most?
   Monday, May 23, 2011
   

   View Slide

8. least?
   Monday, May 23, 2011
   

   View Slide

9. most memory?
   Monday, May 23, 2011
   

   View Slide

10. highest latency?
    Monday, May 23, 2011
    

    View Slide

11. does it work?
    Monday, May 23, 2011
    

    View Slide

12. http://www.picpop.com/
    Monday, May 23, 2011
    

    View Slide

13. what’s happening in
    production?
    Monday, May 23, 2011
    

    View Slide

14. right. this. second.
    Monday, May 23, 2011
    

    View Slide

15. measure!
    Monday, May 23, 2011
    

    View Slide

16. everything!
    Monday, May 23, 2011
    

    View Slide

17. in real time!
    Monday, May 23, 2011
    

    View Slide

18. it’s cheap...
    Monday, May 23, 2011
    

    View Slide

19. ...and simple!
    Monday, May 23, 2011
    

    View Slide

20. meter = metrics.meter("results",
    "records",
    TimeUnit.SECONDS)
    ...calculate result...
    meter.mark(results.size())
    Monday, May 23, 2011
    

    View Slide

21. counter = Metrics.newCounter( getClass(),
    "requests")
    ...receive request...
    counter.inc()
    Monday, May 23, 2011
    

    View Slide

22. counters
    Monday, May 23, 2011
    

    View Slide

23. timers
    Monday, May 23, 2011
    

    View Slide

24. averages
    Monday, May 23, 2011
    

    View Slide

25. etc...
    Monday, May 23, 2011
    

    View Slide

26. how?
    Monday, May 23, 2011
    

    View Slide

27. develop something, or...
    Monday, May 23, 2011
    

    View Slide

28. https://github.com
    /codahale/metrics
    ...for java & scala use:
    Monday, May 23, 2011
    ...or at least look at this to get inspired
    

    View Slide

29. JMX
    Monday, May 23, 2011
    

    View Slide

30. Servlet
    =
    HTTP + JSON
    Monday, May 23, 2011
    

    View Slide

31. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet
    counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}
    tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter",
    r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta
    ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon
    Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok
    ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService
    "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status":
    calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c
    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ
    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag
    Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"}
    n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore
    observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit
    tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3,
    Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor
    "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot":
    ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService
    "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":"
    nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte
    er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo
    atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re
    e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts"
    tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name"
    nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath
    ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]}
    Monday, May 23, 2011
    JSON, not always funny to look at
    The really interesting things are small tidbits of information “hidden” in this blob of data
    

    View Slide

32. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet
    counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}
    tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter",
    r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta
    ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon
    Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok
    ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService
    "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status":
    calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c
    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ
    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag
    Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"}
    n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore
    observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit
    s":"Ok"}]},{"name":"autocomplete","application_status":"ok","counter_tot
    Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor
    "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot":
    ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService
    "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":"
    nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte
    er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo
    atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re
    e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts"
    tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name"
    nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath
    ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]}
    Monday, May 23, 2011
    Is it ok?
    

    View Slide

33. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet
    counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}
    tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter",
    r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta
    ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon
    Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok
    ication_status":"Ok","counter_tot":19,"request_counters":[{"name":"ImageService
    "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status":
    calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c
    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ
    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag
    Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"}
    n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore
    observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit
    tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3,
    Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor
    "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot":
    ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService
    "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":"
    nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte
    079},{"name":"memory.used","counter": 1329},{"name":"cpu","counter":8}
    atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re
    e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts"
    tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name"
    nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath
    ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]}
    Monday, May 23, 2011
    How much memory do we consume?
    

    View Slide

34. )","counter":0},{"name":"AutocompleteService","counter":0},{"name":"Autocomplet
    counter":0}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}
    tion_status":"Ok","counter_tot":0,"request_counters":[{"name":"PointConverter",
    r-deprecated","counter":0}],"monitored_services":[{"name":"PointConverter","sta
    ion_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhon
    Service","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok
    tion_status":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageServ
    "name":"ImageService","status":"Ok"}]},{"name":"locality","application_status":
    calityResolverService-cityName","counter":0},{"name":"LocalityResolverService-c
    e-countyName","counter":0}],"monitored_services":[{"name":"LocalityResolverServ
    ion_status":"Ok","counter_tot":0,"request_counters":[{"name":"SmsService:messag
    Service","counter":0}],"monitored_services":[{"name":"SmsService","status":"Ok"}
    n_status":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-fore
    observation","counter":0},{"name":"WeatherService-forecast","counter":0}],"monit
    tatus":"Ok"}]},{"name":"autocomplete","application_status":"Ok","counter_tot":3,
    Service:route","counter":0},{"name":"AutocompleteService","counter":3}],"monitor
    "status":"Ok"}]},{"name":"poi-service","application_status":"Ok","counter_tot":
    ks","counter":47},{"name":"PoiService:metadata","counter":8},{"name":"PoiService
    "name":"PoiService","status":"Ok"}]},{"name":"JvmMonitor","application_status":"
    nters":[{"name":"memory.allocated","counter":4079},{"name":"memory.free","counte
    er":4079},{"name":"memory.used","counter":1329},{"name":"cpu","counter":8}],"mo
    atus":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot":0,"re
    e-hourbyhour-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts"
    tails-forecast","counter":0},{"name":"WeatherService-total","counter":0},{"name"
    nter":0},{"name":"WeatherService-longrange-forecast","counter":0},{"name":"Weath
    ,"counter":0}],"monitored_services":[{"name":"WeatherService","status":"Ok"}]}]}
    Monday, May 23, 2011
    How many requests have we served?
    

    View Slide

35. data is good
    Monday, May 23, 2011
    

    View Slide

36. visualized data
    is better
    Monday, May 23, 2011
    

    View Slide

37. visualize
    Monday, May 23, 2011
    

    View Slide

38. Monday, May 23, 2011
    Visualization makes it easy to distinguish worse from better
    

    View Slide

39. Monday, May 23, 2011
    Visualization makes it easy to distinguish worse from better
    

    View Slide

40. Monday, May 23, 2011
    Visualization makes it easy to distinguish worse from better
    

    View Slide

41. visualization = insight
    Monday, May 23, 2011
    

    View Slide

42. insight = knowledge
    Monday, May 23, 2011
    

    View Slide

43. knowledge = safety
    Monday, May 23, 2011
    

    View Slide

44. does it work?
    Monday, May 23, 2011
    

    View Slide

45. No.
    Monday, May 23, 2011
    

    View Slide

46. does it work?
    Monday, May 23, 2011
    

    View Slide

47. Yes.
    Monday, May 23, 2011
    

    View Slide

48. which services are utilized?
    Monday, May 23, 2011
    

    View Slide

49. Monday, May 23, 2011
    Obviously the top one
    

    View Slide

50. available memory?
    Monday, May 23, 2011
    

    View Slide

51. Monday, May 23, 2011
    

    View Slide

52. 1 screen
    +
    1 computer
    =
    visualization!
    Monday, May 23, 2011
    

    View Slide

53. why not more?
    Monday, May 23, 2011
    

    View Slide

54. think build server...
    Monday, May 23, 2011
    Your project / team probably have a build server monitor, which is a good thing.
    But how much “dashboard estate” do you spend on production - where your code actually
    (hopefully) delivers value?
    

    View Slide

55. ...only bigger...
    Monday, May 23, 2011
    

    View Slide

56. ...and more
    Monday, May 23, 2011
    

    View Slide

57. what’s happening in
    production?
    Monday, May 23, 2011
    

    View Slide

58. Monday, May 23, 2011
    The dashboard where I work - all servers, most services
    

    View Slide

59. visualizing data in
    real time gives
    insight and safety
    Monday, May 23, 2011
    

    View Slide

60. Monday, May 23, 2011
    Thanks
    

    View Slide

61. Monday, May 23, 2011
    Thanks
    

    View Slide