Graphite and Friends at Leeds DevOps

Transcript

1. MONITORING! GRAPHITE and friends
2. Structure ☑ Context ☐ Monitoring ☐ Graphite ☐ Front Ends ☐ Feeding ☐ Graphite Tips ☐ Managing / Scaling
3. context
4. Mark Crossfield, @mrmanc http://markcrossfield.co.uk Auto Trader Engineer for 7 years Continuous Delivery & Web
5. Hiring! http://careers.autotrader.co.uk/
6. AWESOME NEW OFFICES
7. page impressions / day 70 million
8. unique monthly users 14.5 million
9. searches / second at peak 1,500
10. adverts 435k
11. product & technology staff 323
12. servers 2000
13. code bases 250
14. monitoring
15. Velocity 2011 https://www.flickr.com/photos/kellan/5839797269/
16. January 2012
17. http://blog.tagman.com/2012/03/just-one-second-delay-in-page-load-can-cause-7-loss-in-customer-conversions/ One Second Delay In Page-Load Can Cause 
 7% Loss In Customer Conversions “ ” 47% of consumers expect a page to load in 2 seconds or less “
18. types of
 metric… http://code.flickr.net/2008/10/27/counting-timing/ 3
19. counters easy as pie to aggregate—just sum
20. timers harder to aggregate—need percentiles
 this can make scaling difficult
21. gauges more nuanced—keep last value and use means
22. graphite
23. time series database graphite lets you visualise metrics from it’s
24. turns this local.random.diceroll 4 1415727632 <metric path> <metric value> <metric timestamp>
25. into this
26. Die, composer. Die.
27. grafana is a nicer front end… more later
28. functions let you transform and summarise the values over time
29. aggregation precision reduces over time One year : daily One month: hourly One week: 5min One day: min
30. fixed size roll up means metrics do not grow over time
32. THESE ARE NOT THE DOCS YOU ARE LOOKING FOR
33. Docs Moved
34. Old, out dated docs New docs
35. graphite for summarised long term trends
36. elasticsearch
 logstash
 kibana for fine grained events
37. graphite architecture
38. Bucky, Collectl, Diamond, Ganglia, Graphite PowerShell Functions, HoardD, Host sFlow, jmxtrans, Logster, metrics-sampler, Sensu, SqlToGraphite, SSC Serv, Backstop, Evenflow, Graphite-Newrelic, Graphite-relay, Graphios, Grockets, Ledbetter, pipe-to-graphite, statsd, Charcoal, Descartes, Dusk, Firefly, Gdash, Giraffe, Grafana, graphitus, Graph-Explorer, Graph-Index, Graphene, Graphite- Observer, Graphite-Tattle, Graphiti, Graphitoid, Graphsky, Hubot, Leonardo, Orion, Pencil, Seyren, Tasseo, Tessera, TimeseriesWidget, Cabot, graphite-beacon, rearview, Rocksteady, Shinken, Therry Plenty of adoption
39. front ends
40. image api for individual charts
41. grafana is awesome & free as in beer and speech http://play.grafana.org/#/dashboard/db/grafana-play-home
42. http://grafana.org/
44. interactive charts let you explore values / filter series
45. time window synchronises
46. replaces composer with expression parser and builder
47. explore metrics
48. dynamic filtering using filters from queries
49. annotations show events from graphite or a query
50. no back end (elasticsearch optional)
51. dashboard sharing and playlists
52. dark and light themes
53. no backend awesome concept, better with elastic search
54. presentation grafana presentation at monitorama portland 2014 http://grafana.org/blog/2014/05/25/monitorama-video-and-update.html
55. http://shopify.github.io/dashing/
57. feeding need input
58. mark@spaceport  ~ $ echo "random.diceroll 4 `date +%s`" | nc -q0 graphite.org 2003
59. decouple don’t make everything talk graphite
60. label your axes, before it is too late talk about units, frequencies, and include in the metric name
61. log lines parsed with logster / second 11,000
62. statsd aggregates metrics in real time and sends to graphite
63. collectd can collect and send server metrics to graphite
64. 10s Frequency! be careful that collectd doesn’t flood graphite
65. logstash graphite supported as output (and input)
66. metrics by coda hale.
67. http://metrics.codahale.com/ codahale metrics
68. instruments your java components.
69. aggregates timers, histograms etc
70. no statsd unless you need cross host aggregation
71. tips mostly plagiarised
72. writing twice overwrites carbon does no aggregation for you
73. feeding interval == graphite bucket this is no coincidence
74. xFilesFactor sparse metrics might not appear http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-9
75. carbon limits writing new metrics avoids swamping disk with write IO http://bit.ly/maxcreates
76. graphite bookmarklet useful to load charts from images http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-2
77. timeShift(series, duration) e.g. show yesterday’s metric against today’s timeShift(“apache.http.requests”, “-1day”) http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.timeShift
78. groupByNode(series, node, aggregate) e.g. aggregate many series using one node groupByNode(“collectd.*.cpu.*.value”, “2”, “maxSeries”) http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.groupByNode
79. cumulative(seriesList) show how a rate (e.g. per sec) adds up over the day http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.cumulative
80. host.cpu-[0-7].cpu-{user,system}.value wild cards allow filtering of nodes http://graphite.readthedocs.org/en/0.9.12/terminology.html#term-series-list
81. teatime is 4pm Graphite understands the UNIX At time specification e.g. noon yesterday, now-2weeks http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html#IAT_STYLE_TIME_SPECIFICATION
82. monitor carbon carbon records it’s own metrics per minute to graphite http://obfuscurity.com/2012/06/Watching-the-Carbon-Feed
83. whisper-*.py create, dump, fetch, info, merge, resize, set-aggregation method, update, diff https://github.com/graphite-project/whisper
84. summarize be suspicious cautious around aggregation boundaries http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.summarize
85. holt winters intelligent aberration detection http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.holtWintersAberration
86. events(*tags) number of events matching tags at this point in time use to annotate your charts e.g. events(“deploy”, “change”) http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.events
89. managing / scaling
90. metrics relayed / minute 300,000
91. application metrics 30,000
92. write iops / second / cache (~8% of total activity) 8,000
93. sqlite do not run in production
94. scale with one box while you can…
95. Graphite Architecture
96. relay lets you shard (consistent hash) or replicate
98. carbon docs leave a lot to be desired
99. blogs only slightly contradictory
100. http://bitprophet.org/blog/2013/03/07/graphite/
101. https://gist.github.com/obfuscurity/63399584ea4d95f921e4
102. https://answers.launchpad.net/graphite/+question/178969
103. http://grey-boundary.com/the-architecture-of-clustering-graphite/
104. carbonate provides missing bits of carbon to assist scaling https://github.com/jssjr/carbonate
105. graphite-api without the front end https://github.com/brutasse/graphite-api functional goodness
106. @obfuscurity great blogger and an authority on graphite http://obfuscurity.com/
108. Summary •Good tool for long term trending of time series data •Use Grafana and perhaps Dashing as a front end •Feed your data in through statsd, collectd & metrics •Lever the functions •You may experience scaling pain
109. QUESTIONS?