Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stupid Graphite Tips
Search
Jason Dixon
November 04, 2013
Technology
37
4.8k
Stupid Graphite Tips
My Ignite talk from DevOpsDays Portland on November 4, 2013.
Jason Dixon
November 04, 2013
Tweet
Share
More Decks by Jason Dixon
See All by Jason Dixon
Graphite 1.0-ish
obfuscurity
0
340
Monitorama PDX 2015 - Closing Plenary
obfuscurity
1
450
The State of Open-Source Monitoring
obfuscurity
146
49k
Trending with Purpose
obfuscurity
11
1.3k
Other Decks in Technology
See All in Technology
スキルだけでは満たせない、 “組織全体に”なじむオンボーディング/Onboarding that fits “throughout the organization” and cannot be satisfied by skills alone
bitkey
0
180
データベースの負荷を紐解く/untangle-the-database-load
emiki
2
520
Share my, our lessons from the road to re:Invent
naospon
0
150
ExaDB-XSで利用されているExadata Exascaleについて
oracle4engineer
PRO
3
260
設計を積み重ねてシステムを刷新する
sansantech
PRO
0
170
実は強い 非ViTな画像認識モデル
tattaka
3
1.3k
Ruby on Railsで持続可能な開発を行うために取り組んでいること
am1157154
3
160
Snowflakeの開発・運用コストをApache Icebergで効率化しよう!~機能と活用例のご紹介~
sagara
1
470
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
220
Pwned Labsのすゝめ
ken5scal
2
450
入門 PEAK Threat Hunting @SECCON
odorusatoshi
0
160
急成長する企業で作った、エンジニアが輝ける制度/ 20250227 Rinto Ikenoue
shift_evolve
0
140
Featured
See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Being A Developer After 40
akosma
89
590k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Optimizing for Happiness
mojombo
376
70k
KATA
mclloyd
29
14k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Practical Orchestrator
shlominoach
186
10k
The Cult of Friendly URLs
andyhume
78
6.2k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
30
4.6k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
For a Future-Friendly Web
brad_frost
176
9.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.3k
Transcript
Stupid Graphite Tips DevOpsDays Portland 2013
What is Graphite? carbon whisper metrics
require 'socket' require ‘time’ conn = TCPSocket.new 'graphite.example.com', 2003 conn.puts
"foo #{value} #{Time.now.to_i}\n" conn.close metric key value timestamp $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003 metric key value timestamp Sending metrics to Carbon
Organizing metrics • Prefix with your collector/agent/app source • Keep
test/development stuff in check • Automatically purge test & release metrics • Avoid the urge to include your username
and Tracking deployments drawAsInfinite(color(custom.instances.*.killed,"white")) summarize(sumSeries(group(custom.instances.*.killed)), "1d")
mostDeviant(5, runtime.*.load.1m) Finding the Outliers p.s. Don’t forget about sortByMaxima,
sortByMinima and limit !
Adjusting Rate scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333) doesn’t scale! ... because as soon as
you hit a new resolution, your math is wrong. scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)
Time Shifting sumSeries(mail.hosts.mta-*.messages.in) timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")
Counting metrics sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))
Hybrid Graphs mail.hosts.mta-*.messages.in alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)
Renaming Keys on-the-fly Ever wanted to redefine your metric key
in the middle of a query because the wildcard wasn’t quite in the right spot? aliasSub(hosts.web-01-pdx-prod-example- com.metric,”-(\w+)-prod”, ”.\1.prod”)
Graph Bookmarklet javascript:url=prompt("Enter Url");if (url) {content.Composer.loadMyGraph("temp", decodeURIComponent(url));};
Listing metrics $ curl -s http://graphite/metrics/index.json | json_pp | tail
"graphite-1.processes.ps_state-blocked.value", "graphite-1.processes.ps_state-paging.value", "graphite-1.processes.ps_state-running.value", "graphite-1.processes.ps_state-sleeping.value", "graphite-1.processes.ps_state-stopped.value", "graphite-1.processes.ps_state-zombies.value", "graphite-1.users.users.users", "test.bar", "test.foo" ]
Grokking Whisper $ whisper-info.py foo.wsp maxRetention: 31536000 xFilesFactor: 0.0 aggregationMethod:
average fileSize: 1262932 Archive 0 retention: 86400 secondsPerPoint: 1 points: 86400 size: 1036800 offset: 52 $ whisper-fetch.py foo.wsp | tail 1383585660! None 1383585720! 29.529000 1383585780! 87.016000 1383585840! 33.416000 1383585900! 29.012000 1383585960! 31.164000 1383586020! None 1383586080! 29.702000 1383586140! 29.299000 1383586200! None
Repairing Data $ whisper-fetch.py foo.wsp | grep -v None 1376066700
1.000000 1376066760 226.000000 1376066820 215.000000 1376066880 266.000000 1376066940 237.000000 1376072760 5.000000 1376074560 1.000000 1376075280 2.000000 $ for i in 1376066760 1376066820 1376066880 1376066940; \ do whisper-update.py foo.wsp ${i}:0"; done
Know your Carbon sumSeries(group(carbon.agents.*.updateOperations)) sumSeries(group(carbon.agents.*.metricsReceived)) sumSeries(group(carbon.agents.*.committedPoints)) secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate))) secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage))) secondYAxis(sumSeries(group(carbon.agents.*.creates))) p.s.
Check out @scott_sanders’ collectd-tail examples https://gist.github.com/jssjr/35b592b4852625449c27
Tagged Events $ curl -X POST http://graphite/events/ \ -d '{"what":
"Something Interesting", "tags" : "wtf", "data" : "foo bar"}' events(“wtf”) events(“wtf”,”zomg”) events(“*”)
Dat Spark $ brew install spark $ curl -H 'Accept:
application/json' -s "http://graphite/ render/?target=foo&format=json&from=-30min" | json_pp | grep ',' | grep -v '\]' | grep -v target | spark ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂
Scaling dat Carbon Open Spaces, doge. omg multicore hip fanout
replication me much io vintage metrics
Thank You