Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Metrics Driven Monitoring with Sensu/Graphite/G...
Search
Jaime Gago
February 09, 2015
Technology
0
1.1k
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Jaime Gago
February 09, 2015
Tweet
Share
More Decks by Jaime Gago
See All by Jaime Gago
Automating an AWS Complex Infrastructure With Ansible: Lessons learned
jaimegago
1
270
Other Decks in Technology
See All in Technology
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
470
Getting to Know Your Legacy (System) with AI-Driven Software Archeology (WeAreDevelopers World Congress 2025)
feststelltaste
1
130
ビズリーチにおけるリアーキテクティング実践事例 / JJUG CCC 2025 Spring
visional_engineering_and_design
1
120
面倒な作業はAIにおまかせ。Flutter開発をスマートに効率化
ruideengineer
0
260
成長し続けるアプリのためのテストと設計の関係、そして意思決定の記録。
sansantech
PRO
0
120
How Do I Contact HP Printer Support? [Full 2025 Guide for U.S. Businesses]
harrry1211
0
120
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
340
CRE Camp #1 エンジニアリングを民主化するCREチームでありたい話
mntsq
1
130
マネジメントって難しい、けどおもしろい / Management is tough, but fun! #em_findy
ar_tama
7
1.1k
スタートアップに選択肢を 〜生成AIを活用したセカンダリー事業への挑戦〜
nstock
0
200
Lazy application authentication with Tailscale
bluehatbrit
0
210
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
230
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Navigating Team Friction
lara
187
15k
Designing for Performance
lara
610
69k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Producing Creativity
orderedlist
PRO
346
40k
4 Signs Your Business is Dying
shpigford
184
22k
Thoughts on Productivity
jonyablonski
69
4.7k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Transcript
Metrics Driven Monitoring with Sensu/Graphite/Grafana @JaimeGagoTech
VS 10/7/14, 5:14 PM ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000
,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940
GRAPHITE METRIC or EVENT SOURCE FOO METRIC or EVENT SOURCE
BAR Render URL API Render URL API Sensu Graphite Grafana 30000ft Overview TCP Socket TCP Socket STDOUT ALERT! RESTART SERVICE FOO
http://grafana.org • FOSS • Kibana Fork • Client Only •
Fast Growing community
None
Getting Data From Graphite http://my.graphite.org/render?target=absolute(some.metric*.memory.memory-free)&format=json&from=-20min { { { { Graphite
Host Metric Target Accepts: -Graphite functions -Globs Output Period for query JSON PNG CSV
Getting Metrics to Graphite Plain text protocol 'root.foo.metric 42 1412728523'
STDOUT TCP SOCKET Python snippet message = 'root.foo.metric 42 %d\n' % int(time.time())' carbon_server = 'graphite.ops.bluejeans.com' carbon_port = 2003 sock = socket.socket() sock.connect((carbon_server, carbon_port) sock.sendall(message) sock.close() GRAPHITE Whisper
Getting Events to Graphite Json Blob {"what": "Event - deploy",
"tags": “deploy", "data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"}' STDOUT Http post http://graphite/events/ Python Snippet graphite_event_tags = 'sensu-mco-restart %s %s' % (init_script, " ".join(restarted_hosts)) graphite_event_data = ('The Sensu-Mcollective remediation pipeline ' 'send restart to %s on : %s') % (init_script, ",".join(restarted_hosts)) graphite_event_payload = {'what': 'Sensu-MCollective-Restart', 'tags': graphite_event_tags, 'data': graphite_event_data} try: if options.debug: print "Sent graphite event" print 'graphite payload: %s' % (graphite_event_payload) request = urllib2.Request(graphite_events_url, json.dumps(graphite_event_payload)) response = urllib2.urlopen(request) except Exception as e: if options.debug: print 'graphite payload: %s' % (graphite_event_payload) print response print e logger.critical(e) GRAPHITE MySQL
Getting Events from Graphite Http GET GRAPHITE MySQL http://graphite/events/get_data [
{ "data": "deploy happened", "id": 6, "tags": "tag1 ", "what": "grafana demo", "when": 1412974823.0 }, { "data": "deploy happened", "id": 7, "tags": "tag1 ", "what": "grafana demo", "when": 1412976219.0 }, { "data": "Foo happened", "id": 8, "tags": "foo ", "what": "grafana", "when": 1412977300.0 } … ]
Sensu API /aggregates /checks /clients /events /health /info /stashes
Config Dive In
Self Healing Example
Scaling
Self Service Monitoring