Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Metrics Driven Monitoring with Sensu/Graphite/G...
Search
Jaime Gago
February 09, 2015
Technology
0
1.1k
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Jaime Gago
February 09, 2015
Tweet
Share
More Decks by Jaime Gago
See All by Jaime Gago
Automating an AWS Complex Infrastructure With Ansible: Lessons learned
jaimegago
1
270
Other Decks in Technology
See All in Technology
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
950
より良いプロダクトの開発を目指して - 情報を中心としたプロダクト開発 #phpcon #phpcon2025
bengo4com
1
3.2k
5min GuardDuty Extended Threat Detection EKS
takakuni
0
180
Lambda Web Adapterについて自分なりに理解してみた
smt7174
5
150
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 完全版 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming - Expanded
tomzoh
4
3.4k
MUITにおける開発プロセスモダナイズの取り組みと開発生産性可視化の取り組みについて / Modernize the Development Process and Visualize Development Productivity at MUIT
muit
1
11k
LangSmith×Webhook連携で実現するプロンプトドリブンCI/CD
sergicalsix
1
190
KubeCon + CloudNativeCon Japan 2025 に行ってきた! & containerd の新機能紹介
honahuku
0
120
怖くない!はじめてのClaude Code
shinya337
0
330
PO初心者が考えた ”POらしさ”
nb_rady
0
180
FOSS4G 2025 KANSAI QGISで点群データをいろいろしてみた
kou_kita
0
360
Yamla: Rustでつくるリアルタイム性を追求した機械学習基盤 / Yamla: A Rust-Based Machine Learning Platform Pursuing Real-Time Capabilities
lycorptech_jp
PRO
4
210
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Fireside Chat
paigeccino
37
3.5k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
680
How GitHub (no longer) Works
holman
314
140k
Speed Design
sergeychernyshev
32
1k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
4 Signs Your Business is Dying
shpigford
184
22k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Transcript
Metrics Driven Monitoring with Sensu/Graphite/Grafana @JaimeGagoTech
VS 10/7/14, 5:14 PM ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000
,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940
GRAPHITE METRIC or EVENT SOURCE FOO METRIC or EVENT SOURCE
BAR Render URL API Render URL API Sensu Graphite Grafana 30000ft Overview TCP Socket TCP Socket STDOUT ALERT! RESTART SERVICE FOO
http://grafana.org • FOSS • Kibana Fork • Client Only •
Fast Growing community
None
Getting Data From Graphite http://my.graphite.org/render?target=absolute(some.metric*.memory.memory-free)&format=json&from=-20min { { { { Graphite
Host Metric Target Accepts: -Graphite functions -Globs Output Period for query JSON PNG CSV
Getting Metrics to Graphite Plain text protocol 'root.foo.metric 42 1412728523'
STDOUT TCP SOCKET Python snippet message = 'root.foo.metric 42 %d\n' % int(time.time())' carbon_server = 'graphite.ops.bluejeans.com' carbon_port = 2003 sock = socket.socket() sock.connect((carbon_server, carbon_port) sock.sendall(message) sock.close() GRAPHITE Whisper
Getting Events to Graphite Json Blob {"what": "Event - deploy",
"tags": “deploy", "data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"}' STDOUT Http post http://graphite/events/ Python Snippet graphite_event_tags = 'sensu-mco-restart %s %s' % (init_script, " ".join(restarted_hosts)) graphite_event_data = ('The Sensu-Mcollective remediation pipeline ' 'send restart to %s on : %s') % (init_script, ",".join(restarted_hosts)) graphite_event_payload = {'what': 'Sensu-MCollective-Restart', 'tags': graphite_event_tags, 'data': graphite_event_data} try: if options.debug: print "Sent graphite event" print 'graphite payload: %s' % (graphite_event_payload) request = urllib2.Request(graphite_events_url, json.dumps(graphite_event_payload)) response = urllib2.urlopen(request) except Exception as e: if options.debug: print 'graphite payload: %s' % (graphite_event_payload) print response print e logger.critical(e) GRAPHITE MySQL
Getting Events from Graphite Http GET GRAPHITE MySQL http://graphite/events/get_data [
{ "data": "deploy happened", "id": 6, "tags": "tag1 ", "what": "grafana demo", "when": 1412974823.0 }, { "data": "deploy happened", "id": 7, "tags": "tag1 ", "what": "grafana demo", "when": 1412976219.0 }, { "data": "Foo happened", "id": 8, "tags": "foo ", "what": "grafana", "when": 1412977300.0 } … ]
Sensu API /aggregates /checks /clients /events /health /info /stashes
Config Dive In
Self Healing Example
Scaling
Self Service Monitoring