Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Search
Jaime Gago
February 09, 2015
Technology
0
1k
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Jaime Gago
February 09, 2015
Tweet
Share
More Decks by Jaime Gago
See All by Jaime Gago
Automating an AWS Complex Infrastructure With Ansible: Lessons learned
jaimegago
1
260
Other Decks in Technology
See All in Technology
M5stackで使用できるpHセンサの開発
shinrinakamura
0
120
AWS学習者向けにAzureの解説スライドを作成した話
handy
3
190
MixIT 2024 - Pulumi : Gérer son infra avec son langage de programmation préféré
ju_hnny5
1
120
競技としてのKaggle、役に立つKaggle
yu4u
6
2.3k
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
1k
Improve Your Development Workflow with Gemini Code Assist
meteatamel
0
130
web-application-security
matsuihidetoshi
1
190
Amplify 🩷 Bedrock 〜生成AI入門〜
minorun365
PRO
8
600
コードファーストの考え方。 Amplify Gen2から学ぶAWS次世代のWeb開発体験
yoshiitaka
2
360
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
270
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
330
認知症フレンドリーテックとスタックチャン
naokiuc
0
190
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Bash Introduction
62gerente
605
210k
Building Adaptive Systems
keathley
32
1.9k
Testing 201, or: Great Expectations
jmmastey
30
6.4k
The Invisible Customer
myddelton
114
12k
Documentation Writing (for coders)
carmenintech
61
4k
Become a Pro
speakerdeck
PRO
13
4.6k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
How STYLIGHT went responsive
nonsquared
92
4.8k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
Transcript
Metrics Driven Monitoring with Sensu/Graphite/Grafana @JaimeGagoTech
VS 10/7/14, 5:14 PM ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000
,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940
GRAPHITE METRIC or EVENT SOURCE FOO METRIC or EVENT SOURCE
BAR Render URL API Render URL API Sensu Graphite Grafana 30000ft Overview TCP Socket TCP Socket STDOUT ALERT! RESTART SERVICE FOO
http://grafana.org • FOSS • Kibana Fork • Client Only •
Fast Growing community
None
Getting Data From Graphite http://my.graphite.org/render?target=absolute(some.metric*.memory.memory-free)&format=json&from=-20min { { { { Graphite
Host Metric Target Accepts: -Graphite functions -Globs Output Period for query JSON PNG CSV
Getting Metrics to Graphite Plain text protocol 'root.foo.metric 42 1412728523'
STDOUT TCP SOCKET Python snippet message = 'root.foo.metric 42 %d\n' % int(time.time())' carbon_server = 'graphite.ops.bluejeans.com' carbon_port = 2003 sock = socket.socket() sock.connect((carbon_server, carbon_port) sock.sendall(message) sock.close() GRAPHITE Whisper
Getting Events to Graphite Json Blob {"what": "Event - deploy",
"tags": “deploy", "data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"}' STDOUT Http post http://graphite/events/ Python Snippet graphite_event_tags = 'sensu-mco-restart %s %s' % (init_script, " ".join(restarted_hosts)) graphite_event_data = ('The Sensu-Mcollective remediation pipeline ' 'send restart to %s on : %s') % (init_script, ",".join(restarted_hosts)) graphite_event_payload = {'what': 'Sensu-MCollective-Restart', 'tags': graphite_event_tags, 'data': graphite_event_data} try: if options.debug: print "Sent graphite event" print 'graphite payload: %s' % (graphite_event_payload) request = urllib2.Request(graphite_events_url, json.dumps(graphite_event_payload)) response = urllib2.urlopen(request) except Exception as e: if options.debug: print 'graphite payload: %s' % (graphite_event_payload) print response print e logger.critical(e) GRAPHITE MySQL
Getting Events from Graphite Http GET GRAPHITE MySQL http://graphite/events/get_data [
{ "data": "deploy happened", "id": 6, "tags": "tag1 ", "what": "grafana demo", "when": 1412974823.0 }, { "data": "deploy happened", "id": 7, "tags": "tag1 ", "what": "grafana demo", "when": 1412976219.0 }, { "data": "Foo happened", "id": 8, "tags": "foo ", "what": "grafana", "when": 1412977300.0 } … ]
Sensu API /aggregates /checks /clients /events /health /info /stashes
Config Dive In
Self Healing Example
Scaling
Self Service Monitoring