Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Metrics Driven Monitoring with Sensu/Graphite/G...
Search
Jaime Gago
February 09, 2015
Technology
0
1.1k
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Metrics Driven Monitoring with Sensu/Graphite/Grafana
Jaime Gago
February 09, 2015
Tweet
Share
More Decks by Jaime Gago
See All by Jaime Gago
Automating an AWS Complex Infrastructure With Ansible: Lessons learned
jaimegago
1
270
Other Decks in Technology
See All in Technology
OPENLOGI Company Profile for engineer
hr01
1
34k
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
7
5.3k
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
960
整頓のジレンマとの戦い〜Tidy First?で振り返る事業とキャリアの歩み〜/Fighting the tidiness dilemma〜Business and Career Milestones Reflected on in Tidy First?〜
bitkey
2
16k
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
340
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
340
AI専用のリンターを作る #yumemi_patch
bengo4com
5
4.3k
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
470
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
230
OPENLOGI Company Profile
hr01
0
67k
生成AI開発案件におけるClineの業務活用事例とTips
shinya337
0
260
20250707-AI活用の個人差を埋めるチームづくり
shnjtk
4
3.9k
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
Why Our Code Smells
bkeepers
PRO
336
57k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Typedesign – Prime Four
hannesfritz
42
2.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Embracing the Ebb and Flow
colly
86
4.7k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Transcript
Metrics Driven Monitoring with Sensu/Graphite/Grafana @JaimeGagoTech
VS 10/7/14, 5:14 PM ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000
,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 ,000,000 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940
GRAPHITE METRIC or EVENT SOURCE FOO METRIC or EVENT SOURCE
BAR Render URL API Render URL API Sensu Graphite Grafana 30000ft Overview TCP Socket TCP Socket STDOUT ALERT! RESTART SERVICE FOO
http://grafana.org • FOSS • Kibana Fork • Client Only •
Fast Growing community
None
Getting Data From Graphite http://my.graphite.org/render?target=absolute(some.metric*.memory.memory-free)&format=json&from=-20min { { { { Graphite
Host Metric Target Accepts: -Graphite functions -Globs Output Period for query JSON PNG CSV
Getting Metrics to Graphite Plain text protocol 'root.foo.metric 42 1412728523'
STDOUT TCP SOCKET Python snippet message = 'root.foo.metric 42 %d\n' % int(time.time())' carbon_server = 'graphite.ops.bluejeans.com' carbon_port = 2003 sock = socket.socket() sock.connect((carbon_server, carbon_port) sock.sendall(message) sock.close() GRAPHITE Whisper
Getting Events to Graphite Json Blob {"what": "Event - deploy",
"tags": “deploy", "data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"}' STDOUT Http post http://graphite/events/ Python Snippet graphite_event_tags = 'sensu-mco-restart %s %s' % (init_script, " ".join(restarted_hosts)) graphite_event_data = ('The Sensu-Mcollective remediation pipeline ' 'send restart to %s on : %s') % (init_script, ",".join(restarted_hosts)) graphite_event_payload = {'what': 'Sensu-MCollective-Restart', 'tags': graphite_event_tags, 'data': graphite_event_data} try: if options.debug: print "Sent graphite event" print 'graphite payload: %s' % (graphite_event_payload) request = urllib2.Request(graphite_events_url, json.dumps(graphite_event_payload)) response = urllib2.urlopen(request) except Exception as e: if options.debug: print 'graphite payload: %s' % (graphite_event_payload) print response print e logger.critical(e) GRAPHITE MySQL
Getting Events from Graphite Http GET GRAPHITE MySQL http://graphite/events/get_data [
{ "data": "deploy happened", "id": 6, "tags": "tag1 ", "what": "grafana demo", "when": 1412974823.0 }, { "data": "deploy happened", "id": 7, "tags": "tag1 ", "what": "grafana demo", "when": 1412976219.0 }, { "data": "Foo happened", "id": 8, "tags": "foo ", "what": "grafana", "when": 1412977300.0 } … ]
Sensu API /aggregates /checks /clients /events /health /info /stashes
Config Dive In
Self Healing Example
Scaling
Self Service Monitoring