Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Infrastructure as Code & Monitoring
Search
portertech
September 16, 2015
Programming
6
1.8k
Infrastructure as Code & Monitoring
Automacon 2015 talk: Infrastructure as Code & monitoring, sharing the same development workflow.
portertech
September 16, 2015
Tweet
Share
More Decks by portertech
See All by portertech
Monitoring Event Pipelines
portertech
0
130
Scaling Sensu Go
portertech
0
300
Understanding Habitat & Kube - A Home Lab Experiment
portertech
0
100
Today's Challenges of Big Infra Monitoring
portertech
1
61
Monitoring with Sensu Go
portertech
2
430
7 Years Of Sensu
portertech
1
110
Chef and Sensu - Delightful Monitoring
portertech
0
100
What is Sensu?
portertech
1
150
Ignite - The Art Of Monitoring
portertech
0
240
Other Decks in Programming
See All in Programming
カクヨムAndroidアプリのリブート
numeroanddev
0
430
複数アプリケーションを育てていくための共通化戦略
irof
10
4k
生成AIで日々のエラー調査を進めたい
yuyaabo
0
620
Cursor AI Agentと伴走する アプリケーションの高速リプレイス
daisuketakeda
1
120
DroidKnights 2025 - 다양한 스크롤 뷰에서의 영상 재생
gaeun5744
3
300
Julia という言語について (FP in Julia « SIDE: F ») for 関数型まつり2025
antimon2
3
970
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
1
270
Haskell でアルゴリズムを抽象化する / 関数型言語で競技プログラミング
naoya
17
4.8k
Datadog RUM 本番導入までの道
shinter61
1
310
イベントストーミングから始めるドメイン駆動設計
jgeem
4
870
「Cursor/Devin全社導入の理想と現実」のその後
saitoryc
0
110
Benchmark
sysong
0
230
Featured
See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
53k
How to train your dragon (web standard)
notwaldorf
92
6.1k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
4
200
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Product Roadmaps are Hard
iamctodd
PRO
53
11k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Designing for humans not robots
tammielis
253
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
920
The Art of Programming - Codeland 2020
erikaheidi
54
13k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Transcript
Infrastructure as Code & Monitoring Sharing the same development workflow.
Sean Porter @PorterTech
FOCUS • The Sensu origin story - where it came
from • What makes Sensu different • IaC development workflows • Testing & monitoring in an IaC workflow • Chef & Sensu in practice
None
THE SENSU ORIGIN STORY • I started at Sonian ~
2010 ◦ Chef 0.7 with AWS #YOLO infrastructure ◦ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite
~ 50 alerts a day Mostly noise.
THE SENSU ORIGIN STORY Triggered @lusis to write: “Why Monitoring
Sucks”
Wanted MOAR!
Build it. Automatic (de)registration Config Management friendly Secure connectivity REST
API Elastic scalability UNIX
July 11th, 2011
What makes Sensu different? What makes it better for CM
driven environments?
None
JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":
["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
CHECK EXECUTION METHODS • Pub/Sub (central orchestration) ◦ e.g. execute
http check on all API nodes • Standalone ◦ Define checks while provisioning node(s) ◦ Scheduled by the local Sensu client
LOCAL CLIENT SOCKET echo '{ \ "name": "mysql_backup", \ "output":
"could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
None
Let’s talk about IaC workflows “The sequence of processes through
which a piece of work passes from initiation to completion” - Google.
BASIC IaC WORKFLOW “It’s all software.”
BASIC IaC WORKFLOW
BASIC IaC WORKFLOW “It’s all software.”
BASIC IaC WORKFLOW Use tests. Still need to review tests
& code quality.
BASIC IaC WORKFLOW TEST ≈ MONITOR
BASIC IaC WORKFLOW Shorten the feedback loop.
Writing IaC tests “A procedure intended to establish the quality,
performance, or reliability of something, especially before it is taken into widespread use” - Google.
TESTING TOOLS • Serverspec ◦ RSpec tests for your servers
◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'
do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
BATS #!/usr/bin/env bats @test "httpd should be running" { run
service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
RUNNING TESTS • Test Kitchen ◦ kitchen.ci • Vagrant plugins
◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
RUNNING TESTS AS SENSU CHECKS • Use the Sensu Serverspec
check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
Chef & Sensu In practice.
SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu • Recipes to install & manage
services • LWRPs for configuring handlers, checks, etc. • Intended to be used by wrapper cookbook ◦ e.g. recipe['monitor::haproxy']
EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe'
command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end
USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check =
data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end
SENSU SERVERSPEC CHECKS # recipe['monitor::serverspec'] sensu_gem 'sensu-plugins-serverspec' sensu_check 'serverspec' do
command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb ' standalone true interval 30 end # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures
BASIC IaC WORKFLOW TEST ≈ MONITOR
Questions? Sean Porter (@PorterTech) sensuapp.org