Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Infrastructure as Code & Monitoring
Search
portertech
September 16, 2015
Programming
6
1.7k
Infrastructure as Code & Monitoring
Automacon 2015 talk: Infrastructure as Code & monitoring, sharing the same development workflow.
portertech
September 16, 2015
Tweet
Share
More Decks by portertech
See All by portertech
Monitoring Event Pipelines
portertech
0
130
Scaling Sensu Go
portertech
0
210
Understanding Habitat & Kube - A Home Lab Experiment
portertech
0
93
Today's Challenges of Big Infra Monitoring
portertech
1
54
Monitoring with Sensu Go
portertech
2
410
7 Years Of Sensu
portertech
1
100
Chef and Sensu - Delightful Monitoring
portertech
0
94
What is Sensu?
portertech
1
130
Ignite - The Art Of Monitoring
portertech
0
230
Other Decks in Programming
See All in Programming
Dev ContainersとGitHub Codespacesの素敵な関係
ymd65536
1
120
Progressive Web Apps für Desktop und Mobile mit Angular (Hands-on)
christianliebel
PRO
0
110
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
130
OpenTelemetryでRailsのパフォーマンス分析を始めてみよう(KoR2024)
ymtdzzz
4
1.5k
Vue SFCのtemplateでTypeScriptの型を活用しよう
tsukkee
3
1.5k
Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines
sucitw
1
190
役立つログに取り組もう
irof
26
8.6k
Nuxt UI Pro、NuxtHub、Nuxt Scripts、Nuxtエコシステムをふんだんに利用して開発するコーポレートサイト@Vue Fes Japan 2024
shingangan
3
880
推し活の ハイトラフィックに立ち向かう Railsとアーキテクチャ - Kaigi on Rails 2024
falcon8823
6
2.1k
Streams APIとTCPフロー制御 / Web Streams API and TCP flow control
tasshi
1
290
【Kaigi on Rails 2024】YOUTRUST スポンサーLT
krpk1900
1
240
Content Security Policy入門 セキュリティ設定と 違反レポートのはじめ方 / Introduction to Content Security Policy Getting Started with Security Configuration and Violation Reporting
uskey512
1
420
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
7.9k
RailsConf 2023
tenderlove
29
880
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Music & Morning Musume
bryan
46
6.1k
Designing for Performance
lara
604
68k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
3
370
Facilitating Awesome Meetings
lara
49
6k
Why Our Code Smells
bkeepers
PRO
334
57k
Bash Introduction
62gerente
608
210k
Transcript
Infrastructure as Code & Monitoring Sharing the same development workflow.
Sean Porter @PorterTech
FOCUS • The Sensu origin story - where it came
from • What makes Sensu different • IaC development workflows • Testing & monitoring in an IaC workflow • Chef & Sensu in practice
None
THE SENSU ORIGIN STORY • I started at Sonian ~
2010 ◦ Chef 0.7 with AWS #YOLO infrastructure ◦ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite
~ 50 alerts a day Mostly noise.
THE SENSU ORIGIN STORY Triggered @lusis to write: “Why Monitoring
Sucks”
Wanted MOAR!
Build it. Automatic (de)registration Config Management friendly Secure connectivity REST
API Elastic scalability UNIX
July 11th, 2011
What makes Sensu different? What makes it better for CM
driven environments?
None
JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":
["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
CHECK EXECUTION METHODS • Pub/Sub (central orchestration) ◦ e.g. execute
http check on all API nodes • Standalone ◦ Define checks while provisioning node(s) ◦ Scheduled by the local Sensu client
LOCAL CLIENT SOCKET echo '{ \ "name": "mysql_backup", \ "output":
"could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
None
Let’s talk about IaC workflows “The sequence of processes through
which a piece of work passes from initiation to completion” - Google.
BASIC IaC WORKFLOW “It’s all software.”
BASIC IaC WORKFLOW
BASIC IaC WORKFLOW “It’s all software.”
BASIC IaC WORKFLOW Use tests. Still need to review tests
& code quality.
BASIC IaC WORKFLOW TEST ≈ MONITOR
BASIC IaC WORKFLOW Shorten the feedback loop.
Writing IaC tests “A procedure intended to establish the quality,
performance, or reliability of something, especially before it is taken into widespread use” - Google.
TESTING TOOLS • Serverspec ◦ RSpec tests for your servers
◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'
do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
BATS #!/usr/bin/env bats @test "httpd should be running" { run
service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
RUNNING TESTS • Test Kitchen ◦ kitchen.ci • Vagrant plugins
◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
RUNNING TESTS AS SENSU CHECKS • Use the Sensu Serverspec
check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
Chef & Sensu In practice.
SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu • Recipes to install & manage
services • LWRPs for configuring handlers, checks, etc. • Intended to be used by wrapper cookbook ◦ e.g. recipe['monitor::haproxy']
EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe'
command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end
USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check =
data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end
SENSU SERVERSPEC CHECKS # recipe['monitor::serverspec'] sensu_gem 'sensu-plugins-serverspec' sensu_check 'serverspec' do
command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb ' standalone true interval 30 end # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures
BASIC IaC WORKFLOW TEST ≈ MONITOR
Questions? Sean Porter (@PorterTech) sensuapp.org