Tweet
Tweet

Slide 1

Slide 1 text

Infrastructure as Code & Monitoring Sharing the same development workflow.

Slide 2

Slide 2 text

Sean Porter @PorterTech

Slide 3

Slide 3 text

FOCUS ● The Sensu origin story - where it came from ● What makes Sensu different ● IaC development workflows ● Testing & monitoring in an IaC workflow ● Chef & Sensu in practice

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

THE SENSU ORIGIN STORY ● I started at Sonian ~ 2010 ○ Chef 0.7 with AWS #YOLO infrastructure ○ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite

Slide 6

Slide 6 text

~ 50 alerts a day Mostly noise.

Slide 7

Slide 7 text

THE SENSU ORIGIN STORY Triggered @lusis to write: “Why Monitoring Sucks”

Slide 8

Slide 8 text

Wanted MOAR!

Slide 9

Slide 9 text

Build it. Automatic (de)registration Config Management friendly Secure connectivity REST API Elastic scalability UNIX

Slide 10

Slide 10 text

July 11th, 2011

Slide 11

Slide 11 text

What makes Sensu different? What makes it better for CM driven environments?

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers": ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }

Slide 14

Slide 14 text

CHECK EXECUTION METHODS ● Pub/Sub (central orchestration) ○ e.g. execute http check on all API nodes ● Standalone ○ Define checks while provisioning node(s) ○ Scheduled by the local Sensu client

Slide 15

Slide 15 text

LOCAL CLIENT SOCKET echo '{ \ "name": "mysql_backup", \ "output": "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Let’s talk about IaC workflows “The sequence of processes through which a piece of work passes from initiation to completion” - Google.

Slide 18

Slide 18 text

BASIC IaC WORKFLOW “It’s all software.”

Slide 19

Slide 19 text

BASIC IaC WORKFLOW

Slide 20

Slide 20 text

BASIC IaC WORKFLOW “It’s all software.”

Slide 21

Slide 21 text

BASIC IaC WORKFLOW Use tests. Still need to review tests & code quality.

Slide 22

Slide 22 text

BASIC IaC WORKFLOW TEST ≈ MONITOR

Slide 23

Slide 23 text

BASIC IaC WORKFLOW Shorten the feedback loop.

Slide 24

Slide 24 text

Writing IaC tests “A procedure intended to establish the quality, performance, or reliability of something, especially before it is taken into widespread use” - Google.

Slide 25

Slide 25 text

TESTING TOOLS ● Serverspec ○ RSpec tests for your servers ○ serverspec.org ● Bats ○ Bash Automated Testing System ○ Bash script with special syntax for defining test cases

Slide 26

Slide 26 text

SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end

Slide 27

Slide 27 text

BATS #!/usr/bin/env bats @test "httpd should be running" { run service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }

Slide 28

Slide 28 text

RUNNING TESTS ● Test Kitchen ○ kitchen.ci ● Vagrant plugins ○ github.com/jvoorhis/vagrant-serverspec ● Serverspec SSH ● … choose your own adventure!

Slide 29

Slide 29 text

RUNNING TESTS AS SENSU CHECKS ● Use the Sensu Serverspec check plugin ○ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' ● Run Bats scripts

Slide 30

Slide 30 text

Chef & Sensu In practice.

Slide 31

Slide 31 text

SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu ● Recipes to install & manage services ● LWRPs for configuring handlers, checks, etc. ● Intended to be used by wrapper cookbook ○ e.g. recipe['monitor::haproxy']

Slide 32

Slide 32 text

EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe' command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end

Slide 33

Slide 33 text

USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check = data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end

Slide 34

Slide 34 text

SENSU SERVERSPEC CHECKS # recipe['monitor::serverspec'] sensu_gem 'sensu-plugins-serverspec' sensu_check 'serverspec' do command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb ' standalone true interval 30 end # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures

Slide 35

Slide 35 text

BASIC IaC WORKFLOW TEST ≈ MONITOR

Slide 36

Slide 36 text

Questions? Sean Porter (@PorterTech) sensuapp.org