Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Monitoring Infrastructure with SaltStack

Slide 3

Slide 3 text

Peter Baumgartner Founder, Lincoln Loop ! ! http://lincolnloop.com [email protected] @ipmb

Slide 4

Slide 4 text

Why Salt?

Slide 5

Slide 5 text

Familiar Tools Configure servers with Python YAML Jinja2

Slide 6

Slide 6 text

You had me at hello ♥

Slide 7

Slide 7 text

There’s More? “SaltStack delivers a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management and
 much more.”

Slide 8

Slide 8 text

Monitoring with Salt? Blame this guy →

Slide 9

Slide 9 text

Systems Monitoring (my experience)

Slide 10

Slide 10 text

!10 Wrong Size

Slide 11

Slide 11 text

!11 Difficult

Slide 12

Slide 12 text

Old !12 Old

Slide 13

Slide 13 text

!13 Slow

Slide 14

Slide 14 text

!14 Insecure

Slide 15

Slide 15 text

What Do They Have in Common?

Slide 16

Slide 16 text

Monitoring System Traits 1. Data is collected at regular intervals 2. Results are shipped to a 
 central datastore 3. A front-end to view the results

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Salt Can Do That

Slide 19

Slide 19 text

Collecting Data Salt Modules

Slide 20

Slide 20 text

Built-in Modules • ps CPU, RAM, and disk • service system services • file hashes, last modified, metadata, etc. • cmd run arbitrary commands

Slide 21

Slide 21 text

# salt-call ps.disk_usage "/" local: ---------- free: 31721873408 percent: 19.9 total: 42274717696 used: 8405417984

Slide 22

Slide 22 text

# salt-call service.status redis-server local: True

Slide 23

Slide 23 text

Or Build Your Own…

Slide 24

Slide 24 text

# salt-call redis.info local: ---------- aof_current_rewrite_time_sec: -1 aof_enabled: 0 aof_last_bgrewrite_status: ok aof_last_rewrite_time_sec: -1 aof_rewrite_in_progress: 0 …

Slide 25

Slide 25 text

Build Your Own Module 1. Run a command 2. Parse the results 3. Return a dictionary

Slide 26

Slide 26 text

Run a command def info(host='localhost', port='6379', password=None): cmd = ['redis-cli', '-h {0}'.format(host), '-p {0}'.format(port), 'info'] if password: cmd.insert(1, '-a {0}'.format(password)) out = __salt__['cmd.run'](' '.join(cmd))

Slide 27

Slide 27 text

Parse the Results data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value

Slide 28

Slide 28 text

Return a Dictionary return data

Slide 29

Slide 29 text

def info(host='localhost', port='6379', password=None): cmd = ['redis-cli -h {host} -p {port}'.format(host=host, port=port), 'info'] if password: cmd.insert(1, '-a {password}'.format(password=password)) out = __salt__['cmd.run'](' '.join(cmd)) data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value return data

Slide 30

Slide 30 text

Deploy 1. Save file on Salt master:
 /_modules/redis.py 2. Deploy to minions:
 salt '*' saltutil.sync_modules 3. Run:
 salt '*' redis.info

Slide 31

Slide 31 text

Shipping to Datastore Salt Returners

Slide 32

Slide 32 text

Built-in Returners • syslog • mysql, postgres RDBMS • redis, mongo, cassandra NoSQL • carbon, sentry Apps

Slide 33

Slide 33 text

Examples # salt-call --return syslog \ ps.physical_memory_usage ! # salt-call --return syslog,carbon \ ps.physical_memory_usage

Slide 34

Slide 34 text

Or Build Your Own…

Slide 35

Slide 35 text

Build Your Own Returner 1. Parse a dictionary 2. Ship the results

Slide 36

Slide 36 text

Returner Dictionary { 'fun': 'ps.cpu_percent', 'fun_args': [ 'interval=5', 'per_cpu=True', ], 'id': 'www.botbot.me', 'return': [0.2, 0.4] 'jid': '20140127204114791430', 'retcode': 0, }

Slide 37

Slide 37 text

import json, redis def _get_serv(): return redis.Redis( host=__salt__['config.option']('redis.host'), port=__salt__['config.option']('redis.port'), db=__salt__['config.option']('redis.db')) def returner(ret): serv = _get_serv() serv.set('{0}:{1}'.format(ret['id'], ret['jid']), json.dumps(ret)) serv.lpush('{0}:{1}'.format(ret['id'], ret['fun']), ret['jid']) serv.sadd('minions', ret['id']) serv.sadd('jids', ret['jid'])

Slide 38

Slide 38 text

Configuration • Most built-ins use the Salt master/minion config • Use pillars instead (easier management) ! __salt__[‘pillar.get']('your_returner:some_key', '')

Slide 39

Slide 39 text

Example Configuration raven: servers: - http://192.168.1.1 - https://sentry.example.com public_key: yourpublickeygoeshere secret_key: yoursecretkeygoeshere project: 1

Slide 40

Slide 40 text

Handling Requirements Use the __virtual__ function • Sets the public name (default = file name) • If False, it is not available • Works for modules and returners

Slide 41

Slide 41 text

The __virtual__ Function try: import redis HAS_REDIS = True except ImportError: HAS_REDIS = False ! def __virtual__(): if not HAS_REDIS: return False return 'redis'

Slide 42

Slide 42 text

Deploy 1. Save file on Salt master:
 /_returners/redis_return.py 2. Deploy to minions:
 salt '*' saltutil.sync_returners 3. Run:
 salt '*' --return redis test.ping

Slide 43

Slide 43 text

Scheduling Execution Salt Scheduler

Slide 44

Slide 44 text

Salt Scheduler It’s easier than cron

Slide 45

Slide 45 text

Salt Scheduler Glues together: • A module • A returner • An execution interval

Slide 46

Slide 46 text

The schedule Pillar schedule: virt_mem_use: function: ps.virtual_memory_usage minutes: 5 returner: redis

Slide 47

Slide 47 text

The schedule Pillar schedule: disk_use: function: ps.disk_usage args: - "/" minutes: 5 returner: redis

Slide 48

Slide 48 text

The schedule Pillar schedule: vmm_usage: function: ps.virtual_memory_usage minutes: 5 returner: redis disk_usage: function: ps.disk_usage args: - "/" minutes: 5 returner: redis

Slide 49

Slide 49 text

Salt Scheduler That’s it.

Slide 50

Slide 50 text

What Can Salt Do? • Configure minions (Pillars and States) • Gather data (Modules) • Ship to datastore (Returners) • Schedule the process at regular intervals (Scheduler)

Slide 51

Slide 51 text

Returner Modules Minions ??

Slide 52

Slide 52 text

What about a UI?

Slide 53

Slide 53 text

UI Options • Graphite (built-in returner) • Hosted: Librato, StatHat, etc.
 https://github.com/lincolnloop/salt-stats • Build your own
 https://github.com/lincolnloop/salmon


Slide 54

Slide 54 text

Conclusions Salt-minion is a perfectly capable backend-agnostic monitoring agent.

Slide 55

Slide 55 text

Conclusions Using salt-minion for monitoring means fewer daemons to manage and configure.

Slide 56

Slide 56 text

Conclusions Help make SaltStack better by contributing your own modules and returners.

Slide 57

Slide 57 text

Questions? http://www.flickr.com/photos/helmutrs/6470847673 http://www.flickr.com/photos/jepoirrier/6166735961 http://www.flickr.com/photos/kplawver/2089275079 http://www.flickr.com/photos/pbyrne/165215403 Peter Baumgartner [email protected] @ipmb