Monitoring Infrastructure with SaltStack

Using SaltStack to monitor servers and applications using SaltStack's modules, returners, and scheduler. See how to link these pieces together to funnel data into apps like Graphite, Librato, and others. The presentation will also take you through the process of building your own custom modules and returners to store application-specific data to any location.

Transcript

2. Monitoring Infrastructure with SaltStack
3. Peter Baumgartner Founder, Lincoln Loop ! ! http://lincolnloop.com pete@lincolnloop.com @ipmb
4. Why Salt?
5. Familiar Tools Configure servers with Python YAML Jinja2
6. You had me at hello ♥
7. There’s More? “SaltStack delivers a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management and
 much more.”
8. Monitoring with Salt? Blame this guy →
9. Systems Monitoring (my experience)
10. !10 Wrong Size
11. !11 Difficult
12. Old !12 Old
13. !13 Slow
14. !14 Insecure
15. What Do They Have in Common?
16. Monitoring System Traits 1. Data is collected at regular intervals 2. Results are shipped to a 
 central datastore 3. A front-end to view the results
18. Salt Can Do That
19. Collecting Data Salt Modules
20. Built-in Modules • ps CPU, RAM, and disk • service system services • file hashes, last modified, metadata, etc. • cmd run arbitrary commands
21. # salt-call ps.disk_usage "/" local: ---------- free: 31721873408 percent: 19.9 total: 42274717696 used: 8405417984
22. # salt-call service.status redis-server local: True
23. Or Build Your Own…
24. # salt-call redis.info local: ---------- aof_current_rewrite_time_sec: -1 aof_enabled: 0 aof_last_bgrewrite_status: ok aof_last_rewrite_time_sec: -1 aof_rewrite_in_progress: 0 …
25. Build Your Own Module 1. Run a command 2. Parse the results 3. Return a dictionary
26. Run a command def info(host='localhost', port='6379', password=None): cmd = ['redis-cli', '-h {0}'.format(host), '-p {0}'.format(port), 'info'] if password: cmd.insert(1, '-a {0}'.format(password)) out = __salt__['cmd.run'](' '.join(cmd))
27. Parse the Results data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value
28. Return a Dictionary return data
29. def info(host='localhost', port='6379', password=None): cmd = ['redis-cli -h {host} -p {port}'.format(host=host, port=port), 'info'] if password: cmd.insert(1, '-a {password}'.format(password=password)) out = __salt__['cmd.run'](' '.join(cmd)) data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value return data
30. Deploy 1. Save file on Salt master:
 <file_root>/_modules/redis.py 2. Deploy to minions:
 salt '*' saltutil.sync_modules 3. Run:
 salt '*' redis.info
31. Shipping to Datastore Salt Returners
32. Built-in Returners • syslog • mysql, postgres RDBMS • redis, mongo, cassandra NoSQL • carbon, sentry Apps
33. Examples # salt-call --return syslog \ ps.physical_memory_usage ! # salt-call --return syslog,carbon \ ps.physical_memory_usage
34. Or Build Your Own…
35. Build Your Own Returner 1. Parse a dictionary 2. Ship the results
36. Returner Dictionary { 'fun': 'ps.cpu_percent', 'fun_args': [ 'interval=5', 'per_cpu=True', ], 'id': 'www.botbot.me', 'return': [0.2, 0.4] 'jid': '20140127204114791430', 'retcode': 0, }
37. import json, redis def _get_serv(): return redis.Redis( host=__salt__['config.option']('redis.host'), port=__salt__['config.option']('redis.port'), db=__salt__['config.option']('redis.db')) def returner(ret): serv = _get_serv() serv.set('{0}:{1}'.format(ret['id'], ret['jid']), json.dumps(ret)) serv.lpush('{0}:{1}'.format(ret['id'], ret['fun']), ret['jid']) serv.sadd('minions', ret['id']) serv.sadd('jids', ret['jid'])
38. Configuration • Most built-ins use the Salt master/minion config • Use pillars instead (easier management) ! __salt__[‘pillar.get']('your_returner:some_key', '')
39. Example Configuration raven: servers: - http://192.168.1.1 - https://sentry.example.com public_key: yourpublickeygoeshere secret_key: yoursecretkeygoeshere project: 1
40. Handling Requirements Use the __virtual__ function • Sets the public name (default = file name) • If False, it is not available • Works for modules and returners
41. The __virtual__ Function try: import redis HAS_REDIS = True except ImportError: HAS_REDIS = False ! def __virtual__(): if not HAS_REDIS: return False return 'redis'
42. Deploy 1. Save file on Salt master:
 <file_root>/_returners/redis_return.py 2. Deploy to minions:
 salt '*' saltutil.sync_returners 3. Run:
 salt '*' --return redis test.ping
43. Scheduling Execution Salt Scheduler
44. Salt Scheduler It’s easier than cron
45. Salt Scheduler Glues together: • A module • A returner • An execution interval
46. The schedule Pillar schedule: virt_mem_use: function: ps.virtual_memory_usage minutes: 5 returner: redis
47. The schedule Pillar schedule: disk_use: function: ps.disk_usage args: - "/" minutes: 5 returner: redis
48. The schedule Pillar schedule: vmm_usage: function: ps.virtual_memory_usage minutes: 5 returner: redis disk_usage: function: ps.disk_usage args: - "/" minutes: 5 returner: redis
49. Salt Scheduler That’s it.
50. What Can Salt Do? • Configure minions (Pillars and States) • Gather data (Modules) • Ship to datastore (Returners) • Schedule the process at regular intervals (Scheduler)
51. Returner Modules Minions ??
52. What about a UI?
53. UI Options • Graphite (built-in returner) • Hosted: Librato, StatHat, etc.
 https://github.com/lincolnloop/salt-stats • Build your own
 https://github.com/lincolnloop/salmon

54. Conclusions Salt-minion is a perfectly capable backend-agnostic monitoring agent.
55. Conclusions Using salt-minion for monitoring means fewer daemons to manage and configure.
56. Conclusions Help make SaltStack better by contributing your own modules and returners.
57. Questions? http://www.flickr.com/photos/helmutrs/6470847673 http://www.flickr.com/photos/jepoirrier/6166735961 http://www.flickr.com/photos/kplawver/2089275079 http://www.flickr.com/photos/pbyrne/165215403 Peter Baumgartner pete@lincolnloop.com @ipmb