Monitoring Infrastructure with SaltStack

Monitoring Infrastructure with SaltStack

Using SaltStack to monitor servers and applications using SaltStack's modules, returners, and scheduler. See how to link these pieces together to funnel data into apps like Graphite, Librato, and others. The presentation will also take you through the process of building your own custom modules and returners to store application-specific data to any location.

A464f1f39d9d840f9ca156e9abcfd4a9?s=128

Peter Baumgartner

January 29, 2014
Tweet

Transcript

  1. None
  2. Monitoring Infrastructure with SaltStack

  3. Peter Baumgartner Founder, Lincoln Loop ! ! http://lincolnloop.com pete@lincolnloop.com @ipmb

  4. Why Salt?

  5. Familiar Tools Configure servers with Python YAML Jinja2

  6. You had me at hello ♥

  7. There’s More? “SaltStack delivers a dynamic infrastructure communication bus used

    for orchestration, remote execution, configuration management and
 much more.”
  8. Monitoring with Salt? Blame this guy →

  9. Systems Monitoring (my experience)

  10. !10 Wrong Size

  11. !11 Difficult

  12. Old !12 Old

  13. !13 Slow

  14. !14 Insecure

  15. What Do They Have in Common?

  16. Monitoring System Traits 1. Data is collected at regular intervals

    2. Results are shipped to a 
 central datastore 3. A front-end to view the results
  17. None
  18. Salt Can Do That

  19. Collecting Data Salt Modules

  20. Built-in Modules • ps CPU, RAM, and disk • service

    system services • file hashes, last modified, metadata, etc. • cmd run arbitrary commands
  21. # salt-call ps.disk_usage "/" local: ---------- free: 31721873408 percent: 19.9

    total: 42274717696 used: 8405417984
  22. # salt-call service.status redis-server local: True

  23. Or Build Your Own…

  24. # salt-call redis.info local: ---------- aof_current_rewrite_time_sec: -1 aof_enabled: 0 aof_last_bgrewrite_status:

    ok aof_last_rewrite_time_sec: -1 aof_rewrite_in_progress: 0 …
  25. Build Your Own Module 1. Run a command 2. Parse

    the results 3. Return a dictionary
  26. Run a command def info(host='localhost', port='6379', password=None): cmd = ['redis-cli',

    '-h {0}'.format(host), '-p {0}'.format(port), 'info'] if password: cmd.insert(1, '-a {0}'.format(password)) out = __salt__['cmd.run'](' '.join(cmd))
  27. Parse the Results data = {} for line in out.splitlines():

    if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value
  28. Return a Dictionary return data

  29. def info(host='localhost', port='6379', password=None): cmd = ['redis-cli -h {host} -p

    {port}'.format(host=host, port=port), 'info'] if password: cmd.insert(1, '-a {password}'.format(password=password)) out = __salt__['cmd.run'](' '.join(cmd)) data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value return data
  30. Deploy 1. Save file on Salt master:
 <file_root>/_modules/redis.py 2. Deploy

    to minions:
 salt '*' saltutil.sync_modules 3. Run:
 salt '*' redis.info
  31. Shipping to Datastore Salt Returners

  32. Built-in Returners • syslog • mysql, postgres RDBMS • redis,

    mongo, cassandra NoSQL • carbon, sentry Apps
  33. Examples # salt-call --return syslog \ ps.physical_memory_usage ! # salt-call

    --return syslog,carbon \ ps.physical_memory_usage
  34. Or Build Your Own…

  35. Build Your Own Returner 1. Parse a dictionary 2. Ship

    the results
  36. Returner Dictionary { 'fun': 'ps.cpu_percent', 'fun_args': [ 'interval=5', 'per_cpu=True', ],

    'id': 'www.botbot.me', 'return': [0.2, 0.4] 'jid': '20140127204114791430', 'retcode': 0, }
  37. import json, redis def _get_serv(): return redis.Redis( host=__salt__['config.option']('redis.host'), port=__salt__['config.option']('redis.port'), db=__salt__['config.option']('redis.db'))

    def returner(ret): serv = _get_serv() serv.set('{0}:{1}'.format(ret['id'], ret['jid']), json.dumps(ret)) serv.lpush('{0}:{1}'.format(ret['id'], ret['fun']), ret['jid']) serv.sadd('minions', ret['id']) serv.sadd('jids', ret['jid'])
  38. Configuration • Most built-ins use the Salt master/minion config •

    Use pillars instead (easier management) ! __salt__[‘pillar.get']('your_returner:some_key', '')
  39. Example Configuration raven: servers: - http://192.168.1.1 - https://sentry.example.com public_key: yourpublickeygoeshere

    secret_key: yoursecretkeygoeshere project: 1
  40. Handling Requirements Use the __virtual__ function • Sets the public

    name (default = file name) • If False, it is not available • Works for modules and returners
  41. The __virtual__ Function try: import redis HAS_REDIS = True except

    ImportError: HAS_REDIS = False ! def __virtual__(): if not HAS_REDIS: return False return 'redis'
  42. Deploy 1. Save file on Salt master:
 <file_root>/_returners/redis_return.py 2. Deploy

    to minions:
 salt '*' saltutil.sync_returners 3. Run:
 salt '*' --return redis test.ping
  43. Scheduling Execution Salt Scheduler

  44. Salt Scheduler It’s easier than cron

  45. Salt Scheduler Glues together: • A module • A returner

    • An execution interval
  46. The schedule Pillar schedule: virt_mem_use: function: ps.virtual_memory_usage minutes: 5 returner:

    redis
  47. The schedule Pillar schedule: disk_use: function: ps.disk_usage args: - "/"

    minutes: 5 returner: redis
  48. The schedule Pillar schedule: vmm_usage: function: ps.virtual_memory_usage minutes: 5 returner:

    redis disk_usage: function: ps.disk_usage args: - "/" minutes: 5 returner: redis
  49. Salt Scheduler That’s it.

  50. What Can Salt Do? • Configure minions (Pillars and States)

    • Gather data (Modules) • Ship to datastore (Returners) • Schedule the process at regular intervals (Scheduler)
  51. Returner Modules Minions ??

  52. What about a UI?

  53. UI Options • Graphite (built-in returner) • Hosted: Librato, StatHat,

    etc.
 https://github.com/lincolnloop/salt-stats • Build your own
 https://github.com/lincolnloop/salmon

  54. Conclusions Salt-minion is a perfectly capable backend-agnostic monitoring agent.

  55. Conclusions Using salt-minion for monitoring means fewer daemons to manage

    and configure.
  56. Conclusions Help make SaltStack better by contributing your own modules

    and returners.
  57. Questions? http://www.flickr.com/photos/helmutrs/6470847673 http://www.flickr.com/photos/jepoirrier/6166735961 http://www.flickr.com/photos/kplawver/2089275079 http://www.flickr.com/photos/pbyrne/165215403 Peter Baumgartner pete@lincolnloop.com @ipmb