Upgrade to Pro — share decks privately, control downloads, hide ads and more …

House of Salt: Extending and leveraging SaltStack by Marco Slaviero and Max Mclaughlin

Pycon ZA
October 06, 2017

House of Salt: Extending and leveraging SaltStack by Marco Slaviero and Max Mclaughlin

Once upon a time, sysadmins had cute naming schemes for their boxen, and it was good. Suddenly, virtualisation! And the cute names withered as herds of servers were born. Infrastructure automation and management became a thing. Helper software was released (without which a nine person company could not hope to manage 400-odd servers.) And it was good again, until the edges showed. Then we started to build.

In this talk we’ll discuss the challenges encountered in managing hundreds of servers delivering a single product on AWS. Relying on SaltStack for deployment, management and diagnostics, we’ve automated away painful steps and reduced our attack surface. SaltStack handles configuration and package management well, but is capable of much more once you start to customise it.

Along the way we’ll show examples of custom SaltStack modules, runners, and event handlers. We’ll describe threats and show how SaltStack helps us limit their exposure and impact. We’ll catalogue the lessons we’ve faced in two years of running SaltStack in our production environment.

We won't rehash content from previous talks, and novice users will get to see how the tool can be stretched. If you’re wanting to get more out of SaltStack than the builtin modules allow for, this talk is for you.

Pycon ZA

October 06, 2017
Tweet

More Decks by Pycon ZA

Other Decks in Programming

Transcript

  1. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  2. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  3. DEPLOYING HIGHLY-AVAILABLE ARCHITECTURE WITH A PINCH OF SALT Petrus Theron,

    PyCon ZA 2013 HAND ME THE SALT WHILE I READ MY NEWS Johann du Toit, PyCon ZA 2014
  4. Basics • Master / Minions • Mostly agent-based • Uses

    pub/sub to distribute jobs and collect results • States • Targeting Master Minion1 Minion2 Minion3 ZeroMQ
  5. Configuration defined in state files /srv/salt/*.sls • Root is top.sls

    • Nestable • First passed through Jinja2 • Then passed through YAML loader
  6. /srv/salt/top.sls: base: '*': - webserver /srv/salt/webserver.sls: redis-server: service.running: - enable:

    True - name: redis-server flask: cmd.run: - name: "gunicorn -D -w 15 -b 0.0.0.0:5000 app:app" - cwd: /code
  7. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  8. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  9. Upgrading Salt • We use the PPA rather than distro

    repo • Ensure both master and minions are on same version
  10. Failures • Broken states • Broken extensions (modules / runners

    / reactors) • 100’s of servers trying to pull from a single git-repo • Network issues preventing packages from downloading • Minions in different regions from master mean EC2 security groups fill up with allow rules • Doesn’t always restart uwsgi correctly when using upstart restart
  11. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  12. Salt module types Execution modules Grain modules Runner modules Returner

    modules State modules Renderer modules Pillar modules SDB modules Outputter modules Cloud modules Beacons External authentication modules Wheel modules Proxy minion modules Engines The Master Tops Roster modules Queue modules pkgdb and pkgfile modules Reactor modules Extending SaltStack, Joseph Hall
  13. Custom exec modules • Master: /srv/salt/_modules/thinkst.py • Distributed to minions

    with salt '*' saltutil.sync_modules • Called with salt '*' thinkst.some_method • Documentation viewed with salt 'minion' -d thinkst
  14. thinkst.py import salt def some_function(): ''' A function that returns

    something. Example:: salt '*' thinkst.some_function ''' return "I ran."
  15. Directly interacting with data (bad) import salt import redis def

    set_hits(val=None): r = redis.Redis(host='localhost', port=6379) return r.set('hits', val)
  16. salt/thinkst.py: import salt import sys sys.path.insert(0, '/testapp') def set_hits(val=None): import

    queries return queries.set_hits(val) testapp/queries.py: def set_hits(hits): #modify database
  17. Module uses • Data gathering console_commit console_list_users • Feature enablement

    change_setting disconnects_enable • Ops send_weekly_report update_device dump_logs • Security console_create_sso_token
  18. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  19. Aggregating Minion Responses On the Master: client = salt.client.LocalClient(__opts__['conf_file']) minion_resps

    = client.cmd('*', 'thinkst.hits2', timeout=1) for minion in minion_resps: #aggregate results
  20. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  21. Reactor Setup In the Master config file: reactor: - 'salt/minion/*/start':

    - /srv/salt/reactor/start.sls /srv/salt/reactor/start.sls: slack_notification: runner.runner.notify: - targets: {{data['id']}}
  22. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  23. Output formats • Stick to Python objects • Avoid formatted

    strings • Makes salt output easier to handler in both runners and shell scripts
  24. Using builtin Salt modules • There’s already Redis and file

    upload modules, reuse them. import salt def dump_database(): __salt__['redis.save']() return __salt__['cp.push']('/var/lib/redis/dump.rdb')
  25. Accessing grains and pillar data import salt os = __grains__.get('os',

    None) feature_foo = __pillar__.get('feature_foo', None)
  26. Debugging minion modules • Set “log_level: debug” in /etc/salt/minion #top

    of module import logging log = logging.getLogger(__name__) def some_function(): log.debug('Trying something’) • tail -f /var/log/salt/minion
  27. Debugging with salt-call • On a minion directly edit /var/cache/salt/minion/extmods/thinkst.py

    • Call with: $ salt-call -d thinkst • Changes blown away with next module push from the master
  28. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
  29. Testing salt states • YAML Syntax • Wrong top file

    definitions / conditions • Jinja issues in template files • Networking related issues • State ordering • Logical changes (removing a required service)
  30. $ docker run \ -v /path/to/your/local/saltstack:/saltstack \ -v /path/to/your/local/saltstack/tests/grains:/etc/salt/grains \

    -v /path/to/your/local/saltstack/tests/minion:/etc/salt/minion \ salt-tester salt-call state.show_highstate
  31. Testing salt states • YAML Syntax • Wrong top file

    definitions / conditions • Jinja issues in template files • Networking related issues • State ordering • Logical changes (removing a required service)
  32. Talk Map • Salt background • Basic Salt setup •

    Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing