House of Salt
Extending and leveraging SaltStack
{max,marco}@thinkst.com
Slide 2
Slide 2 text
Who are we?
Canary
Slide 3
Slide 3 text
Other players
Slide 4
Slide 4 text
Execution Modules
Runners
Reactors
Slide 5
Slide 5 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 6
Slide 6 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 7
Slide 7 text
• LotR characters?
• Japanese Artists?
• Elements?
• Movies?
Slide 8
Slide 8 text
Our problem
Slide 9
Slide 9 text
> 400 machines
Slide 10
Slide 10 text
> 400 machines
No time for silly names!
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
DEPLOYING HIGHLY-AVAILABLE ARCHITECTURE WITH A
PINCH OF SALT
Petrus Theron, PyCon ZA 2013
HAND ME THE SALT WHILE I READ MY NEWS
Johann du Toit, PyCon ZA 2014
Slide 13
Slide 13 text
Basics
• Master / Minions
• Mostly agent-based
• Uses pub/sub to distribute jobs
and collect results
• States
• Targeting
Master
Minion1 Minion2 Minion3
ZeroMQ
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 36
Slide 36 text
A master, minions and management
Slide 37
Slide 37 text
Demo 1: Basic Salt setup
Slide 38
Slide 38 text
Demo 1: Basic Salt setup
Slide 39
Slide 39 text
Making changes to states
Slide 40
Slide 40 text
Demo 2: Making changes
Slide 41
Slide 41 text
Demo 2: Making changes
Slide 42
Slide 42 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 43
Slide 43 text
Bootstrapping Salt
https://en.wikipedia.org/wiki/Roll-your-own_cigarette#/media/File:Ba-9may-1997-sergievsky-3.jpg
Slide 44
Slide 44 text
Bootstrapping Salt
Slide 45
Slide 45 text
Bootstrapping Salt
http://crobypatterns.com/wp-content/uploads/2015/10/12.png
Slide 46
Slide 46 text
Upgrading Salt
• We use the PPA rather than distro repo
• Ensure both master and minions are on same version
Slide 47
Slide 47 text
Failures
• Broken states
• Broken extensions (modules / runners / reactors)
• 100’s of servers trying to pull from a single git-repo
• Network issues preventing packages from downloading
• Minions in different regions from master mean EC2 security groups fill
up with allow rules
• Doesn’t always restart uwsgi correctly when using upstart restart
Slide 48
Slide 48 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 49
Slide 49 text
Salt module types
Execution modules
Grain modules
Runner modules
Returner modules
State modules
Renderer modules
Pillar modules
SDB modules
Outputter modules
Cloud modules
Beacons
External authentication
modules
Wheel modules
Proxy minion modules
Engines
The Master Tops
Roster modules
Queue modules
pkgdb and pkgfile modules
Reactor modules
Extending SaltStack, Joseph Hall
Custom exec modules
• Master:
/srv/salt/_modules/thinkst.py
• Distributed to minions with
salt '*' saltutil.sync_modules
• Called with
salt '*' thinkst.some_method
• Documentation viewed with
salt 'minion' -d thinkst
Slide 60
Slide 60 text
thinkst.py
import salt
def some_function():
'''
A function that returns something.
Example::
salt '*' thinkst.some_function
'''
return "I ran."
Slide 61
Slide 61 text
Demo 3: Basic modules
Slide 62
Slide 62 text
Demo 3: Basic modules
Slide 63
Slide 63 text
def myfunc(arg1, optarg1=“”)
Slide 64
Slide 64 text
Demo 4: Method parameters
Slide 65
Slide 65 text
Demo 4: Method parameters
Slide 66
Slide 66 text
Configuration management that’s
extensible
Slide 67
Slide 67 text
Directly interacting with data (bad)
import salt
import redis
def set_hits(val=None):
r = redis.Redis(host='localhost', port=6379)
return r.set('hits', val)
Aggregating Minion Responses
On the Master:
client = salt.client.LocalClient(__opts__['conf_file'])
minion_resps = client.cmd('*', 'thinkst.hits2', timeout=1)
for minion in minion_resps:
#aggregate results
Slide 85
Slide 85 text
Demo 6: Runner
Slide 86
Slide 86 text
Demo 6: Runner
Slide 87
Slide 87 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 88
Slide 88 text
Reactors
Or how to write event handlers
Slide 89
Slide 89 text
Events
• Minion startup and shutdown
• Job creation
• Result returned
• Plus a ton more
Master
Minion1
Reactor
Restart
Start Event
Post Request
Reactors
Slide 95
Slide 95 text
Reactor Setup
In the Master config file:
reactor:
- 'salt/minion/*/start':
- /srv/salt/reactor/start.sls
/srv/salt/reactor/start.sls:
slack_notification:
runner.runner.notify:
- targets: {{data['id']}}
Slide 96
Slide 96 text
Reactor Setup 2
/srv/salt/_runners/runner.py:
def notify(targets=''):
'''
Slack notification on minion start.
'''
#code to post to slack
Slide 97
Slide 97 text
Demo 7: Reactors
Slide 98
Slide 98 text
Demo 7: Reactors
Slide 99
Slide 99 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 100
Slide 100 text
Output formats
• Stick to Python objects
• Avoid formatted strings
• Makes salt output easier to handler in both runners and shell scripts
Slide 101
Slide 101 text
Using builtin Salt modules
• There’s already Redis and file upload modules, reuse them.
import salt
def dump_database():
__salt__['redis.save']()
return __salt__['cp.push']('/var/lib/redis/dump.rdb')
Slide 102
Slide 102 text
Accessing grains and pillar data
import salt
os = __grains__.get('os', None)
feature_foo = __pillar__.get('feature_foo', None)
Slide 103
Slide 103 text
Debugging minion modules
• Set “log_level: debug” in /etc/salt/minion
#top of module
import logging
log = logging.getLogger(__name__)
def some_function():
log.debug('Trying something’)
• tail -f /var/log/salt/minion
Slide 104
Slide 104 text
Debugging with salt-call
• On a minion directly edit /var/cache/salt/minion/extmods/thinkst.py
• Call with:
$ salt-call -d thinkst
• Changes blown away with next module push from the master
Slide 105
Slide 105 text
Docstrings
Slide 106
Slide 106 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Testing salt states
• YAML Syntax
• Wrong top file definitions / conditions
• Jinja issues in template files
• Networking related issues
• State ordering
• Logical changes (removing a required service)
Slide 109
Slide 109 text
Syntax checking states
salt-call state.show_highstate --retcode-passthrough
Testing salt states
• YAML Syntax
• Wrong top file definitions / conditions
• Jinja issues in template files
• Networking related issues
• State ordering
• Logical changes (removing a required service)
Slide 112
Slide 112 text
Preventing broken deploys with git
pre-receive hooks
Slide 113
Slide 113 text
Demo 8: Catching broken states
Slide 114
Slide 114 text
Demo 8: Catching broken states
Slide 115
Slide 115 text
Talk Map
• Salt background
• Basic Salt setup
• Notes on usage
• Exec Modules
• Runners
• Reactors
• Lessons learned
• Testing
Slide 116
Slide 116 text
Wrap-up
Execution Modules
Runners
Reactors
Slide 117
Slide 117 text
Questions?
{max,marco}@thinkst.com
max / mls
@marcoslaviero