House of Salt: Extending and leveraging SaltStack by Marco Slaviero and Max Mclaughlin

Once upon a time, sysadmins had cute naming schemes for their boxen, and it was good. Suddenly, virtualisation! And the cute names withered as herds of servers were born. Infrastructure automation and management became a thing. Helper software was released (without which a nine person company could not hope to manage 400-odd servers.) And it was good again, until the edges showed. Then we started to build.

In this talk we’ll discuss the challenges encountered in managing hundreds of servers delivering a single product on AWS. Relying on SaltStack for deployment, management and diagnostics, we’ve automated away painful steps and reduced our attack surface. SaltStack handles configuration and package management well, but is capable of much more once you start to customise it.

Along the way we’ll show examples of custom SaltStack modules, runners, and event handlers. We’ll describe threats and show how SaltStack helps us limit their exposure and impact. We’ll catalogue the lessons we’ve faced in two years of running SaltStack in our production environment.

We won't rehash content from previous talks, and novice users will get to see how the tool can be stretched. If you’re wanting to get more out of SaltStack than the builtin modules allow for, this talk is for you.

Transcript

1. House of Salt Extending and leveraging SaltStack {max,marco}@thinkst.com
2. Who are we? Canary
3. Other players
4. Execution Modules Runners Reactors
5. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
6. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
7. • LotR characters? • Japanese Artists? • Elements? • Movies?
8. Our problem
9. > 400 machines
10. > 400 machines No time for silly names!
12. DEPLOYING HIGHLY-AVAILABLE ARCHITECTURE WITH A PINCH OF SALT Petrus Theron, PyCon ZA 2013 HAND ME THE SALT WHILE I READ MY NEWS Johann du Toit, PyCon ZA 2014
13. Basics • Master / Minions • Mostly agent-based • Uses pub/sub to distribute jobs and collect results • States • Targeting Master Minion1 Minion2 Minion3 ZeroMQ
14. Master Minion1 Minion2 Minion3 ZeroMQ
15. Master Minion1 Minion2 Minion3 ZeroMQ Job
16. Master Minion1 Minion2 Minion3 ZeroMQ Job
17. Master Minion1 Minion2 Minion3 ZeroMQ Job Job Job
18. Master Minion1 Minion2 Minion3 ZeroMQ
19. Master Minion1 Minion2 Minion3 ZeroMQ Results Results Results
20. Master Minion1 Minion2 Minion3 ZeroMQ Results Results Results
21. Master Minion1 Minion2 Minion3 ZeroMQResults Results Results
22. Master Minion1 Minion2 Minion3 ZeroMQResults Results Results
23. Master Minion1 Minion2 Minion3 ZeroMQ Results Results Results
24. Configuration defined in state files /srv/salt/*.sls • Root is top.sls • Nestable • First passed through Jinja2 • Then passed through YAML loader
25. /srv/salt/top.sls: base: '*': - webserver /srv/salt/webserver.sls: redis-server: service.running: - enable: True - name: redis-server flask: cmd.run: - name: "gunicorn -D -w 15 -b 0.0.0.0:5000 app:app" - cwd: /code
26. How we use Salt
27. Deploying states
28. Deploying states
29. Deploying states Master Filerserver 
 backends
30. Deploying states Master Filerserver 
 backends Commit + push
31. Deploying states Master Filerserver 
 backends Commit + push
32. Deploying states Master Filerserver 
 backends Commit + push
33. Deploying states Master Filerserver 
 backends Commit + push Call highstate
34. Deploying states Master Filerserver 
 backends Minion1 Minion2 Minion3 Commit + push Call highstate
35. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
36. A master, minions and management
37. Demo 1: Basic Salt setup
38. Demo 1: Basic Salt setup
39. Making changes to states
40. Demo 2: Making changes
41. Demo 2: Making changes
42. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
43. Bootstrapping Salt https://en.wikipedia.org/wiki/Roll-your-own_cigarette#/media/File:Ba-9may-1997-sergievsky-3.jpg
44. Bootstrapping Salt
45. Bootstrapping Salt http://crobypatterns.com/wp-content/uploads/2015/10/12.png
46. Upgrading Salt • We use the PPA rather than distro repo • Ensure both master and minions are on same version
47. Failures • Broken states • Broken extensions (modules / runners / reactors) • 100’s of servers trying to pull from a single git-repo • Network issues preventing packages from downloading • Minions in different regions from master mean EC2 security groups fill up with allow rules • Doesn’t always restart uwsgi correctly when using upstart restart
48. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
49. Salt module types Execution modules Grain modules Runner modules Returner modules State modules Renderer modules Pillar modules SDB modules Outputter modules Cloud modules Beacons External authentication modules Wheel modules Proxy minion modules Engines The Master Tops Roster modules Queue modules pkgdb and pkgfile modules Reactor modules Extending SaltStack, Joseph Hall
50. Exec modules
51. Master Minion2 Execution Module Execution Module Exec modules
52. Master Minion2 Execution Module Execution Module sync_all Exec modules
53. Master Minion2 Execution Module Execution Module Exec modules
54. Master Minion2 Execution Module Execution Module call module.function Exec modules
55. Master Minion2 Execution Module Execution Module call module.function Exec modules
56. Master Minion2 Execution Module Execution Module call module.function Exec modules
57. Master Minion2 Execution Module Execution Module call module.function Exec modules
58. Master Minion2 Execution Module Execution Module call module.function Exec modules
59. Custom exec modules • Master: /srv/salt/_modules/thinkst.py • Distributed to minions with salt '*' saltutil.sync_modules • Called with salt '*' thinkst.some_method • Documentation viewed with salt 'minion' -d thinkst
60. thinkst.py import salt def some_function(): ''' A function that returns something. Example:: salt '*' thinkst.some_function ''' return "I ran."
61. Demo 3: Basic modules
62. Demo 3: Basic modules
63. def myfunc(arg1, optarg1=“”)
64. Demo 4: Method parameters
65. Demo 4: Method parameters
66. Configuration management that’s extensible
67. Directly interacting with data (bad) import salt import redis def set_hits(val=None): r = redis.Redis(host='localhost', port=6379) return r.set('hits', val)
68. But what about reusing parts of your app?
69. Obvious answer: massage the imports
70. salt/thinkst.py: import salt import sys sys.path.insert(0, '/testapp') def set_hits(val=None): import queries return queries.set_hits(val) testapp/queries.py: def set_hits(hits): #modify database
71. Demo 5: Massaging imports
72. Demo 5: Massaging imports
73. Module uses • Data gathering console_commit console_list_users • Feature enablement change_setting disconnects_enable • Ops send_weekly_report update_device dump_logs • Security console_create_sso_token
74. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
75. Runners Or how to aggregate data from multiple minions
76. Runners
77. Runners
78. Master Runner Runners
79. Master Runner Runner.run Runners
80. Master Minion1 Minion2 Minion3 Runner Runner.run Runners
81. Master Minion1 Minion2 Minion3 Runner Runner.run Runners
82. Master Minion1 Minion2 Minion3 Runner Runner.run Runners
83. Master Minion1 Minion2 Minion3 Runner Runner.run Runners
84. Aggregating Minion Responses On the Master: client = salt.client.LocalClient(__opts__['conf_file']) minion_resps = client.cmd('*', 'thinkst.hits2', timeout=1) for minion in minion_resps: #aggregate results
85. Demo 6: Runner
86. Demo 6: Runner
87. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
88. Reactors Or how to write event handlers
89. Events • Minion startup and shutdown • Job creation • Result returned • Plus a ton more
90. Reactors
91. Master Minion1 Reactor Reactors
92. Master Minion1 Reactor Restart Reactors
93. Master Minion1 Reactor Restart Start Event Reactors
94. Master Minion1 Reactor Restart Start Event Post Request Reactors
95. Reactor Setup In the Master config file: reactor: - 'salt/minion/*/start': - /srv/salt/reactor/start.sls /srv/salt/reactor/start.sls: slack_notification: runner.runner.notify: - targets: {{data['id']}}
96. Reactor Setup 2 /srv/salt/_runners/runner.py: def notify(targets=''): ''' Slack notification on minion start. ''' #code to post to slack
97. Demo 7: Reactors
98. Demo 7: Reactors
99. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
100. Output formats • Stick to Python objects • Avoid formatted strings • Makes salt output easier to handler in both runners and shell scripts
101. Using builtin Salt modules • There’s already Redis and file upload modules, reuse them. import salt def dump_database(): __salt__['redis.save']() return __salt__['cp.push']('/var/lib/redis/dump.rdb')
102. Accessing grains and pillar data import salt os = __grains__.get('os', None) feature_foo = __pillar__.get('feature_foo', None)
103. Debugging minion modules • Set “log_level: debug” in /etc/salt/minion #top of module import logging log = logging.getLogger(__name__) def some_function(): log.debug('Trying something’) • tail -f /var/log/salt/minion
104. Debugging with salt-call • On a minion directly edit /var/cache/salt/minion/extmods/thinkst.py • Call with: $ salt-call -d thinkst • Changes blown away with next module push from the master
105. Docstrings
106. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
107. Testing http://tech.trivago.com/2016/10/12/configuration-management---how-to-start-testing-your-salt-formulas/
108. Testing salt states • YAML Syntax • Wrong top file definitions / conditions • Jinja issues in template files • Networking related issues • State ordering • Logical changes (removing a required service)
109. Syntax checking states salt-call state.show_highstate --retcode-passthrough
110. $ docker run \ -v /path/to/your/local/saltstack:/saltstack \ -v /path/to/your/local/saltstack/tests/grains:/etc/salt/grains \ -v /path/to/your/local/saltstack/tests/minion:/etc/salt/minion \ salt-tester salt-call state.show_highstate
111. Testing salt states • YAML Syntax • Wrong top file definitions / conditions • Jinja issues in template files • Networking related issues • State ordering • Logical changes (removing a required service)
112. Preventing broken deploys with git pre-receive hooks
113. Demo 8: Catching broken states
114. Demo 8: Catching broken states
115. Talk Map • Salt background • Basic Salt setup • Notes on usage • Exec Modules • Runners • Reactors • Lessons learned • Testing
116. Wrap-up Execution Modules Runners Reactors
117. Questions? {max,marco}@thinkst.com max / mls @marcoslaviero