automating infrastructure using salt . Siddhant Goel October 30, 2015 

“Every of devops team is have at least 3 shell script which is critical and is not in source control or backup” - @DEVOPS_BORAT 2 

this talk . • Server/Cluster provisioning • Common problems • Automating common tasks and reducing manual work • Solution requirements • SaltStack 3 

provisioning servers . • Installing packages • Creating/managing users • Deploying SSH keys • Application deployment • Package versions • Setting up cron jobs • Managing databases • Deploying sensitive information • Make it “usable” 4 

defining clusters . • Define application groups (application servers, database servers, load balancers, etc.) • Which packages go to which machines • Configuration files/scripts • Manage database/application passwords 5 

common problems . • Manual means error-prone • Bus factor • Cannot see the server-state • Is this cron running on server X? • How is this application started? • How are the passwords managed? • Figure out on which of the 1000 servers something failed, and why 6 

requirements from a solution . • Automated • Declarative • Secure • Simple • Explicit 7 

automation . • Shell scripts • No • Fabric • Lightweight abstraction over ssh • Good for automating lightweight tasks • Does not work for defining state • +Concurrent, +Fast, +Testable • -Error handling, -Idempotence 8 

saltstack . 

introduction . • Configuration management system built using Python, ZeroMQ • Like Chef, but in Python • Functions • Automates common tasks • Brings servers into the specified state • Execute commands remotely on hosts • Uses YAML for specifying configuration • Declarative logic • Keeps data separate from state • Secure by default 10 

components . Image from https://xmission.com/blog/2014/04/08/the-salt-of-xmission 11 

components . • Master • Minions • States • Pillars • Grains • Formulas • State Tree 12 

usage . • Label servers into groups they belong to • Define states/data per group • salt ‘*’ state.highstate 13 

syntax - state . /srv/salt/top.sls base : ‘ * ’ : − users /srv/salt/users.sls mango : # ID user : # State − managed # Function 14 

syntax - pillar . /srv/pillar/top.sls base : ‘ * ’ : − users /srv/pillar/users.sls users : mango : hunter123 15 

examples . 

installation/service . nginx : pkg . i n s t a l l e d : [ ] service . running : − watch : − pkg : nginx − f i l e : / etc /nginx/nginx . conf 17 

managed files . / etc /nginx/nginx . conf : f i l e . managed : − source : s a l t :// nginx/ f i l e s /nginx . conf − user : root − group : root − mode : 644 18 

users . mango : user . present : − shell : /bin/bash − home : /home/mango ssh_auth . present : − name : p i l l a r . users . mango . pub_key − user : mango 19 

git . /home/tornado/app : f i l e . directory : − user : tornado − group : tornado − dir_mode : 755 − recurse : − user − mode https :// github . com/tornadoweb/tornado : g i t . l a t e s t : − target : /home/tornado/app − rev : master − user : tornado − force_checkout : True 20 

virtualenv . /home/tornado/env : virtualenv . managed : − system_site_packages : False − requirements : s a l t :// cron/requirements . t x t 21 

crontab . /path/to/python /path/to/ s c r i p t . py : cron . present : − i d e n t i f i e r : CRON_SCRIPT_IDENTIFIER − user : tornado − hour : 4 − minute : 0 22 

postgresql . tornado_db_user : postgres_user . present : − name : p i l l a r . tornado_db . user − password : p i l l a r . tornado_db . password − require : − service : postgresql tornado_db_database : postgres_database . present : − name : p i l l a r . tornado_db . database − owner : p i l l a r . tornado_db . user − encoding : UTF8 − lc_ctype : en_US . UTF8 − l c _ c o l l a t e : en_US . UTF8 23 

configuration . State /path/to/ settings . py : f i l e . managed : − source : s a l t :// tornado/ f i l e s / settings . py − user : tornado − group : tornado /path/to/settings.py DATABASE = { ’ database ’ : { { p i l l a r . tornado_db . database } } , ’ user ’ : { { p i l l a r . tornado_db . user } } , ’ password ’ : { { p i l l a r . tornado_db . password } } , } 24 

salt modules . • Tons of modules included in the standard distribution • fileservers • logging • mysql/postgresql • ... • Modules written in Python • Easy to add new ones • Also available for configuring hadoop master/slaves 25 

advantages . • Reduces infrastructure changes to simple pull requests • Spreads knowledge/responsibility amongst all team members • Reduces manual (error-prone) work • Documents the entire system in simple state files 26 

Thank You! 27