雲端組態佈署自動化 SaltStack Cheng-Lung Sung (clsung@)

Worked at Cloud Service Infrastructure, Studio Engineering, HTC

Funfit -樂動趣

Working at Tensorflow/CSI, HealthCare, HTC

Tricorder Ref: http://www.appledaily.com.tw/realtimenews/article/new/20161109/985048/

Are you Dev, Ops, or DevOps?

“DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” - theagileadmin.com/what-is-devops/

DevOps: Infrastructure as Code

DevOps guideline ● Elements ○ Automated ○ Idempotent, also implies ○ Repeatable ● Avoid SPOF (single point of failure) ● Release fast, release often ○ Testing ○ Continuous Integration ○ Continuous Deployment

What is SaltStack ● Remote execution framework ● Configuration management and orchestration system ● Cloud Provisioning tool ● Written in Python ● Communicate with 0MQ ○ Port: 4505, 4506

What we will not talk about ● Comparison with CM friends ○ Ansible (Python) ○ Chef (Ruby) ○ Puppet (Ruby) ● Comparison with Container-related tools ○ Docker, rxt ○ Kubernetes, Docker-swarm, Mesos ● Some salts ○ Salt-proxy, Salt Orchestration, Salt Virt, Salt Runner ○ RAEF Transport (replacement of ZMQ) ○ Thorium (new Salt Reactor)

Why I choose Saltstack

Why I choose Saltstack ● Written in Python(!) ● Been using Ansible since in 2014~ ○ Always find difficulty in using playbooks ● Also tried Saltstack in 2014 ○ Most modules are already Ansibled ■ “If it ain’t broke, don’t fix it” ● New project in 2016 ○ From scratch, bootstrapping…. ○ Faster than Ansible (thanks to ZeroMQ)

Why I choose Saltstack

Saltstack Introduction

Saltstack is a remote execution framework ● Master/Agent ○ Master commands agents (minions) ● Communication: ○ Publisher (port 4505) ○ ReqServer (port 4506) ● Secure ○ Salt Minion key authentication ○ Communication is ■ AES-encrypted ■ CBC - 192 bits (why 192 instead of 256?) ● Access Control

Saltstack Installation ● Fetch the latest script ○ % curl -L https://bootstrap.saltstack.com -o install_salt.sh ● Salt-master (Master/Agent) ○ % sh install_salt.sh -P -M ● Salt-minion (Master/Agent, Masterless) ○ % sh install_salt.sh -P ● Salt-syndic (Master/Agent) ○ % sh install_salt.sh -P -M -S ● Salt-ssh (Agentless) ○ % # noop

Salt commands ● salt-[master|minion|syndic] ● salt-key ● salt-cloud ● salt ● salt-ssh (agentless) ● salt-run ● salt-call ● salt-proxy

Remote Execution

● Package installation ○ % salt '*' pkg.install nginx ○ % salt '*' service.start nginx ● Direct command ○ % salt '*' cmd.run 'whoami' ● States ○ % salt '*' state.apply ○ % salt '*' state.apply mosh ● Modules ○ % salt '*' saltutil.refresh_pillar Salt root@master:~$ salt

Salt

Salt

Salt

Salt-Call ● Run module functions locally ○ Instead remote executed from the master ● Created for debug/troubleshooting/testing % salt-call state.highstate test=True % salt-call state.sls foo.bar test=True % salt-call state.show_sls foo.bar ● Also known as masterless minion (Standalone)

Salt Masterless ● Modify /etc/salt/minion ○ file_client: local ● Specify in CLI ○ ‘--local’ with salt-call % salt-call --local state.apply ● Always can be test % salt-call --local state.apply test=True

Salt-SSH ● Agentless ○ Do not require a daemon (minion) process ● Roster file: salt22: host: csi-salt2 user: csiuser ● Remote execution via ssh

Salt-Cloud ● Provision systems on cloud providers, hypervisors

Salt-Cloud (AWS) ec2-us-west-2-public: minion: master: ip-172-31-30-32 id: 'AWS id' key: 'AWS key+iIP21RaHNBq1DOMaQMkOAgF' private_key: /etc/salt/secret keyname: csiuser-dl-oregon ssh_interface: public_ips securitygroup: security location: us-west-2 iam_profile: arn:iam_role driver: ec2 del_root_vol_on_destroy: True del_all_vols_on_destroy: True rename_on_destroy: True % salt-cloud -p gpu gpuwork1 % salt-cloud -d gpuwork1 gpu: image: ami-d732f0b7 size: g2.2xlarge location: us-west-2 network: default grains: role: gpu tags: {'Environment', 'dev'} del_root_vol_on_destroy: True block_device_mappings: - DeviceName: /dev/sda1 Ebs.VolumeSize: 120 Ebs.VolumeType: gp2 del_all_vol_on_destroy: True ssh_username: ubuntu make_master: False sync_after_install: grains provider: ec2-us-west-2-public

Salt-Syndic ● Also need salt-master ○ And salt-syndic ● (optionally) ○ salt-minion ● Relay command from ○ salt-master to salt-syndic’s salt-master ● Aggregate result from ○ salt-minion(s) to salt-master

Configuration Management

Configuration / Resource Management tools ● Master/Agent v.s. Standalone ● Standalone: ○ Small / simple deployments ○ Dev / Testing ● Master/Agent ○ Large / complicated deployments ○ Centralization ○ Parallel operations ● Agentless ○ Security concern?

Ref: http://www.slidesshre.net/arthurlutz/debian-meetup-nantes-2015-salt-pour-grer-de-nombreux-serveurs-debian

SaLt State (SLS)

Salt SLS structure Ref: http://blog.infracloud.io/saltstack-tutorial-part-2/

State file format Ref: https://docs.saltstack.com/en/getstarted/config/functions.html

State formula execution order To describe: salt '*' state.show_sls sls1[,sls2,...]

State External formulas (contrib) ● Pre-written salt states ○ Mostly: https://github.com/saltstack-formulas ● Added as GitFS Remote ○ In /etc/salt/master gitfs_remotes: - https://github.com/saltstack-formulas/consul-formula ● Added manually ○ Clone in /salt/formulas (also specified in file_root) file_roots: base: - /srv/salt - /srv/formulas/datadog-formula - /srv/formulas/etcd-formula - /srv/formulas/docker-formula

Grains ● Static information ○ At least not manually changed % salt '*' grains.ls % salt '*' grains.items ● Distinguishable in top.sls base: 'consul*': - consul 'role:nogpu': - match: grain - mxnet.no_gpu 'role:gpu': - match: grain - mxnet.with_gpu - torch

Pillar example: consul.sls consul: service: true url: http://loadbalancer.us-east-1.elb.amazonaws.com:8500 # for master config: retry_interval: 15s {% if salt['grains.get']('ip4_interfaces:eth0', None) %} bind_addr: {{ grains['ip4_interfaces']['eth0'][0] }} {% endif %} start_join: ["172.31.16.89", "172.31.32.89", "172.31.0.89"] {% if grains['id'].startswith('consul')%} client_addr: "0.0.0.0" acl_down_policy: "extend-cache" {% if grains['id'] == 'consulone'%} bootstrap: true {% else %} bootstrap: false {% endif %} server: true {% endif %} datacenter: "main" ui: true enable_syslog: true encrypt: "keyencrypted" log_level: info data_dir: /var/consul

Service Discovery

Event and Reactor

Event salt-run state.event pretty=True

Reactor ● /etc/salt/master.d/reactor.conf reactor: # Salt master config section "reactor" - 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts - 'salt/cloud/*/cache_node_new: # Match tag “salt/cloud/*/cache_node_new” - /srv/reactor/autoscale.sls # Things to check if auto scale required - 'csi/grape/metric/gpu': # React to gpu events - salt://reactor/grape_gpu.sls # what states to apply ● /srv/reactor/start.sls {% if salt['grains.has_value']('ec2_tags') %} update_ec2_name: cmd.run: - name: /opt/update_ec2_name.sh - require: - file: /opt/update_ec2_name.sh {% endif %}

Event and Reactor

Beacons Ref: http://gaming.stackexchange.com/questions/227579/what-blocks-can-you-use-to-put-under-a-beacon

Beacons - fire events to reactor ● /etc/salt/master.d/beacons.conf beacons: inotify: /csi/specific.log: mask: - modify /csi/log: mask: - open - create auto_add: True disable_during_state_run: True load: 5m: - 0.0 - 9.0 onchangeonly: True ● Modules to ○ File system changes ○ System load ○ Service status ○ Shell activity, (e.g. login) ○ Network and disk usage

Event and Reactor beacons

Salt Development and Testing

“Create and configure lightweight, reproducible, and portable development environments.” - www.vagrantup.com

Testing Salt with Vagrant ● Fast local testing ● Provision VMs ● Disposable environment ● Built-in Salt provisioner ● Private/Public Network # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2”) do |config| config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" config.vm.box = "ubuntu/precise64" config.vm.network "public_network", :bridge => "eth0" # config.vm.network "private_network", :ip => "192.168.31.2" config.vm.hostname = "dev_walker1" config.vm.synced_folder "/srv/sls/", "/srv/" config.vm.provision :salt do |salt| salt.pillar({"is_vagrant" => true}) salt.minion_config = "salt/minion" salt.colorize = true salt.run_highstate = true end end

Testing Salt with Docker ● Fast local testing ● Focus on Salt states ● Disposable environment ● Masterless FROM ubuntu:14.04 MAINTAINER Cheng-Lung Sung RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository -y ppa:saltstack/salt RUN apt-get update && apt-get -y install salt-minion RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* VOLUME ["/etc/salt", "/srv/salt", "/srv/pillar", "/srv/formulas"] CMD salt-call --local state.apply % docker build -t test_docker % docker run --rm -ti -v /path/to/etc/salt:/etc/salt \ -v /path/to/salt:/srv/salt \ -v /path/to/pillar:/srv/pillar \ -v /path/to/formulas:/srv/formulas \ test_docker

Discussion (My Salt ongoing list) ● AutoScaling ○ Use Reactor ■ https://github.com/saltstack-formulas/ec2-autoscale-reactor ● Continuous Deployment ○ Salt-API ○ Canary or BlueGreen Deployment ● Monitoring system ○ Dashboard ○ Alert

Dashboard

Thank you! And Q & A