雲端組態佈署自動化 -- SaltStack

SaltStack is a software platform to manage system configuration. It aims at fast and scalable automation of any cloud and application stack, or data center infrastructure. In a common use model, sysadmins and devops use SaltStack to automate configuration management and cloud service orchestration.

This talk will cover SaltStack concepts including execution modules, functions, and configuration management, e.g., states and formulas. In addition to demonstrating a simple cloud service (instances of REST API, cache, database, and service discovery), the speaker will also discuss state-of-the-art cloud service practices.

Transcript

1. 雲端組態佈署自動化 SaltStack Cheng-Lung Sung (clsung@)
2. Worked at Cloud Service Infrastructure, Studio Engineering, HTC
3. Funfit -樂動趣
4. Working at Tensorflow/CSI, HealthCare, HTC
5. Tricorder Ref: http://www.appledaily.com.tw/realtimenews/article/new/20161109/985048/
6. Are you Dev, Ops, or DevOps?
7. “DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” - theagileadmin.com/what-is-devops/
8. DevOps: Infrastructure as Code
9. DevOps guideline • Elements ◦ Automated ◦ Idempotent, also implies ◦ Repeatable • Avoid SPOF (single point of failure) • Release fast, release often ◦ Testing ◦ Continuous Integration ◦ Continuous Deployment
10. What is SaltStack • Remote execution framework • Configuration management and orchestration system • Cloud Provisioning tool • Written in Python • Communicate with 0MQ ◦ Port: 4505, 4506
11. What we will not talk about • Comparison with CM friends ◦ Ansible (Python) ◦ Chef (Ruby) ◦ Puppet (Ruby) • Comparison with Container-related tools ◦ Docker, rxt ◦ Kubernetes, Docker-swarm, Mesos • Some salts ◦ Salt-proxy, Salt Orchestration, Salt Virt, Salt Runner ◦ RAEF Transport (replacement of ZMQ) ◦ Thorium (new Salt Reactor)
12. Why I choose Saltstack
13. Why I choose Saltstack • Written in Python(!) • Been using Ansible since in 2014~ ◦ Always find difficulty in using playbooks • Also tried Saltstack in 2014 ◦ Most modules are already Ansibled ▪ “If it ain’t broke, don’t fix it” • New project in 2016 ◦ From scratch, bootstrapping…. ◦ Faster than Ansible (thanks to ZeroMQ)
14. Why I choose Saltstack
15. Saltstack Introduction
16. Saltstack is a remote execution framework • Master/Agent ◦ Master commands agents (minions) • Communication: ◦ Publisher (port 4505) ◦ ReqServer (port 4506) • Secure ◦ Salt Minion key authentication ◦ Communication is ▪ AES-encrypted ▪ CBC - 192 bits (why 192 instead of 256?) • Access Control
17. Saltstack Installation • Fetch the latest script ◦ % curl -L https://bootstrap.saltstack.com -o install_salt.sh • Salt-master (Master/Agent) ◦ % sh install_salt.sh -P -M • Salt-minion (Master/Agent, Masterless) ◦ % sh install_salt.sh -P • Salt-syndic (Master/Agent) ◦ % sh install_salt.sh -P -M -S • Salt-ssh (Agentless) ◦ % # noop
18. Salt commands • salt-[master|minion|syndic] • salt-key • salt-cloud • salt • salt-ssh (agentless) • salt-run • salt-call • salt-proxy
19. Remote Execution
20. • Package installation ◦ % salt '*' pkg.install nginx ◦ % salt '*' service.start nginx • Direct command ◦ % salt '*' cmd.run 'whoami' • States ◦ % salt '*' state.apply ◦ % salt '*' state.apply mosh • Modules ◦ % salt '*' saltutil.refresh_pillar Salt root@master:~$ salt <target> <module.function> <args>
21. Salt
22. Salt
23. Salt
24. Salt-Call • Run module functions locally ◦ Instead remote executed from the master • Created for debug/troubleshooting/testing % salt-call state.highstate test=True % salt-call state.sls foo.bar test=True % salt-call state.show_sls foo.bar • Also known as masterless minion (Standalone)
25. Salt Masterless • Modify /etc/salt/minion ◦ file_client: local • Specify in CLI ◦ ‘--local’ with salt-call % salt-call --local state.apply • Always can be test % salt-call --local state.apply test=True
26. Salt-SSH • Agentless ◦ Do not require a daemon (minion) process • Roster file: salt22: host: csi-salt2 user: csiuser • Remote execution via ssh
27. Salt-Cloud • Provision systems on cloud providers, hypervisors
28. Salt-Cloud (AWS) ec2-us-west-2-public: minion: master: ip-172-31-30-32 id: 'AWS id' key: 'AWS key+iIP21RaHNBq1DOMaQMkOAgF' private_key: /etc/salt/secret keyname: csiuser-dl-oregon ssh_interface: public_ips securitygroup: security location: us-west-2 iam_profile: arn:iam_role driver: ec2 del_root_vol_on_destroy: True del_all_vols_on_destroy: True rename_on_destroy: True % salt-cloud -p gpu gpuwork1 % salt-cloud -d gpuwork1 gpu: image: ami-d732f0b7 size: g2.2xlarge location: us-west-2 network: default grains: role: gpu tags: {'Environment', 'dev'} del_root_vol_on_destroy: True block_device_mappings: - DeviceName: /dev/sda1 Ebs.VolumeSize: 120 Ebs.VolumeType: gp2 del_all_vol_on_destroy: True ssh_username: ubuntu make_master: False sync_after_install: grains provider: ec2-us-west-2-public
29. Salt-Syndic • Also need salt-master ◦ And salt-syndic • (optionally) ◦ salt-minion • Relay command from ◦ salt-master to salt-syndic’s salt-master • Aggregate result from ◦ salt-minion(s) to salt-master
30. Configuration Management
31. Configuration / Resource Management tools • Master/Agent v.s. Standalone • Standalone: ◦ Small / simple deployments ◦ Dev / Testing • Master/Agent ◦ Large / complicated deployments ◦ Centralization ◦ Parallel operations • Agentless ◦ Security concern?
32. Ref: http://www.slidesshre.net/arthurlutz/debian-meetup-nantes-2015-salt-pour-grer-de-nombreux-serveurs-debian
33. SaLt State (SLS)
34. Salt SLS structure Ref: http://blog.infracloud.io/saltstack-tutorial-part-2/
35. State file format Ref: https://docs.saltstack.com/en/getstarted/config/functions.html
36. State formula execution order To describe: salt '*' state.show_sls sls1[,sls2,...]
37. State External formulas (contrib) • Pre-written salt states ◦ Mostly: https://github.com/saltstack-formulas • Added as GitFS Remote ◦ In /etc/salt/master gitfs_remotes: - https://github.com/saltstack-formulas/consul-formula • Added manually ◦ Clone in /salt/formulas (also specified in file_root) file_roots: base: - /srv/salt - /srv/formulas/datadog-formula - /srv/formulas/etcd-formula - /srv/formulas/docker-formula
38. Grains • Static information ◦ At least not manually changed % salt '*' grains.ls % salt '*' grains.items • Distinguishable in top.sls base: 'consul*': - consul 'role:nogpu': - match: grain - mxnet.no_gpu 'role:gpu': - match: grain - mxnet.with_gpu - torch
39. Pillar example: consul.sls consul: service: true url: http://loadbalancer.us-east-1.elb.amazonaws.com:8500 # for master config: retry_interval: 15s {% if salt['grains.get']('ip4_interfaces:eth0', None) %} bind_addr: {{ grains['ip4_interfaces']['eth0'][0] }} {% endif %} start_join: ["172.31.16.89", "172.31.32.89", "172.31.0.89"] {% if grains['id'].startswith('consul')%} client_addr: "0.0.0.0" acl_down_policy: "extend-cache" {% if grains['id'] == 'consulone'%} bootstrap: true {% else %} bootstrap: false {% endif %} server: true {% endif %} datacenter: "main" ui: true enable_syslog: true encrypt: "keyencrypted" log_level: info data_dir: /var/consul
40. Service Discovery
41. Event and Reactor
42. Event salt-run state.event pretty=True
43. Reactor • /etc/salt/master.d/reactor.conf reactor: # Salt master config section "reactor" - 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts - 'salt/cloud/*/cache_node_new: # Match tag “salt/cloud/*/cache_node_new” - /srv/reactor/autoscale.sls # Things to check if auto scale required - 'csi/grape/metric/gpu': # React to gpu events - salt://reactor/grape_gpu.sls # what states to apply • /srv/reactor/start.sls {% if salt['grains.has_value']('ec2_tags') %} update_ec2_name: cmd.run: - name: /opt/update_ec2_name.sh - require: - file: /opt/update_ec2_name.sh {% endif %}
44. Event and Reactor
45. Beacons Ref: http://gaming.stackexchange.com/questions/227579/what-blocks-can-you-use-to-put-under-a-beacon
46. Beacons - fire events to reactor • /etc/salt/master.d/beacons.conf beacons: inotify: /csi/specific.log: mask: - modify /csi/log: mask: - open - create auto_add: True disable_during_state_run: True load: 5m: - 0.0 - 9.0 onchangeonly: True • Modules to ◦ File system changes ◦ System load ◦ Service status ◦ Shell activity, (e.g. login) ◦ Network and disk usage
47. Event and Reactor beacons
48. Salt Development and Testing
49. “Create and configure lightweight, reproducible, and portable development environments.” - www.vagrantup.com
50. Testing Salt with Vagrant • Fast local testing • Provision VMs • Disposable environment • Built-in Salt provisioner • Private/Public Network # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2”) do |config| config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" config.vm.box = "ubuntu/precise64" config.vm.network "public_network", :bridge => "eth0" # config.vm.network "private_network", :ip => "192.168.31.2" config.vm.hostname = "dev_walker1" config.vm.synced_folder "/srv/sls/", "/srv/" config.vm.provision :salt do |salt| salt.pillar({"is_vagrant" => true}) salt.minion_config = "salt/minion" salt.colorize = true salt.run_highstate = true end end
51. Testing Salt with Docker • Fast local testing • Focus on Salt states • Disposable environment • Masterless FROM ubuntu:14.04 MAINTAINER Cheng-Lung Sung <clsung@gmail.com> RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository -y ppa:saltstack/salt RUN apt-get update && apt-get -y install salt-minion RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* VOLUME ["/etc/salt", "/srv/salt", "/srv/pillar", "/srv/formulas"] CMD salt-call --local state.apply % docker build -t test_docker % docker run --rm -ti -v /path/to/etc/salt:/etc/salt \ -v /path/to/salt:/srv/salt \ -v /path/to/pillar:/srv/pillar \ -v /path/to/formulas:/srv/formulas \ test_docker
52. Discussion (My Salt ongoing list) • AutoScaling ◦ Use Reactor ▪ https://github.com/saltstack-formulas/ec2-autoscale-reactor • Continuous Deployment ◦ Salt-API ◦ Canary or BlueGreen Deployment • Monitoring system ◦ Dashboard ◦ Alert
53. Dashboard
54. Thank you! And Q & A