Upgrade to Pro — share decks privately, control downloads, hide ads and more …

雲端組態佈署自動化 -- SaltStack

雲端組態佈署自動化 -- SaltStack

SaltStack is a software platform to manage system configuration. It aims at fast and scalable automation of any cloud and application stack, or data center infrastructure. In a common use model, sysadmins and devops use SaltStack to automate configuration management and cloud service orchestration.

This talk will cover SaltStack concepts including execution modules, functions, and configuration management, e.g., states and formulas. In addition to demonstrating a simple cloud service (instances of REST API, cache, database, and service discovery), the speaker will also discuss state-of-the-art cloud service practices.

Cheng-Lung Sung

November 11, 2016
Tweet

More Decks by Cheng-Lung Sung

Other Decks in Technology

Transcript

  1. “DevOps is the practice of operations and development engineers participating

    together in the entire service lifecycle, from design through the development process to production support.” - theagileadmin.com/what-is-devops/
  2. DevOps guideline • Elements ◦ Automated ◦ Idempotent, also implies

    ◦ Repeatable • Avoid SPOF (single point of failure) • Release fast, release often ◦ Testing ◦ Continuous Integration ◦ Continuous Deployment
  3. What is SaltStack • Remote execution framework • Configuration management

    and orchestration system • Cloud Provisioning tool • Written in Python • Communicate with 0MQ ◦ Port: 4505, 4506
  4. What we will not talk about • Comparison with CM

    friends ◦ Ansible (Python) ◦ Chef (Ruby) ◦ Puppet (Ruby) • Comparison with Container-related tools ◦ Docker, rxt ◦ Kubernetes, Docker-swarm, Mesos • Some salts ◦ Salt-proxy, Salt Orchestration, Salt Virt, Salt Runner ◦ RAEF Transport (replacement of ZMQ) ◦ Thorium (new Salt Reactor)
  5. Why I choose Saltstack • Written in Python(!) • Been

    using Ansible since in 2014~ ◦ Always find difficulty in using playbooks • Also tried Saltstack in 2014 ◦ Most modules are already Ansibled ▪ “If it ain’t broke, don’t fix it” • New project in 2016 ◦ From scratch, bootstrapping…. ◦ Faster than Ansible (thanks to ZeroMQ)
  6. Saltstack is a remote execution framework • Master/Agent ◦ Master

    commands agents (minions) • Communication: ◦ Publisher (port 4505) ◦ ReqServer (port 4506) • Secure ◦ Salt Minion key authentication ◦ Communication is ▪ AES-encrypted ▪ CBC - 192 bits (why 192 instead of 256?) • Access Control
  7. Saltstack Installation • Fetch the latest script ◦ % curl

    -L https://bootstrap.saltstack.com -o install_salt.sh • Salt-master (Master/Agent) ◦ % sh install_salt.sh -P -M • Salt-minion (Master/Agent, Masterless) ◦ % sh install_salt.sh -P • Salt-syndic (Master/Agent) ◦ % sh install_salt.sh -P -M -S • Salt-ssh (Agentless) ◦ % # noop
  8. Salt commands • salt-[master|minion|syndic] • salt-key • salt-cloud • salt

    • salt-ssh (agentless) • salt-run • salt-call • salt-proxy
  9. • Package installation ◦ % salt '*' pkg.install nginx ◦

    % salt '*' service.start nginx • Direct command ◦ % salt '*' cmd.run 'whoami' • States ◦ % salt '*' state.apply ◦ % salt '*' state.apply mosh • Modules ◦ % salt '*' saltutil.refresh_pillar Salt root@master:~$ salt <target> <module.function> <args>
  10. Salt-Call • Run module functions locally ◦ Instead remote executed

    from the master • Created for debug/troubleshooting/testing % salt-call state.highstate test=True % salt-call state.sls foo.bar test=True % salt-call state.show_sls foo.bar • Also known as masterless minion (Standalone)
  11. Salt Masterless • Modify /etc/salt/minion ◦ file_client: local • Specify

    in CLI ◦ ‘--local’ with salt-call % salt-call --local state.apply • Always can be test % salt-call --local state.apply test=True
  12. Salt-SSH • Agentless ◦ Do not require a daemon (minion)

    process • Roster file: salt22: host: csi-salt2 user: csiuser • Remote execution via ssh
  13. Salt-Cloud (AWS) ec2-us-west-2-public: minion: master: ip-172-31-30-32 id: 'AWS id' key:

    'AWS key+iIP21RaHNBq1DOMaQMkOAgF' private_key: /etc/salt/secret keyname: csiuser-dl-oregon ssh_interface: public_ips securitygroup: security location: us-west-2 iam_profile: arn:iam_role driver: ec2 del_root_vol_on_destroy: True del_all_vols_on_destroy: True rename_on_destroy: True % salt-cloud -p gpu gpuwork1 % salt-cloud -d gpuwork1 gpu: image: ami-d732f0b7 size: g2.2xlarge location: us-west-2 network: default grains: role: gpu tags: {'Environment', 'dev'} del_root_vol_on_destroy: True block_device_mappings: - DeviceName: /dev/sda1 Ebs.VolumeSize: 120 Ebs.VolumeType: gp2 del_all_vol_on_destroy: True ssh_username: ubuntu make_master: False sync_after_install: grains provider: ec2-us-west-2-public
  14. Salt-Syndic • Also need salt-master ◦ And salt-syndic • (optionally)

    ◦ salt-minion • Relay command from ◦ salt-master to salt-syndic’s salt-master • Aggregate result from ◦ salt-minion(s) to salt-master
  15. Configuration / Resource Management tools • Master/Agent v.s. Standalone •

    Standalone: ◦ Small / simple deployments ◦ Dev / Testing • Master/Agent ◦ Large / complicated deployments ◦ Centralization ◦ Parallel operations • Agentless ◦ Security concern?
  16. State External formulas (contrib) • Pre-written salt states ◦ Mostly:

    https://github.com/saltstack-formulas • Added as GitFS Remote ◦ In /etc/salt/master gitfs_remotes: - https://github.com/saltstack-formulas/consul-formula • Added manually ◦ Clone in /salt/formulas (also specified in file_root) file_roots: base: - /srv/salt - /srv/formulas/datadog-formula - /srv/formulas/etcd-formula - /srv/formulas/docker-formula
  17. Grains • Static information ◦ At least not manually changed

    % salt '*' grains.ls % salt '*' grains.items • Distinguishable in top.sls base: 'consul*': - consul 'role:nogpu': - match: grain - mxnet.no_gpu 'role:gpu': - match: grain - mxnet.with_gpu - torch
  18. Pillar example: consul.sls consul: service: true url: http://loadbalancer.us-east-1.elb.amazonaws.com:8500 # for

    master config: retry_interval: 15s {% if salt['grains.get']('ip4_interfaces:eth0', None) %} bind_addr: {{ grains['ip4_interfaces']['eth0'][0] }} {% endif %} start_join: ["172.31.16.89", "172.31.32.89", "172.31.0.89"] {% if grains['id'].startswith('consul')%} client_addr: "0.0.0.0" acl_down_policy: "extend-cache" {% if grains['id'] == 'consulone'%} bootstrap: true {% else %} bootstrap: false {% endif %} server: true {% endif %} datacenter: "main" ui: true enable_syslog: true encrypt: "keyencrypted" log_level: info data_dir: /var/consul
  19. Reactor • /etc/salt/master.d/reactor.conf reactor: # Salt master config section "reactor"

    - 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts - 'salt/cloud/*/cache_node_new: # Match tag “salt/cloud/*/cache_node_new” - /srv/reactor/autoscale.sls # Things to check if auto scale required - 'csi/grape/metric/gpu': # React to gpu events - salt://reactor/grape_gpu.sls # what states to apply • /srv/reactor/start.sls {% if salt['grains.has_value']('ec2_tags') %} update_ec2_name: cmd.run: - name: /opt/update_ec2_name.sh - require: - file: /opt/update_ec2_name.sh {% endif %}
  20. Beacons - fire events to reactor • /etc/salt/master.d/beacons.conf beacons: inotify:

    /csi/specific.log: mask: - modify /csi/log: mask: - open - create auto_add: True disable_during_state_run: True load: 5m: - 0.0 - 9.0 onchangeonly: True • Modules to ◦ File system changes ◦ System load ◦ Service status ◦ Shell activity, (e.g. login) ◦ Network and disk usage
  21. Testing Salt with Vagrant • Fast local testing • Provision

    VMs • Disposable environment • Built-in Salt provisioner • Private/Public Network # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2”) do |config| config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" config.vm.box = "ubuntu/precise64" config.vm.network "public_network", :bridge => "eth0" # config.vm.network "private_network", :ip => "192.168.31.2" config.vm.hostname = "dev_walker1" config.vm.synced_folder "/srv/sls/", "/srv/" config.vm.provision :salt do |salt| salt.pillar({"is_vagrant" => true}) salt.minion_config = "salt/minion" salt.colorize = true salt.run_highstate = true end end
  22. Testing Salt with Docker • Fast local testing • Focus

    on Salt states • Disposable environment • Masterless FROM ubuntu:14.04 MAINTAINER Cheng-Lung Sung <[email protected]> RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository -y ppa:saltstack/salt RUN apt-get update && apt-get -y install salt-minion RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* VOLUME ["/etc/salt", "/srv/salt", "/srv/pillar", "/srv/formulas"] CMD salt-call --local state.apply % docker build -t test_docker % docker run --rm -ti -v /path/to/etc/salt:/etc/salt \ -v /path/to/salt:/srv/salt \ -v /path/to/pillar:/srv/pillar \ -v /path/to/formulas:/srv/formulas \ test_docker
  23. Discussion (My Salt ongoing list) • AutoScaling ◦ Use Reactor

    ▪ https://github.com/saltstack-formulas/ec2-autoscale-reactor • Continuous Deployment ◦ Salt-API ◦ Canary or BlueGreen Deployment • Monitoring system ◦ Dashboard ◦ Alert