Upgrade to Pro — share decks privately, control downloads, hide ads and more …

雲端組態佈署自動化 -- SaltStack

雲端組態佈署自動化 -- SaltStack

SaltStack is a software platform to manage system configuration. It aims at fast and scalable automation of any cloud and application stack, or data center infrastructure. In a common use model, sysadmins and devops use SaltStack to automate configuration management and cloud service orchestration.

This talk will cover SaltStack concepts including execution modules, functions, and configuration management, e.g., states and formulas. In addition to demonstrating a simple cloud service (instances of REST API, cache, database, and service discovery), the speaker will also discuss state-of-the-art cloud service practices.

9dc1fb93b959c0d838bf6a900306d9b9?s=128

Cheng-Lung Sung

November 11, 2016
Tweet

Transcript

  1. 雲端組態佈署自動化 SaltStack Cheng-Lung Sung (clsung@)

  2. Worked at Cloud Service Infrastructure, Studio Engineering, HTC

  3. Funfit -樂動趣

  4. Working at Tensorflow/CSI, HealthCare, HTC

  5. Tricorder Ref: http://www.appledaily.com.tw/realtimenews/article/new/20161109/985048/

  6. Are you Dev, Ops, or DevOps?

  7. “DevOps is the practice of operations and development engineers participating

    together in the entire service lifecycle, from design through the development process to production support.” - theagileadmin.com/what-is-devops/
  8. DevOps: Infrastructure as Code

  9. DevOps guideline • Elements ◦ Automated ◦ Idempotent, also implies

    ◦ Repeatable • Avoid SPOF (single point of failure) • Release fast, release often ◦ Testing ◦ Continuous Integration ◦ Continuous Deployment
  10. What is SaltStack • Remote execution framework • Configuration management

    and orchestration system • Cloud Provisioning tool • Written in Python • Communicate with 0MQ ◦ Port: 4505, 4506
  11. What we will not talk about • Comparison with CM

    friends ◦ Ansible (Python) ◦ Chef (Ruby) ◦ Puppet (Ruby) • Comparison with Container-related tools ◦ Docker, rxt ◦ Kubernetes, Docker-swarm, Mesos • Some salts ◦ Salt-proxy, Salt Orchestration, Salt Virt, Salt Runner ◦ RAEF Transport (replacement of ZMQ) ◦ Thorium (new Salt Reactor)
  12. Why I choose Saltstack

  13. Why I choose Saltstack • Written in Python(!) • Been

    using Ansible since in 2014~ ◦ Always find difficulty in using playbooks • Also tried Saltstack in 2014 ◦ Most modules are already Ansibled ▪ “If it ain’t broke, don’t fix it” • New project in 2016 ◦ From scratch, bootstrapping…. ◦ Faster than Ansible (thanks to ZeroMQ)
  14. Why I choose Saltstack

  15. Saltstack Introduction

  16. Saltstack is a remote execution framework • Master/Agent ◦ Master

    commands agents (minions) • Communication: ◦ Publisher (port 4505) ◦ ReqServer (port 4506) • Secure ◦ Salt Minion key authentication ◦ Communication is ▪ AES-encrypted ▪ CBC - 192 bits (why 192 instead of 256?) • Access Control
  17. Saltstack Installation • Fetch the latest script ◦ % curl

    -L https://bootstrap.saltstack.com -o install_salt.sh • Salt-master (Master/Agent) ◦ % sh install_salt.sh -P -M • Salt-minion (Master/Agent, Masterless) ◦ % sh install_salt.sh -P • Salt-syndic (Master/Agent) ◦ % sh install_salt.sh -P -M -S • Salt-ssh (Agentless) ◦ % # noop
  18. Salt commands • salt-[master|minion|syndic] • salt-key • salt-cloud • salt

    • salt-ssh (agentless) • salt-run • salt-call • salt-proxy
  19. Remote Execution

  20. • Package installation ◦ % salt '*' pkg.install nginx ◦

    % salt '*' service.start nginx • Direct command ◦ % salt '*' cmd.run 'whoami' • States ◦ % salt '*' state.apply ◦ % salt '*' state.apply mosh • Modules ◦ % salt '*' saltutil.refresh_pillar Salt root@master:~$ salt <target> <module.function> <args>
  21. Salt

  22. Salt

  23. Salt

  24. Salt-Call • Run module functions locally ◦ Instead remote executed

    from the master • Created for debug/troubleshooting/testing % salt-call state.highstate test=True % salt-call state.sls foo.bar test=True % salt-call state.show_sls foo.bar • Also known as masterless minion (Standalone)
  25. Salt Masterless • Modify /etc/salt/minion ◦ file_client: local • Specify

    in CLI ◦ ‘--local’ with salt-call % salt-call --local state.apply • Always can be test % salt-call --local state.apply test=True
  26. Salt-SSH • Agentless ◦ Do not require a daemon (minion)

    process • Roster file: salt22: host: csi-salt2 user: csiuser • Remote execution via ssh
  27. Salt-Cloud • Provision systems on cloud providers, hypervisors

  28. Salt-Cloud (AWS) ec2-us-west-2-public: minion: master: ip-172-31-30-32 id: 'AWS id' key:

    'AWS key+iIP21RaHNBq1DOMaQMkOAgF' private_key: /etc/salt/secret keyname: csiuser-dl-oregon ssh_interface: public_ips securitygroup: security location: us-west-2 iam_profile: arn:iam_role driver: ec2 del_root_vol_on_destroy: True del_all_vols_on_destroy: True rename_on_destroy: True % salt-cloud -p gpu gpuwork1 % salt-cloud -d gpuwork1 gpu: image: ami-d732f0b7 size: g2.2xlarge location: us-west-2 network: default grains: role: gpu tags: {'Environment', 'dev'} del_root_vol_on_destroy: True block_device_mappings: - DeviceName: /dev/sda1 Ebs.VolumeSize: 120 Ebs.VolumeType: gp2 del_all_vol_on_destroy: True ssh_username: ubuntu make_master: False sync_after_install: grains provider: ec2-us-west-2-public
  29. Salt-Syndic • Also need salt-master ◦ And salt-syndic • (optionally)

    ◦ salt-minion • Relay command from ◦ salt-master to salt-syndic’s salt-master • Aggregate result from ◦ salt-minion(s) to salt-master
  30. Configuration Management

  31. Configuration / Resource Management tools • Master/Agent v.s. Standalone •

    Standalone: ◦ Small / simple deployments ◦ Dev / Testing • Master/Agent ◦ Large / complicated deployments ◦ Centralization ◦ Parallel operations • Agentless ◦ Security concern?
  32. Ref: http://www.slidesshre.net/arthurlutz/debian-meetup-nantes-2015-salt-pour-grer-de-nombreux-serveurs-debian

  33. SaLt State (SLS)

  34. Salt SLS structure Ref: http://blog.infracloud.io/saltstack-tutorial-part-2/

  35. State file format Ref: https://docs.saltstack.com/en/getstarted/config/functions.html

  36. State formula execution order To describe: salt '*' state.show_sls sls1[,sls2,...]

  37. State External formulas (contrib) • Pre-written salt states ◦ Mostly:

    https://github.com/saltstack-formulas • Added as GitFS Remote ◦ In /etc/salt/master gitfs_remotes: - https://github.com/saltstack-formulas/consul-formula • Added manually ◦ Clone in /salt/formulas (also specified in file_root) file_roots: base: - /srv/salt - /srv/formulas/datadog-formula - /srv/formulas/etcd-formula - /srv/formulas/docker-formula
  38. Grains • Static information ◦ At least not manually changed

    % salt '*' grains.ls % salt '*' grains.items • Distinguishable in top.sls base: 'consul*': - consul 'role:nogpu': - match: grain - mxnet.no_gpu 'role:gpu': - match: grain - mxnet.with_gpu - torch
  39. Pillar example: consul.sls consul: service: true url: http://loadbalancer.us-east-1.elb.amazonaws.com:8500 # for

    master config: retry_interval: 15s {% if salt['grains.get']('ip4_interfaces:eth0', None) %} bind_addr: {{ grains['ip4_interfaces']['eth0'][0] }} {% endif %} start_join: ["172.31.16.89", "172.31.32.89", "172.31.0.89"] {% if grains['id'].startswith('consul')%} client_addr: "0.0.0.0" acl_down_policy: "extend-cache" {% if grains['id'] == 'consulone'%} bootstrap: true {% else %} bootstrap: false {% endif %} server: true {% endif %} datacenter: "main" ui: true enable_syslog: true encrypt: "keyencrypted" log_level: info data_dir: /var/consul
  40. Service Discovery

  41. Event and Reactor

  42. Event salt-run state.event pretty=True

  43. Reactor • /etc/salt/master.d/reactor.conf reactor: # Salt master config section "reactor"

    - 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts - 'salt/cloud/*/cache_node_new: # Match tag “salt/cloud/*/cache_node_new” - /srv/reactor/autoscale.sls # Things to check if auto scale required - 'csi/grape/metric/gpu': # React to gpu events - salt://reactor/grape_gpu.sls # what states to apply • /srv/reactor/start.sls {% if salt['grains.has_value']('ec2_tags') %} update_ec2_name: cmd.run: - name: /opt/update_ec2_name.sh - require: - file: /opt/update_ec2_name.sh {% endif %}
  44. Event and Reactor

  45. Beacons Ref: http://gaming.stackexchange.com/questions/227579/what-blocks-can-you-use-to-put-under-a-beacon

  46. Beacons - fire events to reactor • /etc/salt/master.d/beacons.conf beacons: inotify:

    /csi/specific.log: mask: - modify /csi/log: mask: - open - create auto_add: True disable_during_state_run: True load: 5m: - 0.0 - 9.0 onchangeonly: True • Modules to ◦ File system changes ◦ System load ◦ Service status ◦ Shell activity, (e.g. login) ◦ Network and disk usage
  47. Event and Reactor beacons

  48. Salt Development and Testing

  49. “Create and configure lightweight, reproducible, and portable development environments.” -

    www.vagrantup.com
  50. Testing Salt with Vagrant • Fast local testing • Provision

    VMs • Disposable environment • Built-in Salt provisioner • Private/Public Network # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2”) do |config| config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" config.vm.box = "ubuntu/precise64" config.vm.network "public_network", :bridge => "eth0" # config.vm.network "private_network", :ip => "192.168.31.2" config.vm.hostname = "dev_walker1" config.vm.synced_folder "/srv/sls/", "/srv/" config.vm.provision :salt do |salt| salt.pillar({"is_vagrant" => true}) salt.minion_config = "salt/minion" salt.colorize = true salt.run_highstate = true end end
  51. Testing Salt with Docker • Fast local testing • Focus

    on Salt states • Disposable environment • Masterless FROM ubuntu:14.04 MAINTAINER Cheng-Lung Sung <clsung@gmail.com> RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository -y ppa:saltstack/salt RUN apt-get update && apt-get -y install salt-minion RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* VOLUME ["/etc/salt", "/srv/salt", "/srv/pillar", "/srv/formulas"] CMD salt-call --local state.apply % docker build -t test_docker % docker run --rm -ti -v /path/to/etc/salt:/etc/salt \ -v /path/to/salt:/srv/salt \ -v /path/to/pillar:/srv/pillar \ -v /path/to/formulas:/srv/formulas \ test_docker
  52. Discussion (My Salt ongoing list) • AutoScaling ◦ Use Reactor

    ▪ https://github.com/saltstack-formulas/ec2-autoscale-reactor • Continuous Deployment ◦ Salt-API ◦ Canary or BlueGreen Deployment • Monitoring system ◦ Dashboard ◦ Alert
  53. Dashboard

  54. Thank you! And Q & A