Slide 1

Slide 1 text

digitalocean.com Managing Networks and Applications Using Salt, Without Minions Mircea Ulinic

Slide 2

Slide 2 text

Hi, I’m Mircea - Senior Network Engineer at DigitalOcean - Previously at Cloudflare, EPFL - O’Reilly author - SaltStack contributor of the year 2017 - SSCE 0xAB1DA421 - @mirceaulinic on GH, TWTR, LNKD - https://mirceaulinic.net/ 2

Slide 3

Slide 3 text

Agenda - Typical Salt architectures - Getting started with Salt SProxy - Using Salt SProxy - Demo 3

Slide 4

Slide 4 text

Brief Recap: Typical Architecture https://docs.saltstack.com/en/latest/topics/topology/index.html Master Minion Minion Minion ... 4

Slide 5

Slide 5 text

Brief Recap: Multi-Master Architecture https://docs.saltstack.com/en/latest/topics/topology/index.html Master Minion Minion Minion ... 5 Master Master ...

Slide 6

Slide 6 text

Salt in networking: theoretical approaches 6 Salt Minion Salt SSH

Slide 7

Slide 7 text

Salt Minion Traditional platforms such as Juniper or Cisco are closed and you can’t install custom software. Salt in networking: theoretical approaches 7 Salt SSH X

Slide 8

Slide 8 text

Salt Minion Traditional platforms such as Juniper or Cisco are closed and you can’t install custom software. Salt in networking: theoretical approaches 8 Salt SSH Salt SSH requires Python on the remote target, and uploads an archive on execution. Equally impossible on typical gear. X X

Slide 9

Slide 9 text

Brief Recap: Network Automation Topology (with 1 Master) https://docs.saltstack.com/en/latest/topics/topology/index.html Master Proxy Minion Proxy Minion Proxy Minion ... Network Device Network Device Network Device ... NETCONF HTTP SSH 9

Slide 10

Slide 10 text

Typical Network Automation Topology using Proxies (1) 10 Proxy Minions are simple processes able to run anywhere, as long as: 1) Can connect to the Master. 2) Can connect to the network device (via the channel / API of choice - e.g., SSH / NETCONF / HTTP / gRPC, etc.)

Slide 11

Slide 11 text

Typical Network Automation Topology using Proxies (2) 11 Deployment examples include: ● Running as system services ○ On a single server ○ Distributed on various servers ● (Docker) containers ○ E.g., managed by Kubernetes ● Services running in a cloud ○ See, for example, salt-cloud

Slide 12

Slide 12 text

Typical Network Automation Topology using Proxies (3) 12 Proxy Minions imply a process always running in the background. That means, whenever you execute a command, Salt is instantly available to run the command. But also means: - A process always keeping memory busy. - Typically each proxy requires around 100-120MB. - Due to a memory leak, when using any SSH-based Proxies, the memory usage can spike to over 1GB (for every Proxy Minion). - One Proxy Minion service per device. - System services management, monitoring, etc. Not always beneficial, sometimes you just need a one-off command every X weeks / months.

Slide 13

Slide 13 text

Introducing salt-sproxy (Salt Super Proxy) 13 https://salt-sproxy.readthedocs.io/ Salt plugin to automate the management and configuration of network devices at scale, without running (Proxy) Minions. Using salt-sproxy, you can continue to benefit from the scalability, flexibility and extensibility of Salt, while you don't have to manage thousands of (Proxy) Minion services. However, you are able to use both salt-sproxy and your (Proxy) Minions at the same time.

Slide 14

Slide 14 text

Remember slide #8? https://docs.saltstack.com/en/latest/topics/topology/index.html Master Proxy Minion Proxy Minion Proxy Minion ... Network Device Network Device Network Device ... NETCONF HTTP SSH 14

Slide 15

Slide 15 text

Topology using salt-sproxy salt-sproxy Network Device Network Device Network Device ... NETCONF HTTP SSH 15 https://salt-sproxy.readthedocs.io/

Slide 16

Slide 16 text

Topology using salt-sproxy salt-sproxy Network Device Network Device Network Device ... NETCONF HTTP SSH 16 This can be any server, or your own computer. https://salt-sproxy.readthedocs.io/

Slide 17

Slide 17 text

Getting started with salt-sproxy: Installation 17 https://salt-sproxy.readthedocs.io/ $ pip install salt-sproxy See a recorded demo at: https://asciinema.org/a/247697?autoplay=1

Slide 18

Slide 18 text

Build the database of devices you want to manage. For example, as a file: Getting started with salt-sproxy: Setup example (1) 18 https://salt-sproxy.readthedocs.io/en/latest/quick_start.html router1: driver: junos router2: driver: iosxr spine1: driver: junos leaf1: driver: eos fw1: driver: panos host: fw1.firewall.as1234.net /etc/salt/roster

Slide 19

Slide 19 text

And, finally, let salt-sproxy know that the data is loaded from the Roster file: Getting started with salt-sproxy: Setup example (2) 19 roster: file proxy: proxytype: napalm username: password: /etc/salt/master https://salt-sproxy.readthedocs.io/en/latest/quick_start.html

Slide 20

Slide 20 text

And, finally, let salt-sproxy know that the data is loaded from the Roster file: Getting started with salt-sproxy: Setup example (2) 20 roster: file proxy: proxytype: napalm username: password: /etc/salt/master https://salt-sproxy.readthedocs.io/en/latest/quick_start.html There are different flavours of Roster sources, including NetBox, Pillar (i.e., retrieve data from HTTP APIs, MySQL / PostgreSQL databases, etc.). File is the easiest to understand and demo.

Slide 21

Slide 21 text

And, finally, let salt-sproxy know that the data is loaded from the Roster file: Getting started with salt-sproxy: Setup example (2) 21 roster: file proxy: proxytype: napalm username: password: /etc/salt/master https://salt-sproxy.readthedocs.io/en/latest/quick_start.html You can choose between a variety of Proxy Modules natively available in Salt. If none available for your use case, developing a new Proxy Module in your own environment is easy and straighforward.

Slide 22

Slide 22 text

After these three easy steps, you can start running commands: Getting started with salt-sproxy: Usage 22 https://salt-sproxy.readthedocs.io/en/latest/quick_start.html $ salt-sproxy ‘router*’ --preview-target - router1 - router2 $ salt-sproxy ‘router*’ net.arp … snip … $ salt-sproxy ‘router*’ net.load_config \ text=’set system ntp server 10.0.0.1’ test=True … snip ...

Slide 23

Slide 23 text

After these three easy steps, you can start running commands: Getting started with salt-sproxy: Usage 23 https://salt-sproxy.readthedocs.io/en/latest/quick_start.html $ salt-sproxy ‘router1’ net.load_config \ text=’set system ntp server 10.0.0.1’ test=True router1: ---------- already_configured: False comment: Configuration discarded. diff: [edit system] + ntp { + server 10.0.0.1; + } loaded_config: result: True

Slide 24

Slide 24 text

In the previous examples, we used SLS data from a specific file (i.e., information that we maintain ourselves) as SLS files , to build the list of devices. But there can be plenty of other sources where to load this data from, see https://docs.saltstack.com/en/latest/ref/pillar/all/index.html, examples include: - HTTP API - Postgres / MySQL database - Etcd, Consul, Redis, Mongo, etc. - CSV file :-( Getting started with salt-sproxy: Alternative setup 24 https://salt-sproxy.readthedocs.io/en/latest/roster.html

Slide 25

Slide 25 text

Update /etc/salt/master to let salt-sproxy know that you want to load the list of devices from NetBox: Getting started with salt-sproxy: Alternative setup - NetBox 25 https://salt-sproxy.readthedocs.io/en/latest/examples/netbox.html roster: netbox netbox: url: https://netbox.live/ token: /etc/salt/master

Slide 26

Slide 26 text

Salt has a natively available a REST API, which can be used in combination with salt-sproxy to invoke commands over HTTP, without running Proxy Minions. Enable the API: Using salt-sproxy via the Salt REST API 26 https://salt-sproxy.readthedocs.io/en/latest/salt_api.html rest_cherrypy: port: 8080 ssl_crt: /path/to/crt ssl_key: /path/to/key /etc/salt/master

Slide 27

Slide 27 text

After these three easy steps, you can start running commands: 27 https://salt-sproxy.readthedocs.io/en/latest/salt_api.html $ curl -sS localhost:8080/run -H 'Accept: application/x-yaml' \ -d eauth='pam' \ -d username='mircea' \ -d password='pass' \ -d client=’sproxy’ \ -d tgt=router1 \ -d fun='test.ping' return: router1: true Using salt-sproxy via the Salt REST API

Slide 28

Slide 28 text

Not only for networks Salt SProxy can be used to manage anything, without running Minions: - Unix & Windows machines - Docker containers - VMWare ESXi clusters - VMWare vCenters - Marathon - Chronos clusters - Philips HUE lamps … virtually anything 28

Slide 29

Slide 29 text

Salt SProxy as a replacement for Salt SSH SSH Proxy added in Salt SProxy 2020.7.0 https://salt-sproxy.readthedocs.io/en/latest/ssh.html 29 roster: file proxy: proxytype: ssh user: priv: priv_passwd: /etc/salt/master $ salt-sproxy ’srv1’ cmd.run ‘uname -vs’ return: Linux #51~18.04.1-Ubuntu SMP $ salt-sproxy ’srv1’ grains.get cpu_model return: AMD EPYC 7402P 24-Core Processor $ salt-sproxy ’srv1’ state.apply ...

Slide 30

Slide 30 text

Why salt-sproxy 30 https://github.com/mirceaulinic/salt-sproxy - salt-sproxy is much easier to install (compared to the typical Salt setup). - Say goodbye to the burden of managing hundreds of system services for the Proxy Minion processes. - You can run it locally, on your own computer. - Integrates easily with your existing Salt environment (if you have), by installing salt-sproxy on your Salt Master. - Reuse your existing extension modules, templates, Pillars, States, etc., you may have already developed in your environment, transparently. - Can continue to leverage the event-driven automation and orchestration methodologies. - REST API, see also Using the Salt REST API documentation. - Python programming made a breeze - might go well with the ISalt package.

Slide 31

Slide 31 text

Questions?

Slide 32

Slide 32 text

Live Demo 32

Slide 33

Slide 33 text

Live Demo setup 33 salt-sproxy Juniper VM Arista VM . . . Juniper VM Arista VM Proxy Minion Proxy Minion Juniper VM Linux VM Linux VM NETCONF HTTP SSH Salt Bus

Slide 34

Slide 34 text

Demo setup 34 - Multiple Docker containers for Juniper Junos, Arista EOS, and Linux: vmx1, vmx2, vmx3, eos-spine1, eos-spine2, eos-leaf1, eos-leaf2. - 3 Docker containers running Ubuntu: srv1, srv2, and srv3. - One Docker container for Salt Master + Salt SProxy - A few Docker containers running Proxy Minions (one Proxy Minion per device we manage) - vmx1 - vmx2 - eos-spine1 - eos-spine2 Using salt-sproxy, we can manage both devices that have a Proxy Minion running, as well as devices without.

Slide 35

Slide 35 text

Salt SProxy configuration 35 Configuration files, installation script, and demo CLI available at https://github.com/mirceaulinic/SaltConf20

Slide 36

Slide 36 text

Salt-sproxy configuration essentials 36 roster: file roster_file: /etc/salt/roster use_existing_proxy : true vmx3: grains: role: router eos-leaf1: grains: role: leaf eos-leaf2: grains: role: leaf {%- for i in [1,2,3] %} srv{{ i }}: grains: role: hypervisor {%- endfor %} /etc/salt/master /etc/salt/roster

Slide 37

Slide 37 text

Salt-sproxy configuration essentials 37 roster: file roster_file: /etc/salt/roster use_existing_proxy : true vmx3: grains: role: router eos-leaf1: grains: role: leaf eos-leaf2: grains: role: leaf {%- for i in [1,2,3] %} srv{{ i }}: grains: role: hypervisor {%- endfor %} /etc/salt/master /etc/salt/roster Ensure salt-sproxy has access to the running Proxy Minions.

Slide 38

Slide 38 text

Salt-sproxy configuration essentials 38 roster: file roster_file: /etc/salt/roster use_existing_proxy : true vmx3: grains: role: router eos-leaf1: grains: role: leaf eos-leaf2: grains: role: leaf {%- for i in [1,2,3] %} srv{{ i }}: grains: role: hypervisor {%- endfor %} /etc/salt/master /etc/salt/roster The Roster file is Salt SLS format (Jinja + YAML)

Slide 39

Slide 39 text

Salt-sproxy configuration essentials 39 base: ‘vmx*’: - vmx ‘eos*’: - eos ‘srv*’: - ssh proxy: proxytype: napalm driver: junos host: {{ opts.id }} username: saltconf password: SaltConf20 /srv/salt/top.sls /srv/salt/vmx.sls proxy: proxytype: ssh host: {{ opts.id }} user: root priv: /etc/salt/ssh_key priv_passwd: SaltConf20 /srv/salt/ssh.sls

Slide 40

Slide 40 text

Running salt-sproxy 40 $ salt * --preview-target - eos-spine1 - eos-spine2 - vmx1 - vmx2 $ salt-sproxy * --preview-target - eos-spine1 - eos-spine2 - vmx1 - vmx2 - vmx3 - vmx4 - eos-leaf1 - eos-leaf2 - srv1 - srv2 - srv3 Devices managed through running Proxy Minions. Devices managed without Proxy Minions (defined in the Roster file)

Slide 41

Slide 41 text

Running salt-sproxy 41 More examples in the cli.sh script

Slide 42

Slide 42 text

Thanks!

Slide 43

Slide 43 text