Lisa N Roach - Exploring Network Programmability with Python and YANG

Recently, networking vendors and Silicon Valley giants have been putting forth a concerted effort to build standardized models for networking devices. These models allow for building reusable and versatile scripts with predictable, standardized data. Without such models, the wide variety of inputs and outputs required by different devices and vendors made scripting a tedious and challenging endeavor. The modeling language in use is called YANG, and a variety of standards have emerged. A vendor agnostic standard called OpenConfig has lately become stable enough to begin programming devices with it. Using Python, YANG is surprisingly easy to work with, and extremely powerful applications can be written with basic knowledge of JSON or XML and RPCs.

The talk will start with use-cases for programming networking devices, and will detail a specific, trivial, use case that will be used in the talk. Next, we will discuss the ‘legacy’ way of programming devices (SSH and screenscraping), and highlight the challenges, such as complex regular expressions, slow responses, and lack of reusability between devices. From there we will dive into YANG, focusing on OpenConfig models. A YANG model is essentially a template, and JSON or XML can be mapped to the YANG template. This makes it perfect for Pythonic manipulation. In the use case there will be a GET RPC returning a YANG representation of the box’s state in JSON, which we will search for the relevant health indicator by drilling down in the JSON dictionary. A simple change to the dictionary will remediate the problem, and a PATCH RPC merges the new configuration onto the box. Since open, standard models are in use, this script could be run on many devices across a network to achieve the same effect with no changes needed. We will finish up with the pros and cons of YANG before opening the talk for Q&A.

https://us.pycon.org/2017/schedule/presentation/734/

Transcript

1. Exploring Network Programmability with Python and YANG Lisa Roach
2. Who am I? • Previously a Software Systems Engineer at Cisco Systems • Currently a Production Engineer at Facebook in security
3. What can we program? • Configuration management • Health monitoring and problem remediation • Customized protocols
4. How to program networking devices? • APIs • SSH • YANG models + RPC
5. Switch A Switch B GigabitEthernet0/0/0/0 Our Scenario
6. Diagnosing without YANG or APIs SSH + Screenscraping + Regex import paramiko import re ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') def check_box(): command = 'show interface GigabitEthernet 0/0/0/0' stdin, stdout, stderr = ssh.exec_command(command) output = '' for line in stdout: output += line return bool(re.search('GigabitEthernet0/0/0/0 is up', output))
7. Diagnosing without YANG or APIs SSH + Screenscraping + Regex import paramiko import re ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') def check_box(): command = 'show interface GigabitEthernet 0/0/0/0' stdin, stdout, stderr = ssh.exec_command(command) output = '' for line in stdout: output += line return bool(re.search('GigabitEthernet0/0/0/0 is up', output))
8. Networks are Mixed
9. The Emergence of YANG “Yet Another Next Generation” What is YANG? A data modeling language used to describe configuration and state data. How does YANG help? Provides a consistent model to configure and monitor network devices in a software driven way, regardless of vendor.
10. Yang Models Need to be Standardized xkcd.com
11. What does YANG look like? module openconfig-interfaces { yang-version "1"; grouping interfaces-top { container interfaces { list interface { key "name"; leaf name { type leafref { path "../config/name"; } } container config { uses interface-phys-config; } ...
12. How can we use YANG? • Configuration and state data is modeled on box to YANG specifications • RPC calls can be used to grab the data off the box and return it as XML or JSON • Python can be used to manipulate the XML or JSON values • A RPC can be used to send the manipulated data back to the box
13. Pyang module: openconfig-interfaces +--rw interfaces +--rw interface* [name] +--rw name -> ../config/name +--rw config | +--rw type identityref | +--rw mtu? uint16 | +--rw name? string | +--rw description? string | +--rw enabled? boolean +--ro state | +--ro type identityref ... pyang -f tree pyang/public/release/models/interfaces/openconfig-interfaces.yang
14. How can we use YANG? RP/0/RP0/CPU0:rtr1#show interface gigabitEthernet 0/0/0/0 Tue May 9 23:54:59.353 UTC GigabitEthernet0/0/0/0 is up, line protocol is up Interface state transitions: 1 Hardware is GigabitEthernet, address is 0800.2703.e97f (bia 0800.2703.e97f) Description: to pop router Internet address is 12.1.1.20/24 MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Full-duplex, 1000Mb/s, unknown, link type is force-up output flow control is off, input flow control is off Carrier delay (up) is 10 msec loopback not set, Last link flapped 00:01:12 ARP type ARPA, ARP timeout 04:00:00 Last input never, output 00:00:01 Last clearing of "show interface" counters never 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 1000 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 total input drops 0 drops for unrecognized upper-level protocol Received 0 broadcast packets, 0 multicast packets 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 6796 packets output, 9746584 bytes, 0 total output drops Output 0 broadcast packets, 0 multicast packets 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 13 carrier transitions Last link flapped 00:01:12 ARP type ARPA, ARP timeout 04:00:00 Last input never, output 00:00:01 Last clearing of "show interface" counters never 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 1000 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 total input drops 0 drops for unrecognized upper-level protocol 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
15. How can we use YANG? RP/0/RP0/CPU0:rtr1#show interface gigabitEthernet 0/0/0/0 Tue May 9 23:54:59.353 UTC GigabitEthernet0/0/0/0 is up, line protocol is up Interface state transitions: 1 Hardware is GigabitEthernet, address is 0800.2703.e97f (bia 0800.2703.e97f) Description: to pop router Internet address is 12.1.1.20/24 MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Full-duplex, 1000Mb/s, unknown, link type is force-up output flow control is off, input flow control is off Carrier delay (up) is 10 msec loopback not set, Last link flapped 00:01:12 ARP type ARPA, ARP timeout 04:00:00 Last input never, output 00:00:01 . . { "openconfig-interfaces:interfaces": { "interface": [ { "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": true, "description": "to DC router" }, "openconfig-if-ethernet:ethernet": { "config": { "auto-negotiate": false } }, "subinterfaces": { "subinterface": [
16. How can we use YANG? RPC Client RPC Server { "openconfig-interfaces:interfaces": { "interface": [ { "openconfig-if-ethernet:ethernet": { "config": { "auto-negotiate": false } }, "config": { "enabled": false, ...
17. How can we use YANG? RPC Client RPC Server { "openconfig-interfaces:interfaces": { "interface": [ { "openconfig-if-ethernet:ethernet": { "config": { "auto-negotiate": false } }, "config": { "enabled": false, ...
18. module openconfig-interfaces { ... grouping interfaces-top { container interfaces { list interface { key "name"; leaf name { type leafref { path "../config/name"; } ... YANG { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0" }] } } JSON Mapping Mapping YANG to JSON GigabitEthernet0/0/0/0
19. Let’s Fix Our Switch Switch A Switch B GigabitEthernet0/0/0/0
20. The SSH + Screenscraping Way ... if not check_box(): ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') shell = ssh.invoke_shell() while not shell.recv_ready(): sleep(0.1) shell.send('conf t\n') shell.send('interface GigabitEthernet 0/0/0/0\n') shell.send('no shutdown\n') shell.send('commit\n') sleep(1) ...
21. The SSH + Screenscraping Way ... if not check_box(): ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') shell = ssh.invoke_shell() while not shell.recv_ready(): sleep(0.1) shell.send('conf t\n') shell.send('interface GigabitEthernet 0/0/0/0\n') shell.send('no shutdown\n') shell.send('commit\n') sleep(1) ...
22. The SSH + Screenscraping Way ... if not check_box(): ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') shell = ssh.invoke_shell() while not shell.recv_ready(): sleep(0.1) shell.send('conf t\n') shell.send('interface GigabitEthernet 0/0/0/0\n') shell.send('no shutdown\n') shell.send('commit\n') sleep(1) ...
23. The SSH + Screenscraping Way ... if not check_box(): ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') shell = ssh.invoke_shell() while not shell.recv_ready(): sleep(0.1) shell.send('conf t\n') shell.send('interface GigabitEthernet 0/0/0/0\n') shell.send('no shutdown\n') shell.send('commit\n') sleep(1) ...
24. The SSH + Screenscraping Way ... if not check_box(): ssh.connect('10.1.1.5', 22, 'vagrant', 'vagrant') shell = ssh.invoke_shell() while not shell.recv_ready(): sleep(0.1) shell.send('conf t\n') shell.send('interface GigabitEthernet 0/0/0/0\n') shell.send('no shutdown\n') shell.send('commit\n') sleep(1) ...
25. The YANG way
26. Formulate Get Request model = '''{ "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0" }] } }''' { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0" }] } } from Tools.grpc_client import GRPCClient client = GRPCClient('10.1.1.5', 57777, 10, 'vagrant', 'vagrant') config = client.getconfig(model)
27. Response >>> response = json.loads(config, object_pairs_hook=OrderedDict)
28. Response >>> response = json.loads(config, object_pairs_hook=OrderedDict) >>> print(json.dumps(response, indent=4))
29. Response >>> response = json.loads(config, object_pairs_hook=OrderedDict) >>> print(json.dumps(response, indent=4)) { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": false, ...
30. Response >>> response = json.loads(config, object_pairs_hook=OrderedDict) >>> print(json.dumps(response, indent=4)) { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": false, ...
31. Response >>> response = json.loads(config, object_pairs_hook=OrderedDict) >>> print(json.dumps(response, indent=4)) { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": false, ...
32. Manipulate the data response = json.loads(config, object_pairs_hook=OrderedDict) interface = response['openconfig-interfaces:interfaces']['interface'] >>> print(json.dumps(interface, indent=4)) [0] { "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": false, ...
33. Manipulate the data response = json.loads(config, object_pairs_hook=OrderedDict) interface = response['openconfig-interfaces:interfaces']['interface'][0] interface['config']['enabled'] = True >>> print(json.dumps(response, indent=4)) { "openconfig-interfaces:interfaces": { "interface": [{ "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": true,
34. RPC Put/Patch/Merge response = json.loads(config, object_pairs_hook=OrderedDict) interface = response['openconfig-interfaces:interfaces']['interface'][0] interface['config']['enabled'] = True updated_config = json.dumps(response) err = client.mergeconfig(updated_config)
35. Check Change is Successful config = client.getconfig(model) >>> print(json.dumps(response, indent=4)) { "openconfig-interfaces:interfaces": { "interface": [ { "name": "GigabitEthernet0/0/0/0", "config": { "name": "GigabitEthernet0/0/0/0", "type": "iana-if-type:ethernetCsmacd", "enabled": true, ... response = json.loads(config, object_pairs_hook=OrderedDict)
36. Let’s Fix Our Switch Switch A Switch B GigabitEthernet0/0/0/0
37. Timing Type $ time python yang_change.py $ time python screenscrape.py Difference real 0.551s 2.800s 2.249s user 0.161s 0.453s 0.292s sys 0.029s 0.053s 0.024s
38. Drawbacks of YANG • Not 100% stable • Not largely available (yet) • Devices can be blocking • Few tools and documentation for support
39. Helpful Resources • RFC: https://tools.ietf.org/html/rfc6020 Github • Yang: https://github.com/YangModels/yang • Pipedown: https://github.com/cisco-ie/Pipedown • Solenoid: https://github.com/lisroach/Solenoid • GRPC Client: https://github.com/cisco-ie/ios-xr-grpc-python • Napalm (for ssh and YANG!): https://github.com/napalm- automation/napalm