Live together, NSO and manual operations in multi-vendor "growing brownfield"

NSO Developer Days 2020
Robots vs. human hands. How can humans manage devices alongside NSO? What happens when the topology changes drastically? How can NSO services work with abstract device roles rather than concrete device types? NTT shares some ideas and results from their NSO lab.

Transcript

1. Copyright © NTT Communications Corporation. All rights reserved. Live together, NSO and manual operations in multi-vendor "growing brownfield" Teruhisa Tajima H.Okui, F.Morifuji, S.Kimura NTT Communications NSO Developer Days 2020 @virtual, 24 Jun 2020
2. Copyright © NTT Communications Corporation. All rights reserved. Making Config in NSO 2 devices device config service config operator and/or outer system terminal, API template apply deploy (NED) ref count: how many reference by each line
3. Copyright © NTT Communications Corporation. All rights reserved. ◼ changing config outside of NSO • balance of impl. cost and effect ✓ to avoid impl. of exceptions ✓ to use limited function handmade NED • training of device operation, NSO development Our Usage of NSO 3 devices device config service config operator
4. Copyright © NTT Communications Corporation. All rights reserved. ◼ implement multi-vendor topology model using augment → can change topology after deployed NSO ◼ control ref-count using redeploy and reconcile → can do both manual config and NSO operation Our Key Benefits 4
5. Copyright © NTT Communications Corporation. All rights reserved. ◼ change topology drastically • due to circuit outage, increase bandwidth, etc. Our Use Case 5 A B C 1 2 3 4 A B C 1 2 3 4 5
6. Copyright © NTT Communications Corporation. All rights reserved. 1. pre-configuration of SW C and 5 2. change cable connections 3. post-configuration in NSO • clear interconnection interfaces • set ref-count Operation Steps 6 A B C 5 1 2 3 4
7. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to sync connection ◼ sync step • modify service config • redeploy (no-networking) • sync-from • service discovery • redeploy • reconcile 3. Post-Config = NSO Maintenance 7 A B C 5 1 2 3 4 physical connection in CDB device config in CDB service config
8. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to sync connection ◼ sync step • modify service config • redeploy (no-networking) • sync-from • service discovery • redeploy • reconcile 3. Post-Config = NSO Maintenance 8 A B C 5 1 2 3 4 physical connection in CDB device config in CDB service config
9. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to sync connection ◼ sync step • modify service config • redeploy (no-networking) • sync-from • service discovery • redeploy • reconcile 3. Post-Config = NSO Maintenance 9 A B C 5 1 2 3 4 physical connection in CDB device config in CDB service config device config is out-of- sync
10. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to sync connection ◼ sync step • modify service config • redeploy (no-networking) • sync-from • service discovery • redeploy • reconcile 3. Post-Config = NSO Maintenance 10 A B C 5 1 2 3 4 physical connection in CDB device config in CDB service config device config has incorrect ref-count
11. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to sync connection ◼ sync step • modify service config • redeploy (no-networking) • sync-from • service discovery • redeploy • reconcile 3. Post-Config = NSO Maintenance 11 A B C 5 1 2 3 4 physical connection in CDB device config in CDB service config synced
12. Copyright © NTT Communications Corporation. All rights reserved. ◼ service model implementation • topology model • multi-vendor model using augment ◼ sync configuration between NSO <> devices • = how to resolve inconsistency • service discovery • redeploy & reconcile Approach 12
13. Copyright © NTT Communications Corporation. All rights reserved. ◼ service model implementation • topology model • multi-vendor model using augment ◼ sync configuration between NSO <> devices • = how to resolve inconsistency • service discovery • redeploy & reconcile Approach 13
14. Copyright © NTT Communications Corporation. All rights reserved. ◼ Test new feature, (inter-)operability, performance, etc. • “stable lab NW” is NOT our goal • our lab NW is DUT NW in sometimes ◼ NW’s main service: L2 VLAN in metro area • some packets through tunnel(L2VPN) ◼ 100+ SWs/RTs • AlaxalA • Brocade • Cisco • Juniper Our Lab’s Objectives, Service and Specs 14
15. Copyright © NTT Communications Corporation. All rights reserved. ◼ service config -> device config -> deploy Components to Develop Service 15 devices device config service config operator template apply deploy, sync-from (NED) service discovery service model
16. Copyright © NTT Communications Corporation. All rights reserved. ◼ physical connections How to Calc Path (1) 16 SW: A port: 10 SW: C port: 20 list testbed-physical-connection { key name; leaf name { tailf:info "Unique service id"; type string; } container down-side { leaf device-type { type common:device-type-down-side; } } container up-side { leaf device-type { type common:device-type-up-side; } } leaf disabled { type empty; } } typedef device-type { type enumeration { enum alaxala; enum junos; enum ios; enum brocade;
17. Copyright © NTT Communications Corporation. All rights reserved. ◼ endpoints := interface ports How to Calc Path (2) 17 SW: A port: 10 SW: C port: 20 list testbed-e2e-service { uses ncs:service-data; ncs:servicepoint testbed-e2e-service-servicepoint; key vlan; leaf vlan { mandatory true; type uint16 { range "1 .. 4094"; } } container endpoints { // for augment }
18. Copyright © NTT Communications Corporation. All rights reserved. ◼ equals to how to describe this one port in yang model ◼ patterns • augment • refer each vendor model directly • translate abstract model to vendor model How to Define Multi-Vendor Model 18 gigabitethernet 0/1 GigabitEthernet1/0/1 ge-0/0/1 TenGigabitEthernet 1/0/1 AlaxalA Cisco Juniper Brocade
19. Copyright © NTT Communications Corporation. All rights reserved. ◼ inject from each vendor model ◼ describe as each list-name and leaf-name ◼ pros: get cli suggest(complement), validation , can add model as another yang file when add new vendor ◼ cons: need to learn augment (but easy :-) Augment 19 container endpoints { // for augment } augment "/e2e:testbed-e2e-service/e2e:endpoints" { uses ios-endpoints; } grouping ios-endpoints { list ios { key "device interface"; leaf device { type leafref {...} } leaf interface { type leafref { path "deref(../device)...
20. Copyright © NTT Communications Corporation. All rights reserved. Example of Augment 20 teruhisa.tajima@ncs# show running-config testbed-e2e-service 1060 testbed-e2e-service 1060 endpoints alaxala s01x.akbu port-channel-64 port-mode trunk ! endpoints alaxala s01x.oki3 gigabitethernet-0/1 port-mode access ! endpoints ios s15c.note FastEthernet0/3 port-mode trunk ! endpoints junos d01j.akbu ae20 port-mode trunk ! endpoints junos s01j.oki1 ge-0/0/2 port-mode trunk
21. Copyright © NTT Communications Corporation. All rights reserved. ◼ service model implementation • topology model • multi-vendor model using augment ◼ sync configuration between NSO <> devices • = how to resolve inconsistency • service discovery • redeploy & reconcile Approach 21
22. Copyright © NTT Communications Corporation. All rights reserved. ◼ need to solve inconsistency If Change Device Config Manually 22 devices device config service config commit: A commit: A commit: A commit: B operator commit: C commit: C commit: C inconsistency
23. Copyright © NTT Communications Corporation. All rights reserved. ◼ hard to get diff between before/after topology • difficult to know what configuration to set If Change Topology Drastically 23 which config can be add or delete after topo. changed, rebuild service config from current config
24. Copyright © NTT Communications Corporation. All rights reserved. ◼ implement service discovery as actions Service Discovery 24 devices device config service config operator service discovery devices device s10c.tama config ios:interface GigabitEthernet1/0/24 switchport mode trunk switchport trunk allowed vlan 1000 switchport trunk allowed vlan 2000 testbed-e2e-service 1000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk testbed-e2e-service 2000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk sync-from ref-count: 0 (hidden: 1)
25. Copyright © NTT Communications Corporation. All rights reserved. ◼ redeploy • make relationships between service and device config • each configs are independent if only service discovery Redeploy & Reconcile (set ref-count) 25 device config service config devices device s10c.tama config ios:interface GigabitEthernet1/0/24 switchport mode trunk switchport trunk allowed vlan 1000 switchport trunk allowed vlan 2000 testbed-e2e-service 1000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk testbed-e2e-service 2000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk redeploy ref-count: 1 (hidden: 1)
26. Copyright © NTT Communications Corporation. All rights reserved. ◼ reconcile • clear relationships which made by sync-from • after reconcile, same status as deployed by NSO Redeploy & Reconcile (set ref-count) 26 device config service config devices device s10c.tama config ios:interface GigabitEthernet1/0/24 switchport mode trunk switchport trunk allowed vlan 1000 switchport trunk allowed vlan 2000 testbed-e2e-service 1000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk testbed-e2e-service 2000 endpoints ios s10c.tama GigabitEthernet1/0/24 port-mode trunk reconcile ref-count: 1 (hidden: 0)
27. Copyright © NTT Communications Corporation. All rights reserved. ◼ use partial-sync-to/from? → definition of partial region is difficult if do partial-sync-from, also need to same discovery, redeploy and reconcile → sync-from whole config ◼ force to using NSO at all? → a few operations are too difficult to impl. in NSO → the most cases: operated by NSO and a few cases: operated by hand (and sync NSO) Other Methods 27
28. Copyright © NTT Communications Corporation. All rights reserved. ◼ implement multi-vendor topology model using augment • can add new device, change topology easily ◼ control ref-count using redeploy and reconcile • operate by hands and by NSO at same time • avoid implementing difficult and rare situation Conclusion 28