Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Live together, NSO and manual operations in multi-vendor "growing brownfield"

Live together, NSO and manual operations in multi-vendor "growing brownfield"

NSO Developer Days 2020
Robots vs. human hands. How can humans manage devices alongside NSO? What happens when the topology changes drastically? How can NSO services work with abstract device roles rather than concrete device types? NTT shares some ideas and results from their NSO lab.

tjmtrhs

June 24, 2020
Tweet

More Decks by tjmtrhs

Other Decks in Technology

Transcript

  1. Copyright © NTT Communications Corporation. All rights reserved.
    Live together,
    NSO and manual operations
    in multi-vendor "growing brownfield"
    Teruhisa Tajima
    H.Okui, F.Morifuji, S.Kimura
    NTT Communications
    NSO Developer Days 2020
    @virtual, 24 Jun 2020

    View Slide

  2. Copyright © NTT Communications Corporation. All rights reserved.
    Making Config in NSO
    2
    devices
    device config
    service config
    operator
    and/or
    outer system
    terminal,
    API
    template
    apply
    deploy
    (NED)
    ref count: how
    many reference
    by each line

    View Slide

  3. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ changing config outside of NSO
    • balance of impl. cost and effect
    ✓ to avoid impl. of exceptions
    ✓ to use limited function handmade NED
    • training of device operation, NSO development
    Our Usage of NSO
    3
    devices
    device config
    service config
    operator

    View Slide

  4. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ implement multi-vendor topology model using augment
    → can change topology after deployed NSO
    ◼ control ref-count using redeploy and reconcile
    → can do both manual config and NSO operation
    Our Key Benefits
    4

    View Slide

  5. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ change topology drastically
    • due to circuit outage, increase bandwidth, etc.
    Our Use Case
    5
    A B C
    1 2
    3 4
    A B C
    1 2 3 4
    5

    View Slide

  6. Copyright © NTT Communications Corporation. All rights reserved.
    1. pre-configuration of SW C and 5
    2. change cable connections
    3. post-configuration in NSO
    • clear interconnection interfaces
    • set ref-count
    Operation Steps
    6
    A B C
    5
    1 2 3 4

    View Slide

  7. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to sync connection
    ◼ sync step
    • modify service config
    • redeploy (no-networking)
    • sync-from
    • service discovery
    • redeploy
    • reconcile
    3. Post-Config = NSO Maintenance
    7
    A B C
    5
    1 2 3 4
    physical connection
    in CDB device config
    in CDB service config

    View Slide

  8. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to sync connection
    ◼ sync step
    • modify service config
    • redeploy (no-networking)
    • sync-from
    • service discovery
    • redeploy
    • reconcile
    3. Post-Config = NSO Maintenance
    8
    A B C
    5
    1 2 3 4
    physical connection
    in CDB device config
    in CDB service config

    View Slide

  9. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to sync connection
    ◼ sync step
    • modify service config
    • redeploy (no-networking)
    • sync-from
    • service discovery
    • redeploy
    • reconcile
    3. Post-Config = NSO Maintenance
    9
    A B C
    5
    1 2 3 4
    physical connection
    in CDB device config
    in CDB service config
    device config
    is out-of-
    sync

    View Slide

  10. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to sync connection
    ◼ sync step
    • modify service config
    • redeploy (no-networking)
    • sync-from
    • service discovery
    • redeploy
    • reconcile
    3. Post-Config = NSO Maintenance
    10
    A B C
    5
    1 2 3 4
    physical connection
    in CDB device config
    in CDB service config
    device config
    has incorrect
    ref-count

    View Slide

  11. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to sync connection
    ◼ sync step
    • modify service config
    • redeploy (no-networking)
    • sync-from
    • service discovery
    • redeploy
    • reconcile
    3. Post-Config = NSO Maintenance
    11
    A B C
    5
    1 2 3 4
    physical connection
    in CDB device config
    in CDB service config
    synced

    View Slide

  12. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ service model implementation
    • topology model
    • multi-vendor model using augment
    ◼ sync configuration between NSO <> devices
    • = how to resolve inconsistency
    • service discovery
    • redeploy & reconcile
    Approach
    12

    View Slide

  13. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ service model implementation
    • topology model
    • multi-vendor model using augment
    ◼ sync configuration between NSO <> devices
    • = how to resolve inconsistency
    • service discovery
    • redeploy & reconcile
    Approach
    13

    View Slide

  14. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ Test new feature, (inter-)operability, performance, etc.
    • “stable lab NW” is NOT our goal
    • our lab NW is DUT NW in sometimes
    ◼ NW’s main service: L2 VLAN in metro area
    • some packets through tunnel(L2VPN)
    ◼ 100+ SWs/RTs
    • AlaxalA
    • Brocade
    • Cisco
    • Juniper
    Our Lab’s Objectives, Service and Specs
    14

    View Slide

  15. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ service config -> device config -> deploy
    Components to Develop Service
    15
    devices
    device config
    service config
    operator
    template
    apply
    deploy,
    sync-from
    (NED)
    service
    discovery
    service model

    View Slide

  16. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ physical connections
    How to Calc Path (1)
    16
    SW: A
    port: 10
    SW: C
    port: 20
    list testbed-physical-connection {
    key name;
    leaf name {
    tailf:info "Unique service id";
    type string;
    }
    container down-side {
    leaf device-type {
    type common:device-type-down-side;
    }
    }
    container up-side {
    leaf device-type {
    type common:device-type-up-side;
    }
    }
    leaf disabled {
    type empty;
    }
    }
    typedef device-type {
    type enumeration {
    enum alaxala;
    enum junos;
    enum ios;
    enum brocade;

    View Slide

  17. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ endpoints := interface ports
    How to Calc Path (2)
    17
    SW: A
    port: 10
    SW: C
    port: 20
    list testbed-e2e-service {
    uses ncs:service-data;
    ncs:servicepoint testbed-e2e-service-servicepoint;
    key vlan;
    leaf vlan {
    mandatory true;
    type uint16 {
    range "1 .. 4094";
    }
    }
    container endpoints {
    // for augment
    }

    View Slide

  18. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ equals to how to describe this one port in yang model
    ◼ patterns
    • augment
    • refer each vendor model directly
    • translate abstract model to vendor model
    How to Define Multi-Vendor Model
    18
    gigabitethernet 0/1
    GigabitEthernet1/0/1
    ge-0/0/1
    TenGigabitEthernet 1/0/1
    AlaxalA
    Cisco
    Juniper
    Brocade

    View Slide

  19. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ inject from each vendor model
    ◼ describe as each list-name and leaf-name
    ◼ pros: get cli suggest(complement), validation , can add
    model as another yang file when add new vendor
    ◼ cons: need to learn augment (but easy :-)
    Augment
    19
    container endpoints {
    // for augment
    }
    augment "/e2e:testbed-e2e-service/e2e:endpoints" {
    uses ios-endpoints;
    }
    grouping ios-endpoints {
    list ios {
    key "device interface";
    leaf device { type leafref {...} }
    leaf interface {
    type leafref {
    path "deref(../device)...

    View Slide

  20. Copyright © NTT Communications Corporation. All rights reserved.
    Example of Augment
    20
    teruhisa.tajima@ncs# show running-config testbed-e2e-service 1060
    testbed-e2e-service 1060
    endpoints alaxala s01x.akbu port-channel-64
    port-mode trunk
    !
    endpoints alaxala s01x.oki3 gigabitethernet-0/1
    port-mode access
    !
    endpoints ios s15c.note FastEthernet0/3
    port-mode trunk
    !
    endpoints junos d01j.akbu ae20
    port-mode trunk
    !
    endpoints junos s01j.oki1 ge-0/0/2
    port-mode trunk

    View Slide

  21. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ service model implementation
    • topology model
    • multi-vendor model using augment
    ◼ sync configuration between NSO <> devices
    • = how to resolve inconsistency
    • service discovery
    • redeploy & reconcile
    Approach
    21

    View Slide

  22. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ need to solve inconsistency
    If Change Device Config Manually
    22
    devices
    device config
    service config
    commit: A commit: A commit: A
    commit: B
    operator
    commit: C commit: C commit: C
    inconsistency

    View Slide

  23. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ hard to get diff between before/after topology
    • difficult to know what configuration to set
    If Change Topology Drastically
    23
    which config
    can be add
    or delete
    after topo. changed,
    rebuild service config
    from current config

    View Slide

  24. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ implement service discovery as actions
    Service Discovery
    24
    devices
    device config
    service config
    operator
    service
    discovery
    devices device s10c.tama
    config
    ios:interface GigabitEthernet1/0/24
    switchport mode trunk
    switchport trunk allowed vlan 1000
    switchport trunk allowed vlan 2000
    testbed-e2e-service 1000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    testbed-e2e-service 2000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    sync-from
    ref-count: 0
    (hidden: 1)

    View Slide

  25. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ redeploy
    • make relationships between service and device config
    • each configs are independent if only service discovery
    Redeploy & Reconcile (set ref-count)
    25
    device config
    service config
    devices device s10c.tama
    config
    ios:interface GigabitEthernet1/0/24
    switchport mode trunk
    switchport trunk allowed vlan 1000
    switchport trunk allowed vlan 2000
    testbed-e2e-service 1000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    testbed-e2e-service 2000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    redeploy
    ref-count: 1
    (hidden: 1)

    View Slide

  26. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ reconcile
    • clear relationships which made by sync-from
    • after reconcile, same status as deployed by NSO
    Redeploy & Reconcile (set ref-count)
    26
    device config
    service config
    devices device s10c.tama
    config
    ios:interface GigabitEthernet1/0/24
    switchport mode trunk
    switchport trunk allowed vlan 1000
    switchport trunk allowed vlan 2000
    testbed-e2e-service 1000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    testbed-e2e-service 2000
    endpoints ios s10c.tama GigabitEthernet1/0/24
    port-mode trunk
    reconcile
    ref-count: 1
    (hidden: 0)

    View Slide

  27. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ use partial-sync-to/from?
    → definition of partial region is difficult
    if do partial-sync-from, also need to same discovery,
    redeploy and reconcile
    → sync-from whole config
    ◼ force to using NSO at all?
    → a few operations are too difficult to impl. in NSO
    → the most cases: operated by NSO
    and a few cases: operated by hand (and sync NSO)
    Other Methods
    27

    View Slide

  28. Copyright © NTT Communications Corporation. All rights reserved.
    ◼ implement multi-vendor topology model using augment
    • can add new device, change topology easily
    ◼ control ref-count using redeploy and reconcile
    • operate by hands and by NSO at same time
    • avoid implementing difficult and rare situation
    Conclusion
    28

    View Slide