Network Source of Truth and Infrastructure as Code revisited - Autocon0

Slide 1

Slide 1 text

Your logo here Network Source of Truth and Infrastructure as Code Revisited Damien Garros, OpsMill Denver November 2023

Slide 2

Slide 2 text

About me : Damien Garros @dgarros damiengarros @damgarros 2 Co-Founder and CEO of Focused on Infrastructure as Code, Automation & Observability for 10+ years Previously leading Technical Architecture at Network to Code

Slide 3

Slide 3 text

Agenda ● State of the Industry ● Analysis ● What can we improve ? ● Our vision of the future 3

Slide 4

Slide 4 text

Current State of the Industry

Slide 5

Slide 5 text

Popular architecture to automate networks Source of Truth Observability Telemetry SLA Compliance Reporting Service Catalogue User Interface Data Governance Deployment Automation Infrastructure Management Platform Network Source of Truth Network Source of Truth DB First Git First Infrastructure as Code 5

Slide 6

Slide 6 text

Database First / DB First ● Data stored in traditional DB ● Schema / API / Query engine ● Templates, Playbooks and some data in Git ● API Integration with other tools ● Deployment with CI/CD Pipeline (optional) Network Source of Truth Deploy Render Transform Data Observability CMDB CI/CD Pull Data Sync 6

Slide 7

Slide 7 text

Infrastructure as Code - Git First ● Everything in Git (YAML) ● All changes requires a Pull Request ● Deployment with CI/CD Pipeline CI/CD Deploy Render Transform Network Source of Truth 7

Slide 8

Slide 8 text

The Power of Automation 8

Slide 9

Slide 9 text

Hype Cycle 9

Slide 10

Slide 10 text

Limitations of DB First approach ● No Version Control ● Complicated to maintain ● Can be hard to extend the schema ● Partial Integration with CI/CD Pipeline Network Source of Truth Deploy Render Transform Data Observability CMDB CI/CD Pull Data Sync 10

Slide 11

Slide 11 text

Limitations of Git First approach ● No API ● Hard to manage the data (no schema, duplicate) ● Pipeline Slow and Error Prone CI/CD Deploy Render Transform Network Source of Truth 11

Slide 12

Slide 12 text

Different Capabilities per Solution User Experience External Integration Schema & Query Engine Isolated Environment Peer Review Complex Changes - - DB First Git First - 12

Slide 13

Slide 13 text

Schema & Query Engine 13

Slide 14

Slide 14 text

A bit of history 2021 GraphQL Simplify how to interact with the data 2016 Netbox brings IPAM, DCIM & Circuit data together 2020 Schema Extensibility Circuit Maintenance Device Lifecycle BGP Security 14

Slide 15

Slide 15 text

Infrastructure Graph Beneﬁts of an Infrastructure Graph ● Documentation ● Reduce inconsistency / errors ● Simplify Conﬁg management ● Enrich Observability Data ● Simplify Troubleshooting, root cause analysis DCIM Circuit IPAM Security Routing Session Maintenance Service Design Business Technical / Implementation Models Design / Service / Business Models Status 15

Slide 16

Slide 16 text

Knowledge Graphs in other Industries Knowledge Graph : Deﬁnition A knowledge graph is a structured representation of interconnected data that captures relationships and semantic meaning, facilitating efﬁcient data retrieval, analysis, and inference. ChatGPT 2023 Retail Industry Pharma Cyber Security 16

Slide 17

Slide 17 text

Analysis

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Running Infrastructure Keep The Lights On Deploy new services to support the business Standard Changes Non Standard Changes Troubleshooting Main Missions Tasks 19

Slide 20

Slide 20 text

Differents type of changes Add Vlan Standardized Changes Drain Device Hardware Refresh Design Change Add Service Non-Standardized Changes User Form - External Integration Isolated Environment Peer Review Continuous Integration - - - - - 20

Slide 21

Slide 21 text

Use Cases & Solutions don’t match - DB First Git First Standardized Changes Non-Standardized Changes Troubleshooting 21

Slide 22

Slide 22 text

Business Requirements UI API Schema & Query Engine Version Control Uniﬁed Storage User Experience, everyone can use it Easy integration, programmability Data Integrity, Context Reduces Risk, Traceability Atomic changes, Reduces risk 22

Slide 23

Slide 23 text

What can we improve ?

Slide 24

Slide 24 text

What can we improve ● Flexible and Extensible Schema ● Version Control ● Uniﬁed Storage ● CI/CD Pipeline 24

Slide 25

Slide 25 text

Flexible & Extensible Schema

Slide 26

Slide 26 text

Schema Extensibility ● Every organization has slightly different requirements ● Schema Extensibility is paramount ● Git is also popular because it imposes no restriction for the schema Circuit Routing Session Maintenance Business Technical / Implementation Models Design / Service / Business Models Status DCIM IPAM Security Service Design LB 26

Slide 27

Slide 27 text

Key Characteristics for a Knowledge Graph ● Graph Structure ● Interconnectedness ● Schema Driven ● Extensibility ● Inference and Reasoning ● Semantics and Context ● Query Engine Graph Databases brings Flexible & Extensible Schema together with a powerful query engine 27

Slide 28

Slide 28 text

Relational vs Graph 28

Slide 29

Slide 29 text

Version Control

Slide 30

Slide 30 text

Version Control & Git Git is Awesome ● Branching / Difﬁng & Merging ● History Tracking ● Collaboration / Peer Review ● Prepare Multiple Changes Mandatory for complex changes BUT No query engine No understanding of what it stores Only work with text ﬁles Git is not a database 30

Slide 31

Slide 31 text

Data Version Control ● Several solutions available to have version control for your data ● Create a Branch on your database !! ● Query each branch individually 31

Slide 32

Slide 32 text

Nautobot + Dolt 32

Slide 33

Slide 33 text

Uniﬁed Storage

Slide 34

Slide 34 text

Tight Coupling challenges with DB First ● Complicated to upgrade the Data Model ● Rendering and the Transformation are tightly coupled with the internal data model / API ● Transformation & Rendering have a lot of business logic Data Deploy Render Transform Internal data model CI / CD Source of Truth Device Speciﬁc language Tight Coupling 34

Slide 35

Slide 35 text

Git First has a uniﬁed storage Git First / Infrastructure as Code makes it easy to manage changes because : ● Data ● Transformation ● Rendering can be managed together CI/CD Deploy Render Transform Network Source of Truth Uniﬁed Storage One PR One Pipeline One Review process Atomic Change 35

Slide 36

Slide 36 text

Redeﬁne the limit of the Source of Truth Manage Data, Transformation and Rendering together Transformation & Rendering have a lot of business logic and should be part of the Source of Truth Data Deploy Render Transform Device Speciﬁc language CI CD Source of Truth Copyright @2023 OpsMill 36

Slide 37

Slide 37 text

CI/CD Pipeline

Slide 38

Slide 38 text

CI/CD pipelines aren’t designed for Infrastructure Management Git doesn’t understand what has changed, it only sees ﬁles 38

Slide 39

Slide 39 text

CI/CD pipelines for infrastructure management Copyright @2023 OpsMill Entity to manage Data structure in Yaml Frequence of Deployment Application Infrastructure 1 Application 100s devices No Yes Seldom Frequent 39

Slide 40

Slide 40 text

Limitations of CI/CD pipelines for infrastructure management Copyright @2023 OpsMill Validation Deployment Usually we validate everything for all changes just to be sure. Difficult to deploy something quickly No workflow possible Partial deployment difficult to manage 40

Slide 41

Slide 41 text

Our vision of the future

Slide 42

Slide 42 text

Infrahub A new generation of Source of Truth Copyright @2023 OpsMill Unified Storage w/ Version Control Infrahub is a new type of Source of Truth, built from the ground up for infrastructure management & automation Data Files Config Rendering Artifact Peer Review CI Pipeline User Defined Schema UI, API, GraphQL 42

Slide 43

Slide 43 text

Infrahub Unified Storage Copyright @2023 OpsMill Unified Storage w/ Version Control Data Files User Defined Schema Dynamic and flexible Schema Immutable and Version Control Graph Branches Diff Merge Built on a Graph Database and a temporal Graph Standard Git Integrated with the Graph Templates Scripts CI Pipelines 43