Network Source of Truth and Infrastructure as Code revisited - Autocon0

1. Your logo here Network Source of Truth and Infrastructure as Code Revisited Damien Garros, OpsMill Denver November 2023
2. About me : Damien Garros @dgarros damiengarros @damgarros 2 Co-Founder and CEO of Focused on Infrastructure as Code, Automation & Observability for 10+ years Previously leading Technical Architecture at Network to Code
3. Agenda • State of the Industry • Analysis • What can we improve ? • Our vision of the future 3
4. Current State of the Industry
5. Popular architecture to automate networks Source of Truth Observability Telemetry SLA Compliance Reporting Service Catalogue User Interface Data Governance Deployment Automation Infrastructure Management Platform Network Source of Truth Network Source of Truth DB First Git First Infrastructure as Code 5
6. Database First / DB First • Data stored in traditional DB • Schema / API / Query engine • Templates, Playbooks and some data in Git • API Integration with other tools • Deployment with CI/CD Pipeline (optional) Network Source of Truth Deploy Render Transform Data Observability CMDB CI/CD Pull Data Sync 6
7. Infrastructure as Code - Git First • Everything in Git (YAML) • All changes requires a Pull Request • Deployment with CI/CD Pipeline CI/CD Deploy Render Transform Network Source of Truth 7
8. The Power of Automation 8
9. Hype Cycle 9
10. Limitations of DB First approach • No Version Control • Complicated to maintain • Can be hard to extend the schema • Partial Integration with CI/CD Pipeline Network Source of Truth Deploy Render Transform Data Observability CMDB CI/CD Pull Data Sync 10
11. Limitations of Git First approach • No API • Hard to manage the data (no schema, duplicate) • Pipeline Slow and Error Prone CI/CD Deploy Render Transform Network Source of Truth 11
12. Different Capabilities per Solution User Experience External Integration Schema & Query Engine Isolated Environment Peer Review Complex Changes - - DB First Git First - 12
13. Schema & Query Engine 13
14. A bit of history 2021 GraphQL Simplify how to interact with the data 2016 Netbox brings IPAM, DCIM & Circuit data together 2020 Schema Extensibility Circuit Maintenance Device Lifecycle BGP Security 14
15. Infrastructure Graph Benefits of an Infrastructure Graph • Documentation • Reduce inconsistency / errors • Simplify Config management • Enrich Observability Data • Simplify Troubleshooting, root cause analysis DCIM Circuit IPAM Security Routing Session Maintenance Service Design Business Technical / Implementation Models Design / Service / Business Models Status 15
16. Knowledge Graphs in other Industries Knowledge Graph : Definition A knowledge graph is a structured representation of interconnected data that captures relationships and semantic meaning, facilitating efficient data retrieval, analysis, and inference. ChatGPT 2023 Retail Industry Pharma Cyber Security 16
17. Analysis
19. Running Infrastructure Keep The Lights On Deploy new services to support the business Standard Changes Non Standard Changes Troubleshooting Main Missions Tasks 19
20. Differents type of changes Add Vlan Standardized Changes Drain Device Hardware Refresh Design Change Add Service Non-Standardized Changes User Form - External Integration Isolated Environment Peer Review Continuous Integration - - - - - 20
21. Use Cases & Solutions don’t match - DB First Git First Standardized Changes Non-Standardized Changes Troubleshooting 21
22. Business Requirements UI API Schema & Query Engine Version Control Unified Storage User Experience, everyone can use it Easy integration, programmability Data Integrity, Context Reduces Risk, Traceability Atomic changes, Reduces risk 22
23. What can we improve ?
24. What can we improve • Flexible and Extensible Schema • Version Control • Unified Storage • CI/CD Pipeline 24
25. Flexible & Extensible Schema
26. Schema Extensibility • Every organization has slightly different requirements • Schema Extensibility is paramount • Git is also popular because it imposes no restriction for the schema Circuit Routing Session Maintenance Business Technical / Implementation Models Design / Service / Business Models Status DCIM IPAM Security Service Design LB 26
27. Key Characteristics for a Knowledge Graph • Graph Structure • Interconnectedness • Schema Driven • Extensibility • Inference and Reasoning • Semantics and Context • Query Engine Graph Databases brings Flexible & Extensible Schema together with a powerful query engine 27
28. Relational vs Graph 28
29. Version Control
30. Version Control & Git Git is Awesome • Branching / Diffing & Merging • History Tracking • Collaboration / Peer Review • Prepare Multiple Changes Mandatory for complex changes BUT No query engine No understanding of what it stores Only work with text files Git is not a database 30
31. Data Version Control • Several solutions available to have version control for your data • Create a Branch on your database !! • Query each branch individually 31
32. Nautobot + Dolt 32
33. Unified Storage
34. Tight Coupling challenges with DB First • Complicated to upgrade the Data Model • Rendering and the Transformation are tightly coupled with the internal data model / API • Transformation & Rendering have a lot of business logic Data Deploy Render Transform Internal data model CI / CD Source of Truth Device Specific language Tight Coupling 34
35. Git First has a unified storage Git First / Infrastructure as Code makes it easy to manage changes because : • Data • Transformation • Rendering can be managed together CI/CD Deploy Render Transform Network Source of Truth Unified Storage One PR One Pipeline One Review process Atomic Change 35
36. Redefine the limit of the Source of Truth Manage Data, Transformation and Rendering together Transformation & Rendering have a lot of business logic and should be part of the Source of Truth Data Deploy Render Transform Device Specific language CI CD Source of Truth Copyright @2023 OpsMill 36
37. CI/CD Pipeline
38. CI/CD pipelines aren’t designed for Infrastructure Management Git doesn’t understand what has changed, it only sees files 38
39. CI/CD pipelines for infrastructure management Copyright @2023 OpsMill Entity to manage Data structure in Yaml Frequence of Deployment Application Infrastructure 1 Application 100s devices No Yes Seldom Frequent 39
40. Limitations of CI/CD pipelines for infrastructure management Copyright @2023 OpsMill Validation Deployment Usually we validate everything for all changes just to be sure. Difficult to deploy something quickly No workflow possible Partial deployment difficult to manage 40
41. Our vision of the future
42. Infrahub A new generation of Source of Truth Copyright @2023 OpsMill Unified Storage w/ Version Control Infrahub is a new type of Source of Truth, built from the ground up for infrastructure management & automation Data Files Config Rendering Artifact Peer Review CI Pipeline User Defined Schema UI, API, GraphQL 42
43. Infrahub Unified Storage Copyright @2023 OpsMill Unified Storage w/ Version Control Data Files User Defined Schema Dynamic and flexible Schema Immutable and Version Control Graph Branches Diff Merge Built on a Graph Database and a temporal Graph Standard Git Integrated with the Graph Templates Scripts CI Pipelines 43
44. Questions?
45. Thank You