LCA2014 Providing a Continuously Updated ITIL CMDB using the Assimilation Project - SysAdmin Miniconf

A 15 minute talk on how the Assimilation Project creates a CMDB and continually keeps it up to date. This was presented at the 2014 linux.conf.au SysAdmin miniconf.

Transcript

1. L C A 2 0 1 4 A Continuously Updated CMDB using The Assimilation Project #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/LCA2014-SysAdmin Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com
2. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 2/18 L C A 2 0 1 4 Assimilation Project Scope Zero-network-footprint continuous Discovery Integrated with extreme-scale Monitoring => Discovery creates a graph-based CMDB
3. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 3/18 L C A 2 0 1 4 Using a CMDB for Risk Management/Mitigation • Intrusions • Licensed Software • Audit Risk • System modeling • Outages
4. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 4/18 L C A 2 0 1 4 Why a Configuration Management Database (CMDB)? • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • Compliance • Our Discovery: continuous, low-profile
5. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 5/18 L C A 2 0 1 4 Discovery Features • Continuous Discovery • Zero network footprint • Discover dependency information • Discovery drives monitoring • Easily extensible • Configuration-free setup (!)
6. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 6/18 L C A 2 0 1 4 What do we discover? • IP and MAC addresses (servers, etc) • Services and service details • Switches, switch connections and settings • Installed services • OS configuration • Whatever you want ;-)
7. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 7/18 L C A 2 0 1 4 Architectural Elements • Collective Management Authority (CMA) – one per installation • Nanoprobes (agents) – one per system
8. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 8/18 L C A 2 0 1 4 How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments from environment • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships
9. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 9/18 L C A 2 0 1 4 OS discovery JSON Snippet { "nodename": "alanr-1225B", "operating-system": "GNU/Linux", "machine": "x86_64", "processor": "x86_64", "hardware-platform": "x86_64", "kernel-name": "Linux", "kernel-release": "3.8.0-31-generic", "kernel-version": "#46-Ubuntu SMP ...", "Distributor ID": "Ubuntu", "Description": "Ubuntu 13.04", "Release": "13.04", "Codename": "raring" }
10. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 10/18 L C A 2 0 1 4 ssh -> sshd dependency graph
11. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 11/18 L C A 2 0 1 4 Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON
12. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 12/18 L C A 2 0 1 4 Current Status • First release April 2013 • Great unit tests • Nanoprobe code works well • Several discovery methods written • Discovery => Automatic Monitoring (WOOT!) • UI development underway • Licensed under GPL: commercial options available
13. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 13/18 L C A 2 0 1 4 Get Involved! We need every talent! • Early adopters (SysAdmins(!)) • Testers • Designers • Developers (C,Python, Shell, PowerShell, JavaScript) • Porters (esp Windows) • Promoters, publicists • Packagers • And so on...
14. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 14/18 L C A 2 0 1 4 Resistance Is Futile! Mailing List bit.ly/AssimML #AssimProj @OSSAlanR Project Web Site assimproj.org Blog techthoughts.typepad.com assimilationsystems.com
15. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 15/18 L C A 2 0 1 4 Why a graph database? (Neo4j) • Humans describe systems as graphs • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization is Natural • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
16. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 16/18 L C A 2 0 1 4 ssh Client JSON Snippet (from netstat and /proc) "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, and so on...
17. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 17/18 L C A 2 0 1 4 sshd Service JSON Snippet (from netstat and /proc) "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, and so on...
18. linux.conf.au 06 January 2014 LCA SysAdmin Miniconf --- © 2014 Assimilation Systems Limited 18/18 L C A 2 0 1 4 A multi-dimensional demo • Demonstrate basic capabilities – Discovery – Automatic monitoring configuration – Monitoring – failures / successes • No configuration was supplied – everything comes from discovery