A talk on using Neo4j to model IT infrastructure given at the 2013 GraphConnect conference in San Francisco.
GRAPHCONNECTModeling IT InfrastuctureusingThe Assimilation Project#AssimProj @OSSAlanRhttp://assimproj.org/http://bit.ly/AssimGC2013Alan Robertson Assimilation Systems Limitedhttp://assimilationsystems.com
View Slide
GraphConnect4 October2013© 2013 Assimilation Systems Limited 2/25GRAPHCONNECTUpcoming EventsGraphConnect San Francisco (today!)Open Source Monitoring Conference - NürnbergNSA / Homeland Security Assimilation Technical TalkLarge Installation System Administration Conference - DCColorado Springs Open Source User’s Grouplinux.conf.au – Awesome Australian Linux Conf - PerthDetails on http://assimilationsystems.com/
GraphConnect4 October2013© 2013 Assimilation Systems Limited 3/25GRAPHCONNECTTalk Outline●Project Overview●Infrastructure Schema●Python Object-Graph-Mapping API
GraphConnect4 October2013© 2013 Assimilation Systems Limited 4/25GRAPHCONNECTAssimilation Project History●Inspired by 2 million core computer (cyclops64)●Concerns for extreme scale●Topology aware monitoring●Topology discovery w/out security issues=►Discovery of everything!
GraphConnect4 October2013© 2013 Assimilation Systems Limited 5/25GRAPHCONNECTProject ScopeZero-network-footprint continuous Discoveryintegrated with extreme-scale Monitoring●Continuous extensible discovery– systems, switches, services, dependencies– zero network footprint●Extensible exception monitoring– more than 100K systems●All data goes into central graph database
GraphConnect4 October2013© 2013 Assimilation Systems Limited 6/25GRAPHCONNECTDiscoveryDiscovering●systems you've forgotten●what you're not monitoring●whatever you'd like●without setting off security alarms
GraphConnect4 October2013© 2013 Assimilation Systems Limited 7/25GRAPHCONNECTWhy Discovery?●Documentation: incomplete, incorrect●Dependencies: unknown●Planning: Needs accurate data●Best Practices: Verification needsdata●ITIL CMDB (Configuration MgmtDataBase)Our Discovery: continuous, low-profile
GraphConnect4 October2013© 2013 Assimilation Systems Limited 8/25GRAPHCONNECTWhy Neo4j (graph db)?●Dependency & Discovery information: graph●Speed of graph traversals depends on sizeof subgraph, not total graph size●Root cause queries graph traversals –notoriously slow in relational databases●Visualization of relationships●Schema-less design: good for constantlychanging heterogeneous environment●Graph Model === Object Model
GraphConnect4 October2013© 2013 Assimilation Systems Limited 9/25GRAPHCONNECTAssimilation CommunicationNeighbor-Rings
GraphConnect4 October2013© 2013 Assimilation Systems Limited 10/25GRAPHCONNECTRing Representation Schema
GraphConnect4 October2013© 2013 Assimilation Systems Limited 11/25GRAPHCONNECTssh -> sshd dependency graph
GraphConnect4 October2013© 2013 Assimilation Systems Limited 12/25GRAPHCONNECTSwitch Discovery Datafrom LLDP (or CDP)CRM transforms LLDP (CDP) Data to JSON
GraphConnect4 October2013© 2013 Assimilation Systems Limited 13/25GRAPHCONNECTOGM – Object Graph Mapping●Managing the Graph Nodes “disappears”●The Object Model is the Graph Model●Significant Improvement in ThinkingClarity– The “mud” is gone
GraphConnect4 October2013© 2013 Assimilation Systems Limited 14/25GRAPHCONNECTOGM rules●Don't use Constructors●Relationships replace hash tables andobject references and so on●Constructor parameter names matchattribute names
GraphConnect4 October2013© 2013 Assimilation Systems Limited 15/25GRAPHCONNECTOGM sample@RegisterGraphClassclass Person(GraphNode):'A Person - or at least an electronic representation of one'def __init__(self, firstname, lastname, dateofbirth=None):GraphNode.__init__(self, domain='global')self.firstname = firstnameself.lastname = lastnameif dateofbirth is not None:self.dateofbirth = dateofbirthelse:self.dateofbirth='unknown'@staticmethoddef __meta_keyattrs__():'Return our key attributes in order of significance'return ['lastname', 'firstname']
GraphConnect4 October2013© 2013 Assimilation Systems Limited 16/25GRAPHCONNECTOGM sample@RegisterGraphClassclass TestSystem(GraphNode):'Some kind of semi-intelligent system'def __init__(self, designation, roles=[]):GraphNode.__init__(self, domain)self.designation = designation.lower()if roles == []:roles = ['']self.roles = rolesdef addroles(self, role):if self.roles[0] == '''':del self.roles[0]if isinstance(role, list):for arole in role:self.addroles(arole)elif role not in self.roles:self.roles.append(role)
GraphConnect4 October2013© 2013 Assimilation Systems Limited 17/25GRAPHCONNECTOGM sample@staticmethoddef __meta_keyattrs__():'Return our key attributes in order of significance'return ['designation']
GraphConnect4 October2013© 2013 Assimilation Systems Limited 18/25GRAPHCONNECTOGM sample# (seven)-[:formerly]->(Annika)Annika = store.load_or_create(Person,firstname='Annika',lastname='Hansen')seven = store.load_or_create(Drone,designation='SevenOfNine', roles='Borg')store.relate(seven, 'formerly', Annika)store.commit()
GraphConnect4 October2013© 2013 Assimilation Systems Limited 19/25GRAPHCONNECTOGM sample@RegisterGraphClassclass TestDrone(TestSystem):def __init__(self, designation, roles=[]):TestSystem.__init__(self, designation=designation)if isinstance(roles, str):roles = [roles]roles.extend(['host', 'Drone'])System.__init__(self, designation, roles=roles)
GraphConnect4 October2013© 2013 Assimilation Systems Limited 20/25GRAPHCONNECTCurrent State●First release was April 2013●Great unit test infrastructure●Nanoprobe code – works well●Service monitoring works●Lacks digital signatures, encryption, compression●Reliable UDP comm code working●Several discovery methods written●CMA and database code restructuring near-complete●UI development underway●Licensed under the GPL, commercial license available
GraphConnect4 October2013© 2013 Assimilation Systems Limited 21/25GRAPHCONNECTFuture Plans●Production grade by end of year●Purchased support●“Real digital signatures, compression, encryption●Other security enhancements●Much more discovery●GUI●Alerting●Reporting●Add Statistical Monitoring●Best Practice Audits●Dynamic (aka cloud) specialization●Hundreds more ideas– See: https://trello.com/b/OpaED3AT
GraphConnect4 October2013© 2013 Assimilation Systems Limited 22/25GRAPHCONNECTGet Involved!Powerful Ideas and InfrastuctureFun, ground-breaking projectLooking for early adopters, testers!!Needs for every kind of skill●Awesome User Interfaces (UI/UX)●Evangelism, community building●Test Code (simulate 106 servers!)●Python, C, script coding●Documentation●Feedback: Testing, Ideas, Plans●Many others!
GraphConnect4 October2013© 2013 Assimilation Systems Limited 23/25GRAPHCONNECTResistance Is Futile!#AssimProj @OSSAlanR#AssimMonProject Web Sitehttp://assimproj.orgBlogtechthoughts.typepad.comlists.community.tummy.com/cgi-bin/mailman/admin/assimilation
GraphConnect4 October2013© 2013 Assimilation Systems Limited 24/25GRAPHCONNECTThe Elder GeekGirl
GraphConnect4 October2013© 2013 Assimilation Systems Limited 25/25GRAPHCONNECTYounger GeekGirl's ComputerRunning LinuxOf Course!