GraphConnect 2013 - Modeling IT infrastructure using the Assimilation Project

A talk on using Neo4j to model IT infrastructure given at the 2013 GraphConnect conference in San Francisco.

Transcript

1. G R A P H C O N N E C T Modeling IT Infrastucture using The Assimilation Project #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimGC2013 Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com
2. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 2/25 G R A P H C O N N E C T Upcoming Events GraphConnect San Francisco (today!) Open Source Monitoring Conference - Nürnberg NSA / Homeland Security Assimilation Technical Talk Large Installation System Administration Conference - DC Colorado Springs Open Source User’s Group linux.conf.au – Awesome Australian Linux Conf - Perth Details on http://assimilationsystems.com/
3. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 3/25 G R A P H C O N N E C T Talk Outline • Project Overview • Infrastructure Schema • Python Object-Graph-Mapping API
4. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 4/25 G R A P H C O N N E C T Assimilation Project History • Inspired by 2 million core computer (cyclops64) • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues =►Discovery of everything!
5. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 5/25 G R A P H C O N N E C T Project Scope Zero-network-footprint continuous Discovery integrated with extreme-scale Monitoring • Continuous extensible discovery – systems, switches, services, dependencies – zero network footprint • Extensible exception monitoring – more than 100K systems • All data goes into central graph database
6. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 6/25 G R A P H C O N N E C T Discovery Discovering • systems you've forgotten • what you're not monitoring • whatever you'd like • without setting off security alarms
7. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 7/25 G R A P H C O N N E C T Why Discovery? • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Mgmt DataBase) Our Discovery: continuous, low-profile
8. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 8/25 G R A P H C O N N E C T Why Neo4j (graph db)? • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization of relationships • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
9. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 9/25 G R A P H C O N N E C T Assimilation Communication Neighbor-Rings
10. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 10/25 G R A P H C O N N E C T Ring Representation Schema
11. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 11/25 G R A P H C O N N E C T ssh -> sshd dependency graph
12. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 12/25 G R A P H C O N N E C T Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON
13. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 13/25 G R A P H C O N N E C T OGM – Object Graph Mapping • Managing the Graph Nodes “disappears” • The Object Model is the Graph Model • Significant Improvement in Thinking Clarity – The “mud” is gone
14. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 14/25 G R A P H C O N N E C T OGM rules • Don't use Constructors • Relationships replace hash tables and object references and so on • Constructor parameter names match attribute names
15. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 15/25 G R A P H C O N N E C T OGM sample @RegisterGraphClass class Person(GraphNode): 'A Person - or at least an electronic representation of one' def __init__(self, firstname, lastname, dateofbirth=None): GraphNode.__init__(self, domain='global') self.firstname = firstname self.lastname = lastname if dateofbirth is not None: self.dateofbirth = dateofbirth else: self.dateofbirth='unknown' @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['lastname', 'firstname']
16. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 16/25 G R A P H C O N N E C T OGM sample @RegisterGraphClass class TestSystem(GraphNode): 'Some kind of semi-intelligent system' def __init__(self, designation, roles=[]): GraphNode.__init__(self, domain) self.designation = designation.lower() if roles == []: roles = [''] self.roles = roles def addroles(self, role): if self.roles[0] == '''': del self.roles[0] if isinstance(role, list): for arole in role: self.addroles(arole) elif role not in self.roles: self.roles.append(role)
17. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 17/25 G R A P H C O N N E C T OGM sample @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['designation']
18. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 18/25 G R A P H C O N N E C T OGM sample # (seven)-[:formerly]->(Annika) Annika = store.load_or_create(Person, firstname='Annika', lastname='Hansen') seven = store.load_or_create(Drone, designation='SevenOfNine', roles='Borg') store.relate(seven, 'formerly', Annika) store.commit()
19. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 19/25 G R A P H C O N N E C T OGM sample @RegisterGraphClass class TestDrone(TestSystem): def __init__(self, designation, roles=[]): TestSystem.__init__(self, designation=designation) if isinstance(roles, str): roles = [roles] roles.extend(['host', 'Drone']) System.__init__(self, designation, roles=roles)
20. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 20/25 G R A P H C O N N E C T Current State • First release was April 2013 • Great unit test infrastructure • Nanoprobe code – works well • Service monitoring works • Lacks digital signatures, encryption, compression • Reliable UDP comm code working • Several discovery methods written • CMA and database code restructuring near-complete • UI development underway • Licensed under the GPL, commercial license available
21. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 21/25 G R A P H C O N N E C T Future Plans • Production grade by end of year • Purchased support • “Real digital signatures, compression, encryption • Other security enhancements • Much more discovery • GUI • Alerting • Reporting • Add Statistical Monitoring • Best Practice Audits • Dynamic (aka cloud) specialization • Hundreds more ideas – See: https://trello.com/b/OpaED3AT
22. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 22/25 G R A P H C O N N E C T Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill • Awesome User Interfaces (UI/UX) • Evangelism, community building • Test Code (simulate 106 servers!) • Python, C, script coding • Documentation • Feedback: Testing, Ideas, Plans • Many others!
23. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 23/25 G R A P H C O N N E C T Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
24. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 24/25 G R A P H C O N N E C T The Elder GeekGirl
25. GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 25/25 G R A P H C O N N E C T Younger GeekGirl's Computer Running Linux Of Course!