LCA2014 Transparently using a Graph Database as a Python Object Store with OGM - Open Programming Miniconf:

The Assimilation Project uses neo4j to store its data, and we use Python to access the data. We developed a transactional API which allows us to simply persist our Python objects to Neo4j with little fanfare. This API is a new implementation modelled after Nigel Small's py2neo OGM API.

Transcript

1. L C A 2 0 1 4 Transparently using a Graph Database as a Python Object Store with OGM #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimLCAprog Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com
2. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 2/24 L C A 2 0 1 4 Assimilation Project Scope Zero-network-footprint continuous Discovery Integrated with extreme-scale Monitoring Discovery creates a graph-based CMDB
3. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 3/24 L C A 2 0 1 4 Why Neo4j (graph db)? • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization of relationships • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
4. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 4/24 L C A 2 0 1 4 ssh -> sshd dependency graph
5. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 5/24 L C A 2 0 1 4 Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON
6. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 6/24 L C A 2 0 1 4 OGM – Object Graph Mapping • Managing the Graph db “disappears” • The Object Model is the Graph Model • Significant Improvement in Thinking Clarity – The “mud” is gone • Based on work by Nigel Small – but fresh implementation – a work in progress
7. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 7/24 L C A 2 0 1 4 Why is this cool? • You mostly just write your objects • Modify attributes, etc at will • Create relationships between objects • Commit changes at end of transaction • Queries yield your objects
8. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 8/24 L C A 2 0 1 4 Class hierarchy for sample • Person – represents a human • System – represents a system – Drone – a Borg drone System My apologies to the Star Trek Universe
9. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 9/24 L C A 2 0 1 4 OGM sample @RegisterGraphClass class Person(GraphNode): def __init__(self, firstname, lastname): GraphNode.__init__(self) self.firstname = firstname self.lastname = lastname @staticmethod def __meta_keyattrs__(): 'Return key attributes in order of significance' return ['lastname', 'firstname']
10. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 10/24 L C A 2 0 1 4 OGM sample (2) @RegisterGraphClass class System(GraphNode): def __init__(self, designation, roles=[]): GraphNode.__init__(self) self.designation = designation.lower() etc... def addroles(self, role): etc... @staticmethod def __meta_keyattrs__(): 'Return key attributes in order of significance' return ['designation']
11. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 11/24 L C A 2 0 1 4 OGM sample (3) @RegisterGraphClass class Drone(System): def __init__(self, designation, roles=None): System.__init__(self, designation=designation , roles=roles) self.addroles('drone')
12. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 12/24 L C A 2 0 1 4 OGM sample (4) # (seven)-[:formerly]->(Annika) store = Store(neo4j_db, etc...) Annika = store.load_or_create(Person, firstname='Annika', lastname='Hansen') seven = store.load_or_create(Drone, designation='SevenOfNine', roles='Borg') store.relate(seven, 'formerly', Annika) store.commit()
13. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 13/24 L C A 2 0 1 4 Resulting Graph from (4) formerly nodetype='Drone' designation='sevenofnine' roles=['Borg', 'drone'] nodetype='Person' firstname='Annika' lastname='Hansen'
14. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 14/24 L C A 2 0 1 4 More OGM sample (5) # (seven)-[:formerly]->(Annika) seven.assignment = 'astrogation' Annika.sex='female' store.commit() who = store.load_related(seven, 'formerly', Person) assert who is Annika
15. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 15/24 L C A 2 0 1 4 Resulting Graph (from 5) formerly nodetype='Drone' designation='sevenofnine' roles=['borg', 'drone'] assignment='astrogation' nodetype='Person' firstname='Annika' lastname='Hansen' sex='female'
16. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 16/24 L C A 2 0 1 4 Alan's OGM rules • Cannot call Constructors directly • Relationships replace hash tables and object references and so on • Constructor parameter names match attribute names • Attributes that you don't want (or can't put) in the database start with “_”. • Attributes in the database must be of types supported by Neo4j – single type (non-empty) arrays – no dicts
17. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 17/24 L C A 2 0 1 4 More OGM rules • Must be a field which represents the class of the graph node (i.e.,nodetype) • Need a “factory constructor” which looks at the nodetype and constructs the right kind of object from the graph attributes • Subclass objects must be indexable by the same key as the superclass
18. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 18/24 L C A 2 0 1 4 Things I wish I had time to fix • Reorganize “things” • Require all graphs to be fully indexed • Add a “default constructor” for the Store • Make all my classes subclasses of Store instead of associated with Stores • Need to tie it into the new indexing methods – with class and subclass indexing
19. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 19/24 L C A 2 0 1 4 Current Assimilation State • First release was April 2013 • Great unit test infrastructure • Nanoprobe code – works well • Service monitoring works • Lacks digital signatures, encryption, compression • Reliable UDP comm code working • Several discovery methods written • CMA and database code restructuring near-complete • UI development underway • Licensed under the GPL, commercial license available
20. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 20/24 L C A 2 0 1 4 Future Plans • Production grade by end of year • Purchased support • “Real digital signatures, compression, encryption • Other security enhancements • Much more discovery • GUI • Alerting • Reporting • Add Statistical Monitoring • Best Practice Audits • Dynamic (aka cloud) specialization • Hundreds more ideas – See: https://trello.com/b/OpaED3AT
21. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 21/24 L C A 2 0 1 4 Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill • Awesome User Interfaces (UI/UX) • Evangelism, community building • Test Code (simulate 106 servers!) • Python, C, script coding • Documentation • Feedback: Testing, Ideas, Plans • Many others!
22. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 22/24 L C A 2 0 1 4 Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
23. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 23/24 L C A 2 0 1 4 The Elder GeekGirl
24. LCA prog miniconf 7 January 2014 © 2013 Assimilation Systems Limited 24/24 L C A 2 0 1 4 Younger GeekGirl's Computer Running Linux Of Course!