LCA2014 Transparently using a Graph Database as a Python Object Store with OGM - Open Programming Miniconf:

The Assimilation Project uses neo4j to store its data, and we use Python to access the data. We developed a transactional API which allows us to simply persist our Python objects to Neo4j with little fanfare. This API is a new implementation modelled after Nigel Small's py2neo OGM API.

Transcript

1. L
   C
   A
   2
   0
   1
   4
   Transparently using a Graph
   Database as a Python Object
   Store with OGM
   #AssimProj @OSSAlanR
   http://assimproj.org/
   http://bit.ly/AssimLCAprog
   Alan Robertson
   Assimilation Systems Limited
   http://assimilationsystems.com
   

   View Slide

2. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 2/24
   L
   C
   A
   2
   0
   1
   4
   Assimilation Project Scope
   Zero-network-footprint continuous Discovery
   Integrated with extreme-scale Monitoring
   Discovery creates a graph-based CMDB
   

   View Slide

3. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 3/24
   L
   C
   A
   2
   0
   1
   4
   Why Neo4j (graph db)?
   ●
   Dependency & Discovery information: graph
   ●
   Speed of graph traversals depends on size
   of subgraph, not total graph size
   ●
   Root cause queries  graph traversals –
   notoriously slow in relational databases
   ●
   Visualization of relationships
   ●
   Schema-less design: good for constantly
   changing heterogeneous environment
   ●
   Graph Model === Object Model
   

   View Slide

4. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 4/24
   L
   C
   A
   2
   0
   1
   4
   ssh -> sshd dependency graph
   

   View Slide

5. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 5/24
   L
   C
   A
   2
   0
   1
   4
   Switch Discovery Data
   from LLDP (or CDP)
   CRM transforms LLDP (CDP) Data to JSON
   

   View Slide

6. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 6/24
   L
   C
   A
   2
   0
   1
   4
   OGM – Object Graph Mapping
   ●
   Managing the Graph db “disappears”
   ●
   The Object Model is the Graph Model
   ●
   Significant Improvement in Thinking
   Clarity
   – The “mud” is gone
   ●
   Based on work by Nigel Small – but fresh
   implementation – a work in progress
   

   View Slide

7. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 7/24
   L
   C
   A
   2
   0
   1
   4
   Why is this cool?
   ●
   You mostly just write your objects
   ●
   Modify attributes, etc at will
   ●
   Create relationships between objects
   ●
   Commit changes at end of transaction
   ●
   Queries yield your objects
   

   View Slide

8. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 8/24
   L
   C
   A
   2
   0
   1
   4
   Class hierarchy for sample
   ●
   Person – represents a human
   ●
   System – represents a system
   – Drone – a Borg drone System
   My apologies to the Star Trek Universe
   

   View Slide

9. LCA prog
   miniconf
   7 January
   2014
   © 2013 Assimilation Systems Limited 9/24
   L
   C
   A
   2
   0
   1
   4
   OGM sample
   @RegisterGraphClass
   class Person(GraphNode):
   def __init__(self, firstname, lastname):
   GraphNode.__init__(self)
   self.firstname = firstname
   self.lastname = lastname
   @staticmethod
   def __meta_keyattrs__():
   'Return key attributes in order of significance'
   return ['lastname', 'firstname']
   

   View Slide

10. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 10/24
    L
    C
    A
    2
    0
    1
    4
    OGM sample (2)
    @RegisterGraphClass
    class System(GraphNode):
    def __init__(self, designation, roles=[]):
    GraphNode.__init__(self)
    self.designation = designation.lower()
    etc...
    def addroles(self, role):
    etc...
    @staticmethod
    def __meta_keyattrs__():
    'Return key attributes in order of significance'
    return ['designation']
    

    View Slide

11. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 11/24
    L
    C
    A
    2
    0
    1
    4
    OGM sample (3)
    @RegisterGraphClass
    class Drone(System):
    def __init__(self, designation, roles=None):
    System.__init__(self, designation=designation
    , roles=roles)
    self.addroles('drone')
    

    View Slide

12. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 12/24
    L
    C
    A
    2
    0
    1
    4
    OGM sample (4)
    # (seven)-[:formerly]->(Annika)
    store = Store(neo4j_db, etc...)
    Annika = store.load_or_create(Person,
    firstname='Annika',
    lastname='Hansen')
    seven = store.load_or_create(Drone,
    designation='SevenOfNine', roles='Borg')
    store.relate(seven, 'formerly', Annika)
    store.commit()
    

    View Slide

13. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 13/24
    L
    C
    A
    2
    0
    1
    4
    Resulting Graph from (4)
    formerly
    nodetype='Drone'
    designation='sevenofnine'
    roles=['Borg', 'drone']
    nodetype='Person'
    firstname='Annika'
    lastname='Hansen'
    

    View Slide

14. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 14/24
    L
    C
    A
    2
    0
    1
    4
    More OGM sample (5)
    # (seven)-[:formerly]->(Annika)
    seven.assignment = 'astrogation'
    Annika.sex='female'
    store.commit()
    who = store.load_related(seven, 'formerly', Person)
    assert who is Annika
    

    View Slide

15. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 15/24
    L
    C
    A
    2
    0
    1
    4
    Resulting Graph (from 5)
    formerly
    nodetype='Drone'
    designation='sevenofnine'
    roles=['borg', 'drone']
    assignment='astrogation'
    nodetype='Person'
    firstname='Annika'
    lastname='Hansen'
    sex='female'
    

    View Slide

16. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 16/24
    L
    C
    A
    2
    0
    1
    4
    Alan's OGM rules
    ●
    Cannot call Constructors directly
    ●
    Relationships replace hash tables and object
    references and so on
    ●
    Constructor parameter names match attribute
    names
    ●
    Attributes that you don't want (or can't put) in
    the database start with “_”.
    ●
    Attributes in the database must be of types
    supported by Neo4j
    – single type (non-empty) arrays
    – no dicts
    

    View Slide

17. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 17/24
    L
    C
    A
    2
    0
    1
    4
    More OGM rules
    ●
    Must be a field which represents the class
    of the graph node (i.e.,nodetype)
    ●
    Need a “factory constructor” which looks
    at the nodetype and constructs the right
    kind of object from the graph attributes
    ●
    Subclass objects must be indexable by
    the same key as the superclass
    

    View Slide

18. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 18/24
    L
    C
    A
    2
    0
    1
    4
    Things I wish I had time to fix
    ●
    Reorganize “things”
    ●
    Require all graphs to be fully indexed
    ●
    Add a “default constructor” for the Store
    ●
    Make all my classes subclasses of Store
    instead of associated with Stores
    ●
    Need to tie it into the new indexing
    methods – with class and subclass
    indexing
    

    View Slide

19. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 19/24
    L
    C
    A
    2
    0
    1
    4
    Current Assimilation State
    ●
    First release was April 2013
    ●
    Great unit test infrastructure
    ●
    Nanoprobe code – works well
    ●
    Service monitoring works
    ●
    Lacks digital signatures, encryption, compression
    ●
    Reliable UDP comm code working
    ●
    Several discovery methods written
    ●
    CMA and database code restructuring near-complete
    ●
    UI development underway
    ●
    Licensed under the GPL, commercial license available
    

    View Slide

20. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 20/24
    L
    C
    A
    2
    0
    1
    4
    Future Plans
    ●
    Production grade by end of year
    ●
    Purchased support
    ●
    “Real digital signatures, compression, encryption
    ●
    Other security enhancements
    ●
    Much more discovery
    ●
    GUI
    ●
    Alerting
    ●
    Reporting
    ●
    Add Statistical Monitoring
    ●
    Best Practice Audits
    ●
    Dynamic (aka cloud) specialization
    ●
    Hundreds more ideas
    – See: https://trello.com/b/OpaED3AT
    

    View Slide

21. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 21/24
    L
    C
    A
    2
    0
    1
    4
    Get Involved!
    Powerful Ideas and Infrastucture
    Fun, ground-breaking project
    Looking for early adopters, testers!!
    Needs for every kind of skill
    ●
    Awesome User Interfaces (UI/UX)
    ●
    Evangelism, community building
    ●
    Test Code (simulate 106 servers!)
    ●
    Python, C, script coding
    ●
    Documentation
    ●
    Feedback: Testing, Ideas, Plans
    ●
    Many others!
    

    View Slide

22. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 22/24
    L
    C
    A
    2
    0
    1
    4
    Resistance Is Futile!
    #AssimProj @OSSAlanR
    #AssimMon
    Project Web Site
    http://assimproj.org
    Blog
    techthoughts.typepad.com
    lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
    

    View Slide

23. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 23/24
    L
    C
    A
    2
    0
    1
    4
    The Elder GeekGirl
    

    View Slide

24. LCA prog
    miniconf
    7 January
    2014
    © 2013 Assimilation Systems Limited 24/24
    L
    C
    A
    2
    0
    1
    4
    Younger GeekGirl's Computer
    Running Linux
    Of Course!
    

    View Slide