The knights who say Neo - storing classes in the graph

Ever wondered what happens if you store your Python classes directly in your database? (And no, we’re not talking about pickling)
When faced with the problem of modelling a large, complex type hierarchy of "stuff one might find in a seven bedroom home", we went down the somewhat unusual route of storing not only the objects themselves, but also their types in a graph database.
In this talk we will take a look under the hood of Kaiso, the open source framework we built to do this, and how we use some metaclass magic to to make it all happen.

Unlike conventional ORMs, we needed the schemas to be as dynamic as the records, and so decided to keep them side by side in our neo4j db. Using the Python type system as our data model means we get features like hierarchy consistency checks and other type calculations for free.

While this might not be the solution to a particularly common problem, come along to see how we’ve used some of the more dynamic aspects of the Python language for fun and profit!

Transcript

1. ©  Lifealike  Limited,  2014. The  knights  who  say  Neo  -­‐  storing  classes  in  the  graph   1 David  Szo:en
 onefinestay   ! ! ! @szo:en   github.com/davidszo:en
2. ©  Lifealike  Limited,  2014. Overview • The  problem   – Modelling  homes   • Some  ingredients   – Graphs  and  dynamic  types   • Our  soluIon   – Kaiso 2
3. ©  Lifealike  Limited,  2014. The  problem Model  homes:   ! • This  home  has  a  sofa  bed  in  the  living  room   • Where  is  the  hoover   • How  many  Beds  does  this  home  have,  and   how  are  they  distributed  across  rooms 3
4. ©  Lifealike  Limited,  2014. Stuff  within  other  stuff Home  has  rooms  which  contain  stuff   SomeImes  stuff  (like  cupboards)  contains  other   stuff   㱺  model  as  a  graph 4
5. ©  Lifealike  Limited,  2014. Ingredient  #1:  Graph  Database Conceptual  model  is  a  graph  as  opposed  to  e.g.  a   collecIon  of  related  tables 5
    
   
   
    
      
6. ©  Lifealike  Limited,  2014. • Looks  like  ascii-­‐art   • Queries  find  pa:erns  in  the  graph Neo4J:  Cypher 6
7. ©  Lifealike  Limited,  2014. Cypher 7 MATCH! (me {name: "David"})-[:FRIEND]->(my_friend)! ! RETURN! me, my_friend!
    
   
   
    
      
8. ©  Lifealike  Limited,  2014. 8 MATCH! (me {name: "David"}),! (expert {knows: "Haskell"}),! path = (me)-[:FRIEND*..3]->(expert)! ! RETURN path!
   
   
   
   
    
        ! {"knows": "Haskell"}! ! {"knows": "Haskell"}!
9. ©  Lifealike  Limited,  2014. I  can  has  ORM? ! ExisIng  (well  OGM,  object  graph  mapper):      -­‐  neomodel  (also  neo4django,  bulbs):   ! Look  like  regular  ORMs  (pseudo-­‐syntax) 9
10. ©  Lifealike  Limited,  2014. ConvenKonal  OGM 10 class Home(Model):! code = String()! ! class Room(Model):! name = String()! # ...! !
11. ©  Lifealike  Limited,  2014. ConvenKonal  OGM 11 class Home(Model):! code = String()! ! class Room(Model):! name = String()! ! class Bed(Model):! size = Choice()! ! class Table(Model):! seats = Integer()! ! class TV(Model):! make = String()! class WasherDryer! class Table! class WiFiRouter! class Jacuzzi! class Xbox! class Fireplace! class Oven! class Garden! class CoffeeMachine! class Lift! class Alarm! class AirCon! class Balcony! class Boiler!
12. ©  Lifealike  Limited,  2014. Currently:  641  types  in  our  taxonomy   12 㱺    We  need  to  make  this  data-­‐driven  
13. ©  Lifealike  Limited,  2014. Data  driven  models? 13 class Stuff(Model):! # what_goes_here?!
14. ©  Lifealike  Limited,  2014. Data  driven  models? 14 class Stuff(Model):! type = RelationshipTo(TypeOfStuff)! ! ! class TypeOfStuff(Model):! attributes = RelationshipFrom(Attribute)! !
15. ©  Lifealike  Limited,  2014. Data  driven  models? • This  would  work,  but  could  we  do  be:er?   ! • Also,  what  about  inheritance?   ! – I'd  like  a  SofaBed  to  be  both  a  Sofa  and  a  Bed 15
16. ©  Lifealike  Limited,  2014. Ingredient  #2:  Dynamic  types   The  `type`  builIn.  This  should  be  familiar:   16 class Spam(object):! ham = "eggs"! ! >>> s = Spam()! >>> type(s)! <class '__main__.Spam'>! !
17. ©  Lifealike  Limited,  2014. Ingredient  #2:  Dynamic  types   Perhaps  less  familiar: 17 >>> Spam = type("Spam", (object,), {"ham": "eggs"})! >>> Spam! <class '__main__.Spam'>! >>> s = Spam()! >>> s! <__main__.Spam object at 0x10c232cd0>! class Spam(object):! ham = "eggs" Equivalent  to
18. ©  Lifealike  Limited,  2014. PuPng  it  all  together:  Kaiso • “Store”  types  in  the  graph   – Well,  enough  data  to  call  type   ! • Best  of  both  worlds:   – Data-­‐driven  taxonomy   – Python  does  the  heavy  lijing 18
19. ©  Lifealike  Limited,  2014. PuPng  it  all  together:  Kaiso 19 from kaiso.attributes import String, Outgoing! from kaiso.persistence import Manager! from kaiso.types import Entity, Relationship! ! ! class Contains(Relationship):! pass! ! class Home(Entity):! code = String(unique=True)! contains = Outgoing(Contains)! ! ! manager = Manager('http://localhost:7474/db/data/')! manager.save(Home)!
20. ©  Lifealike  Limited,  2014. PuPng  it  all  together:  Kaiso 20 >>> my_house = Home(code='house')! >>> manager.save(my_house)! <__main__.Home at 0x105537690>! ! >>> manager.get(Home, code='house')! <__main__.Home at 0x10552ef90>! ! >>> other = manager.get(Home, code='other')! <__main__.Home at 0x10552ed90>! ! >>> other.contains.first()! <kaiso.types.Kitchen at 0x10552edd0>! ! >>> type(other.contains.first())! kaiso.types.Kitchen! ! Bedroom = type(other.contains.first())! >>> Bedroom.__base__! kaiso.types.Room!
21. ©  Lifealike  Limited,  2014. Graph 21 ISA ISA ISA ISA DECLAREDON INSTANCEOF INSTANCEOF INSTANCEOF IN STAN CEO F CO N TAIN S CO NTAINS Entity Home Room Bedroom Kitchen String other house Main Kitchen Master Bedroom
22. ©  Lifealike  Limited,  2014. Nodes 22 {! "name": "Master Bedroom",! }! INSTANCEOF Bedroom Master   Bedroom
23. ©  Lifealike  Limited,  2014. In  closing • Using  Kaiso  we  can   – Store  a  data-­‐driven  type  hierarchy  of  things   – Store  layout  and  contents  of  homes   ! • Use  actual  python  classes  as  our  data  model   – Get  python  type/inheritance  machinery  for  free 23
24. ©  Lifealike  Limited,  2014. Thank  you QuesIons?   ! Kaiso:     github.com/onefinestay/kaiso   ! We’re  Hiring   onefinestay.com/jobs   ! Find  me   github.com/davidszo:en   @davidszo:en 24