$30 off During Our Annual Pro Sale. View Details »

Graph Databases: The internal works of Neo4j

Graph Databases: The internal works of Neo4j

Relational databases (e.g. MySQL, PostgreSQL) — ironically — are not that good at the complex relationships some modern applications need. Multiple joins and complex sub-queries can take a toll on performance. Graph Databases are all about relationships, but how do they handle them internally? In this talk I will go through basic graph theory and explain how the popular graph database 'Neo4j' implements these concepts to handle connected data.

Michelle Sanver

November 21, 2014
Tweet

More Decks by Michelle Sanver

Other Decks in Programming

Transcript

  1. @michellesanver Neo4J: Graph Database NoSQL Day 2014

  2. About: Michelle Sanver @michellesanver

  3. About: Michelle Sanver @michellesanver President (Michelle)-[:LOVES]->(Neo4j)

  4. @michellesanver Code Addict (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver

  5. @michellesanver Zürich, Switzerland (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver

  6. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Michelle is_from is_from lives_in lived_in About: Michelle Sanver

  7. Setting expectations. @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  8. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  9. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graph theory has been studied since

    Leonard Euler’s Bridges 1736
  10. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  11. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  12. Graphs \o/ @michellesanver Graphs are really just connected data… They

    are everywhere (Michelle)-[:LOVES]->(Neo4j)
  13. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  14. @michellesanver It’s modern. (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  15. @michellesanver Facebook open graph (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  16. @michellesanver A graph is an easy way to visualise connected

    data. Michelle Graphs likes (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
  17. Graphs vs. Relational databases @michellesanver Relational question: Average age of

    everyone in this list? Graph question: Who knows me from third parties? (Michelle)-[:LOVES]->(Neo4j)
  18. @michellesanver Relational databases have tables Recipes (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational

    databases
  19. @michellesanver Tables have relationships Recipes Groups (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational

    databases
  20. @michellesanver Which causes a join table… Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  21. @michellesanver You query via the table Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  22. @michellesanver Imagine having *actual* relations Pasta group Spaghetti Italian group

    (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
  23. @michellesanver Recipes Groups RecipeToGroup Pasta group Spaghetti Italian group (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  24. @michellesanver A big query. The EAV model. (Michelle)-[:LOVES]->(Neo4j) Graphs vs.

    Relational databases
  25. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SET @entityid = '3'; SELECT ea.attribute_id, ea.attribute_code, eav.value

    AS 'value', 'varchar' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_varchar eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'int' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_int eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION Graphs vs. Relational databases
  26. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'decimal' AS

    'type' FROM catalog_category_entity e JOIN catalog_category_entity_d ecimal eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'datetime' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_datetime eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'text' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_text eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid Graphs vs. Relational databases
  27. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Having a flexible schema database costs a lot.

    Graphs vs. Relational databases
  28. Neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  29. neo4j.org @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  30. neotechnology.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  31. Java Based @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  32. Open Source! github.com/neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  33. Meetups everywhere neo4j.meetup.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  34. Graphs and Neo4j @michellesanver Graphs have… (Michelle)-[:LOVES]->(Neo4j)

  35. @michellesanver ( Nodes ) (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  36. @michellesanver (Node) { Properties } As many as you want

    Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  37. @michellesanver (Node) [ Relationships ] As many as you want

    Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  38. @michellesanver [Relationship { properties } ] As many as you

    want Nick: geekie Nick: WyriHaximus knows (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  39. @michellesanver Labels Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  40. @michellesanver Indexes for easy lookup Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j)

    Graphs and Neo4j
  41. @michellesanver Common named graphs (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  42. @michellesanver Common named graphs Local McLaughlin graph (Michelle)-[:LOVES]->(Neo4j) Graphs and

    Neo4j
  43. @michellesanver Diamond Butterfly Bull Franklin Tutte (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

    Common named graphs
  44. @michellesanver You can make art out of your DB. (Don’t)

    (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  45. OmNomHub https://github.com/Omnomhub @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  46. OmNomHub @michellesanver Like GitHub but for recipes! (Michelle)-[:LOVES]->(Neo4j)

  47. OmNomHub @michellesanver Fork a recipe (Michelle)-[:LOVES]->(Neo4j)

  48. @michellesanver See all forks (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  49. @michellesanver Join groups (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  50. @michellesanver Search similar recipes (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  51. @michellesanver Have fun! ;-) (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  52. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  53. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j) Disks File System Cache Record Files

    Object Cache Cypher Core API Traversal API Transaction Management Transaction Log
  54. Hang in there (Michelle)-[:LOVES]->(Kittens) @michellesanver

  55. Neo4j architecture: Traversals @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  56. @michellesanver Pathexpanders: Define what to traverse (Michelle)-[:LOVES]->(Neo4j) Traversals

  57. @michellesanver Order: Depth-first or breadth-first. (Michelle)-[:LOVES]->(Neo4j) Traversals

  58. @michellesanver Uniqueness : Visit once. (Michelle)-[:LOVES]->(Neo4j) Traversals

  59. @michellesanver Evaluator : Should I stay or should I go? (Michelle)-[:LOVES]->(Neo4j)

    Traversals
  60. @michellesanver Starting nodes: And so it begins. (Michelle)-[:LOVES]->(Neo4j) Traversals

  61. Neo4j architecture: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  62. Cypher @michellesanver Cypher: SQL for graph databases. (Michelle)-[:LOVES]->(Neo4j)

  63. @michellesanver ) (Michelle)-[:LOVES]->(Neo4j) Parentheses means nodes (Or hugs) ( Cypher

  64. @michellesanver } (Michelle)-[:LOVES]->(Neo4j) Curly braces means properties { Cypher

  65. @michellesanver ] (Michelle)-[:LOVES]->(Neo4j) Square brackets means relationships [ Cypher

  66. @michellesanver Makes it easy to visualise and query the data.

    (Michelle)-[:LOVES]->(Neo4j) Browser
  67. @michellesanver Let’s learn cypher in the browser! (Michelle)-[:LOVES]->(Neo4j) Browser

  68. DEMO: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  69. Neo4j architecture Neostore File Storage @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  70. Neostore @michellesanver Several different store files (Michelle)-[:LOVES]->(Neo4j)

  71. @michellesanver Each store has specific data (Michelle)-[:LOVES]->(Neo4j) Neostore

  72. @michellesanver Nodes neostore.nodestore.db 9 bytes (Michelle)-[:LOVES]->(Neo4j) Neostore

  73. @michellesanver Nodes in-use key: 1 byte (Michelle)-[:LOVES]->(Neo4j) Neostore

  74. Neostore Fixed size == FAST @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  75. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Relationships neostore.relationshipstore.db 33 bytes Neostore

  76. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Properties neostore.propertystore.db neostore.propertystore.db.index neostore.propertystore.db.strings neostore.propertystore.db.arrays Neostore

  77. Hardware matters! @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  78. Cache @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  79. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Two-tiered Cache Cache

  80. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Cache

  81. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Can be fine-tuned Cache

  82. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Object Cache Properties and references to their relationships

    Cache
  83. OmNomHub: Beyond Pizza @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  84. OmNomHub: Beyond Pizza @michellesanver Connecting users “You both like” (Michelle)-[:LOVES]->(Neo4j)

  85. @michellesanver Connecting recipes “These have similar ingredients and user base”

    (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
  86. @michellesanver Being smart “You might like” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza

  87. @michellesanver Being smart Smart recipe collections! (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza

  88. @michellesanver Being creepy “Don’t like meat huh?” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond

    Pizza
  89. Wrapup @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  90. Wrapup @michellesanver Graphs are everywhere (Michelle)-[:LOVES]->(Neo4j)

  91. @michellesanver They make it easy to connect data (Michelle)-[:LOVES]->(Neo4j) Wrapup

  92. @michellesanver Easier to be creepy <3 (Michelle)-[:LOVES]->(Neo4j) Wrapup

  93. @michellesanver OmNomHub will be EPIC :D (Michelle)-[:LOVES]->(Neo4j) Wrapup

  94. @michellesanver Open Source! https://github.com/Omnomhub (Michelle)-[:LOVES]->(Neo4j) Wrapup

  95. Resources @michellesanver docs.neo4j.org Graph Databases - Ian Robinson, 
 neo

    technology gives you the e-book version for free. neo4j.org/learn (Michelle)-[:LOVES]->(Neo4j)
  96. Pleaaaaase…. @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  97. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  98. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  99. Give me feedback https://joind.in/12745 @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  100. Thank you NoSQL Day 2014 @michellesanver (We)-[:LOVE]->(Neo4j)

  101. Questions? @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  102. How would you use it? @michellesanver (Michelle)-[:LOVES]->(Neo4j)