Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Graph Databases: The internal works of Neo4j

Graph Databases: The internal works of Neo4j

Relational databases (e.g. MySQL, PostgreSQL) — ironically — are not that good at the complex relationships some modern applications need. Multiple joins and complex sub-queries can take a toll on performance. Graph Databases are all about relationships, but how do they handle them internally? In this talk I will go through basic graph theory and explain how the popular graph database 'Neo4j' implements these concepts to handle connected data.

01da6d807a29ad6d49801c0157518148?s=128

Michelle Sanver

November 21, 2014
Tweet

Transcript

  1. @michellesanver Neo4J: Graph Database NoSQL Day 2014

  2. About: Michelle Sanver @michellesanver

  3. About: Michelle Sanver @michellesanver President (Michelle)-[:LOVES]->(Neo4j)

  4. @michellesanver Code Addict (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver

  5. @michellesanver Zürich, Switzerland (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver

  6. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Michelle is_from is_from lives_in lived_in About: Michelle Sanver

  7. Setting expectations. @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  8. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  9. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graph theory has been studied since

    Leonard Euler’s Bridges 1736
  10. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  11. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  12. Graphs \o/ @michellesanver Graphs are really just connected data… They

    are everywhere (Michelle)-[:LOVES]->(Neo4j)
  13. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  14. @michellesanver It’s modern. (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  15. @michellesanver Facebook open graph (Michelle)-[:LOVES]->(Neo4j) Graphs \o/

  16. @michellesanver A graph is an easy way to visualise connected

    data. Michelle Graphs likes (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
  17. Graphs vs. Relational databases @michellesanver Relational question: Average age of

    everyone in this list? Graph question: Who knows me from third parties? (Michelle)-[:LOVES]->(Neo4j)
  18. @michellesanver Relational databases have tables Recipes (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational

    databases
  19. @michellesanver Tables have relationships Recipes Groups (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational

    databases
  20. @michellesanver Which causes a join table… Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  21. @michellesanver You query via the table Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  22. @michellesanver Imagine having *actual* relations Pasta group Spaghetti Italian group

    (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
  23. @michellesanver Recipes Groups RecipeToGroup Pasta group Spaghetti Italian group (Michelle)-[:LOVES]->(Neo4j)

    Graphs vs. Relational databases
  24. @michellesanver A big query. The EAV model. (Michelle)-[:LOVES]->(Neo4j) Graphs vs.

    Relational databases
  25. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SET @entityid = '3'; SELECT ea.attribute_id, ea.attribute_code, eav.value

    AS 'value', 'varchar' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_varchar eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'int' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_int eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION Graphs vs. Relational databases
  26. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'decimal' AS

    'type' FROM catalog_category_entity e JOIN catalog_category_entity_d ecimal eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'datetime' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_datetime eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'text' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_text eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid Graphs vs. Relational databases
  27. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Having a flexible schema database costs a lot.

    Graphs vs. Relational databases
  28. Neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  29. neo4j.org @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  30. neotechnology.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  31. Java Based @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  32. Open Source! github.com/neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  33. Meetups everywhere neo4j.meetup.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  34. Graphs and Neo4j @michellesanver Graphs have… (Michelle)-[:LOVES]->(Neo4j)

  35. @michellesanver ( Nodes ) (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  36. @michellesanver (Node) { Properties } As many as you want

    Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  37. @michellesanver (Node) [ Relationships ] As many as you want

    Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  38. @michellesanver [Relationship { properties } ] As many as you

    want Nick: geekie Nick: WyriHaximus knows (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  39. @michellesanver Labels Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  40. @michellesanver Indexes for easy lookup Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j)

    Graphs and Neo4j
  41. @michellesanver Common named graphs (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

  42. @michellesanver Common named graphs Local McLaughlin graph (Michelle)-[:LOVES]->(Neo4j) Graphs and

    Neo4j
  43. @michellesanver Diamond Butterfly Bull Franklin Tutte (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j

    Common named graphs
  44. @michellesanver You can make art out of your DB. (Don’t)

    (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
  45. OmNomHub https://github.com/Omnomhub @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  46. OmNomHub @michellesanver Like GitHub but for recipes! (Michelle)-[:LOVES]->(Neo4j)

  47. OmNomHub @michellesanver Fork a recipe (Michelle)-[:LOVES]->(Neo4j)

  48. @michellesanver See all forks (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  49. @michellesanver Join groups (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  50. @michellesanver Search similar recipes (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  51. @michellesanver Have fun! ;-) (Michelle)-[:LOVES]->(Neo4j) OmNomHub

  52. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  53. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j) Disks File System Cache Record Files

    Object Cache Cypher Core API Traversal API Transaction Management Transaction Log
  54. Hang in there (Michelle)-[:LOVES]->(Kittens) @michellesanver

  55. Neo4j architecture: Traversals @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  56. @michellesanver Pathexpanders: Define what to traverse (Michelle)-[:LOVES]->(Neo4j) Traversals

  57. @michellesanver Order: Depth-first or breadth-first. (Michelle)-[:LOVES]->(Neo4j) Traversals

  58. @michellesanver Uniqueness : Visit once. (Michelle)-[:LOVES]->(Neo4j) Traversals

  59. @michellesanver Evaluator : Should I stay or should I go? (Michelle)-[:LOVES]->(Neo4j)

    Traversals
  60. @michellesanver Starting nodes: And so it begins. (Michelle)-[:LOVES]->(Neo4j) Traversals

  61. Neo4j architecture: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  62. Cypher @michellesanver Cypher: SQL for graph databases. (Michelle)-[:LOVES]->(Neo4j)

  63. @michellesanver ) (Michelle)-[:LOVES]->(Neo4j) Parentheses means nodes (Or hugs) ( Cypher

  64. @michellesanver } (Michelle)-[:LOVES]->(Neo4j) Curly braces means properties { Cypher

  65. @michellesanver ] (Michelle)-[:LOVES]->(Neo4j) Square brackets means relationships [ Cypher

  66. @michellesanver Makes it easy to visualise and query the data.

    (Michelle)-[:LOVES]->(Neo4j) Browser
  67. @michellesanver Let’s learn cypher in the browser! (Michelle)-[:LOVES]->(Neo4j) Browser

  68. DEMO: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  69. Neo4j architecture Neostore File Storage @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  70. Neostore @michellesanver Several different store files (Michelle)-[:LOVES]->(Neo4j)

  71. @michellesanver Each store has specific data (Michelle)-[:LOVES]->(Neo4j) Neostore

  72. @michellesanver Nodes neostore.nodestore.db 9 bytes (Michelle)-[:LOVES]->(Neo4j) Neostore

  73. @michellesanver Nodes in-use key: 1 byte (Michelle)-[:LOVES]->(Neo4j) Neostore

  74. Neostore Fixed size == FAST @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  75. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Relationships neostore.relationshipstore.db 33 bytes Neostore

  76. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Properties neostore.propertystore.db neostore.propertystore.db.index neostore.propertystore.db.strings neostore.propertystore.db.arrays Neostore

  77. Hardware matters! @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  78. Cache @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  79. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Two-tiered Cache Cache

  80. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Cache

  81. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Can be fine-tuned Cache

  82. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Object Cache Properties and references to their relationships

    Cache
  83. OmNomHub: Beyond Pizza @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  84. OmNomHub: Beyond Pizza @michellesanver Connecting users “You both like” (Michelle)-[:LOVES]->(Neo4j)

  85. @michellesanver Connecting recipes “These have similar ingredients and user base”

    (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
  86. @michellesanver Being smart “You might like” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza

  87. @michellesanver Being smart Smart recipe collections! (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza

  88. @michellesanver Being creepy “Don’t like meat huh?” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond

    Pizza
  89. Wrapup @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  90. Wrapup @michellesanver Graphs are everywhere (Michelle)-[:LOVES]->(Neo4j)

  91. @michellesanver They make it easy to connect data (Michelle)-[:LOVES]->(Neo4j) Wrapup

  92. @michellesanver Easier to be creepy <3 (Michelle)-[:LOVES]->(Neo4j) Wrapup

  93. @michellesanver OmNomHub will be EPIC :D (Michelle)-[:LOVES]->(Neo4j) Wrapup

  94. @michellesanver Open Source! https://github.com/Omnomhub (Michelle)-[:LOVES]->(Neo4j) Wrapup

  95. Resources @michellesanver docs.neo4j.org Graph Databases - Ian Robinson, 
 neo

    technology gives you the e-book version for free. neo4j.org/learn (Michelle)-[:LOVES]->(Neo4j)
  96. Pleaaaaase…. @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  97. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  98. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  99. Give me feedback https://joind.in/12745 @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  100. Thank you NoSQL Day 2014 @michellesanver (We)-[:LOVE]->(Neo4j)

  101. Questions? @michellesanver (Michelle)-[:LOVES]->(Neo4j)

  102. How would you use it? @michellesanver (Michelle)-[:LOVES]->(Neo4j)