Graph Databases: The internal works of Neo4j

Relational databases (e.g. MySQL, PostgreSQL) — ironically — are not that good at the complex relationships some modern applications need. Multiple joins and complex sub-queries can take a toll on performance. Graph Databases are all about relationships, but how do they handle them internally? In this talk I will go through basic graph theory and explain how the popular graph database 'Neo4j' implements these concepts to handle connected data.

Transcript

1. @michellesanver Neo4J: Graph Database NoSQL Day 2014
2. About: Michelle Sanver @michellesanver
3. About: Michelle Sanver @michellesanver President (Michelle)-[:LOVES]->(Neo4j)
4. @michellesanver Code Addict (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver
5. @michellesanver Zürich, Switzerland (Michelle)-[:LOVES]->(Neo4j) About: Michelle Sanver
6. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Michelle is_from is_from lives_in lived_in About: Michelle Sanver
7. Setting expectations. @michellesanver (Michelle)-[:LOVES]->(Neo4j)
8. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)
9. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graph theory has been studied since Leonard Euler’s Bridges 1736
10. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
11. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
12. Graphs \o/ @michellesanver Graphs are really just connected data… They are everywhere (Michelle)-[:LOVES]->(Neo4j)
13. Graphs \o/ @michellesanver (Michelle)-[:LOVES]->(Neo4j)
14. @michellesanver It’s modern. (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
15. @michellesanver Facebook open graph (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
16. @michellesanver A graph is an easy way to visualise connected data. Michelle Graphs likes (Michelle)-[:LOVES]->(Neo4j) Graphs \o/
17. Graphs vs. Relational databases @michellesanver Relational question: Average age of everyone in this list? Graph question: Who knows me from third parties? (Michelle)-[:LOVES]->(Neo4j)
18. @michellesanver Relational databases have tables Recipes (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
19. @michellesanver Tables have relationships Recipes Groups (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
20. @michellesanver Which causes a join table… Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
21. @michellesanver You query via the table Recipes Groups RecipeToGroup (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
22. @michellesanver Imagine having *actual* relations Pasta group Spaghetti Italian group (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
23. @michellesanver Recipes Groups RecipeToGroup Pasta group Spaghetti Italian group (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
24. @michellesanver A big query. The EAV model. (Michelle)-[:LOVES]->(Neo4j) Graphs vs. Relational databases
25. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SET @entityid = '3'; SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'varchar' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_varchar eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'int' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_int eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION Graphs vs. Relational databases
26. @michellesanver (Michelle)-[:LOVES]->(Neo4j) SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'decimal' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_d ecimal eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'datetime' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_datetime eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid UNION SELECT ea.attribute_id, ea.attribute_code, eav.value AS 'value', 'text' AS 'type' FROM catalog_category_entity e JOIN catalog_category_entity_text eav ON e.entity_id = eav.entity_id JOIN eav_attribute ea ON eav.attribute_id = ea.attribute_id WHERE e.entity_id = @entityid Graphs vs. Relational databases
27. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Having a flexible schema database costs a lot. Graphs vs. Relational databases
28. Neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)
29. neo4j.org @michellesanver (Michelle)-[:LOVES]->(Neo4j)
30. neotechnology.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)
31. Java Based @michellesanver (Michelle)-[:LOVES]->(Neo4j)
32. Open Source! github.com/neo4j @michellesanver (Michelle)-[:LOVES]->(Neo4j)
33. Meetups everywhere neo4j.meetup.com @michellesanver (Michelle)-[:LOVES]->(Neo4j)
34. Graphs and Neo4j @michellesanver Graphs have… (Michelle)-[:LOVES]->(Neo4j)
35. @michellesanver ( Nodes ) (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
36. @michellesanver (Node) { Properties } As many as you want Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
37. @michellesanver (Node) [ Relationships ] As many as you want Type: Person Nick: geekie Type: Person Nick: WyriHaximus (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
38. @michellesanver [Relationship { properties } ] As many as you want Nick: geekie Nick: WyriHaximus knows (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
39. @michellesanver Labels Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
40. @michellesanver Indexes for easy lookup Type: Person Nick: geekie (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
41. @michellesanver Common named graphs (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
42. @michellesanver Common named graphs Local McLaughlin graph (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
43. @michellesanver Diamond Butterfly Bull Franklin Tutte (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j Common named graphs
44. @michellesanver You can make art out of your DB. (Don’t) (Michelle)-[:LOVES]->(Neo4j) Graphs and Neo4j
45. OmNomHub https://github.com/Omnomhub @michellesanver (Michelle)-[:LOVES]->(Neo4j)
46. OmNomHub @michellesanver Like GitHub but for recipes! (Michelle)-[:LOVES]->(Neo4j)
47. OmNomHub @michellesanver Fork a recipe (Michelle)-[:LOVES]->(Neo4j)
48. @michellesanver See all forks (Michelle)-[:LOVES]->(Neo4j) OmNomHub
49. @michellesanver Join groups (Michelle)-[:LOVES]->(Neo4j) OmNomHub
50. @michellesanver Search similar recipes (Michelle)-[:LOVES]->(Neo4j) OmNomHub
51. @michellesanver Have fun! ;-) (Michelle)-[:LOVES]->(Neo4j) OmNomHub
52. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j)
53. Neo4j architecture @michellesanver (Michelle)-[:LOVES]->(Neo4j) Disks File System Cache Record Files Object Cache Cypher Core API Traversal API Transaction Management Transaction Log
54. Hang in there (Michelle)-[:LOVES]->(Kittens) @michellesanver
55. Neo4j architecture: Traversals @michellesanver (Michelle)-[:LOVES]->(Neo4j)
56. @michellesanver Pathexpanders: Define what to traverse (Michelle)-[:LOVES]->(Neo4j) Traversals
57. @michellesanver Order: Depth-first or breadth-first. (Michelle)-[:LOVES]->(Neo4j) Traversals
58. @michellesanver Uniqueness : Visit once. (Michelle)-[:LOVES]->(Neo4j) Traversals
59. @michellesanver Evaluator : Should I stay or should I go? (Michelle)-[:LOVES]->(Neo4j) Traversals
60. @michellesanver Starting nodes: And so it begins. (Michelle)-[:LOVES]->(Neo4j) Traversals
61. Neo4j architecture: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)
62. Cypher @michellesanver Cypher: SQL for graph databases. (Michelle)-[:LOVES]->(Neo4j)
63. @michellesanver ) (Michelle)-[:LOVES]->(Neo4j) Parentheses means nodes (Or hugs) ( Cypher
64. @michellesanver } (Michelle)-[:LOVES]->(Neo4j) Curly braces means properties { Cypher
65. @michellesanver ] (Michelle)-[:LOVES]->(Neo4j) Square brackets means relationships [ Cypher
66. @michellesanver Makes it easy to visualise and query the data. (Michelle)-[:LOVES]->(Neo4j) Browser
67. @michellesanver Let’s learn cypher in the browser! (Michelle)-[:LOVES]->(Neo4j) Browser
68. DEMO: Cypher and Browser @michellesanver (Michelle)-[:LOVES]->(Neo4j)
69. Neo4j architecture Neostore File Storage @michellesanver (Michelle)-[:LOVES]->(Neo4j)
70. Neostore @michellesanver Several different store files (Michelle)-[:LOVES]->(Neo4j)
71. @michellesanver Each store has specific data (Michelle)-[:LOVES]->(Neo4j) Neostore
72. @michellesanver Nodes neostore.nodestore.db 9 bytes (Michelle)-[:LOVES]->(Neo4j) Neostore
73. @michellesanver Nodes in-use key: 1 byte (Michelle)-[:LOVES]->(Neo4j) Neostore
74. Neostore Fixed size == FAST @michellesanver (Michelle)-[:LOVES]->(Neo4j)
75. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Relationships neostore.relationshipstore.db 33 bytes Neostore
76. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Properties neostore.propertystore.db neostore.propertystore.db.index neostore.propertystore.db.strings neostore.propertystore.db.arrays Neostore
77. Hardware matters! @michellesanver (Michelle)-[:LOVES]->(Neo4j)
78. Cache @michellesanver (Michelle)-[:LOVES]->(Neo4j)
79. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Two-tiered Cache Cache
80. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Cache
81. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Filesystem cache Can be fine-tuned Cache
82. @michellesanver (Michelle)-[:LOVES]->(Neo4j) Object Cache Properties and references to their relationships Cache
83. OmNomHub: Beyond Pizza @michellesanver (Michelle)-[:LOVES]->(Neo4j)
84. OmNomHub: Beyond Pizza @michellesanver Connecting users “You both like” (Michelle)-[:LOVES]->(Neo4j)
85. @michellesanver Connecting recipes “These have similar ingredients and user base” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
86. @michellesanver Being smart “You might like” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
87. @michellesanver Being smart Smart recipe collections! (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
88. @michellesanver Being creepy “Don’t like meat huh?” (Michelle)-[:LOVES]->(Neo4j) OmNomHub: Beyond Pizza
89. Wrapup @michellesanver (Michelle)-[:LOVES]->(Neo4j)
90. Wrapup @michellesanver Graphs are everywhere (Michelle)-[:LOVES]->(Neo4j)
91. @michellesanver They make it easy to connect data (Michelle)-[:LOVES]->(Neo4j) Wrapup
92. @michellesanver Easier to be creepy <3 (Michelle)-[:LOVES]->(Neo4j) Wrapup
93. @michellesanver OmNomHub will be EPIC :D (Michelle)-[:LOVES]->(Neo4j) Wrapup
94. @michellesanver Open Source! https://github.com/Omnomhub (Michelle)-[:LOVES]->(Neo4j) Wrapup
95. Resources @michellesanver docs.neo4j.org Graph Databases - Ian Robinson, 
 neo technology gives you the e-book version for free. neo4j.org/learn (Michelle)-[:LOVES]->(Neo4j)
96. Pleaaaaase…. @michellesanver (Michelle)-[:LOVES]->(Neo4j)
97. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)
98. Please @michellesanver (Michelle)-[:LOVES]->(Neo4j)
99. Give me feedback https://joind.in/12745 @michellesanver (Michelle)-[:LOVES]->(Neo4j)
100. Thank you NoSQL Day 2014 @michellesanver (We)-[:LOVE]->(Neo4j)
101. Questions? @michellesanver (Michelle)-[:LOVES]->(Neo4j)
102. How would you use it? @michellesanver (Michelle)-[:LOVES]->(Neo4j)