Graph Databases: The internal works of Neo4j

Relational databases (e.g. MySQL, PostgreSQL) — ironically — are not that good at the complex relationships some modern applications need. Multiple joins and complex sub-queries can take a toll on performance. Graph Databases are all about relationships, but how do they handle them internally? In this talk I will go through basic graph theory and explain how the popular graph database 'Neo4j' implements these concepts to handle connected data.

Transcript

1. @michellesanver
   Neo4J: Graph Database
   NoSQL Day 2014
   

   View Slide

2. About: Michelle Sanver
   @michellesanver
   

   View Slide

3. About: Michelle Sanver
   @michellesanver
   President
   (Michelle)-[:LOVES]->(Neo4j)
   

   View Slide

4. @michellesanver
   Code Addict
   (Michelle)-[:LOVES]->(Neo4j)
   About: Michelle Sanver
   

   View Slide

5. @michellesanver
   Zürich, Switzerland
   (Michelle)-[:LOVES]->(Neo4j)
   About: Michelle Sanver
   

   View Slide

6. @michellesanver
   (Michelle)-[:LOVES]->(Neo4j)
   Michelle
   is_from
   is_from lives_in
   lived_in
   About: Michelle Sanver
   

   View Slide

7. Setting expectations.
   @michellesanver
   (Michelle)-[:LOVES]->(Neo4j)
   

   View Slide

8. Graphs \o/
   @michellesanver
   (Michelle)-[:LOVES]->(Neo4j)
   

   View Slide

9. Graphs \o/
   @michellesanver
   (Michelle)-[:LOVES]->(Neo4j)
   Graph theory has been
   studied since Leonard
   Euler’s Bridges 1736
   

   View Slide

10. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/
    

    View Slide

11. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/
    

    View Slide

12. Graphs \o/
    @michellesanver
    Graphs are really just connected data… They are
    everywhere
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

13. Graphs \o/
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

14. @michellesanver
    It’s modern.
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/
    

    View Slide

15. @michellesanver
    Facebook open graph
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/
    

    View Slide

16. @michellesanver
    A graph is an easy way to visualise connected data.
    Michelle Graphs
    likes
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/
    

    View Slide

17. Graphs vs. Relational databases
    @michellesanver
    Relational question: Average age of everyone in this list?
    Graph question: Who knows me from third parties?
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

18. @michellesanver
    Relational databases have tables
    Recipes
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

19. @michellesanver
    Tables have relationships
    Recipes Groups
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

20. @michellesanver
    Which causes a join table…
    Recipes Groups
    RecipeToGroup
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

21. @michellesanver
    You query via the table
    Recipes Groups
    RecipeToGroup
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

22. @michellesanver
    Imagine having *actual* relations
    Pasta group Spaghetti Italian group
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

23. @michellesanver
    Recipes Groups
    RecipeToGroup
    Pasta group Spaghetti Italian group
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

24. @michellesanver
    A big query.
    The EAV model.
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases
    

    View Slide

25. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    SET @entityid = '3';
    SELECT ea.attribute_id,
    ea.attribute_code, eav.value AS 'value',
    'varchar' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_varchar
    eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    SELECT ea.attribute_id,
    ea.attribute_code, eav.value AS
    'value', 'int' AS 'type'
    FROM catalog_category_entity
    e
    JOIN
    catalog_category_entity_int eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id =
    ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    Graphs vs. Relational databases
    

    View Slide

26. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    SELECT ea.attribute_id,
    ea.attribute_code,
    eav.value AS 'value',
    'decimal' AS 'type'
    FROM
    catalog_category_entity e
    JOIN
    catalog_category_entity_d
    ecimal eav
    ON e.entity_id =
    eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id =
    ea.attribute_id
    WHERE e.entity_id =
    @entityid
    UNION
    SELECT ea.attribute_id, ea.attribute_code,
    eav.value AS 'value', 'datetime' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_datetime eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    SELECT ea.attribute_id, ea.attribute_code,
    eav.value AS 'value', 'text' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_text eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    Graphs vs. Relational databases
    

    View Slide

27. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Having a flexible schema database costs a lot.
    Graphs vs. Relational databases
    

    View Slide

28. Neo4j
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

29. neo4j.org
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

30. neotechnology.com
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

31. Java Based
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

32. Open Source!
    github.com/neo4j
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

33. Meetups everywhere
    neo4j.meetup.com
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

34. Graphs and Neo4j
    @michellesanver
    Graphs have…
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

35. @michellesanver
    ( Nodes )
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

36. @michellesanver
    (Node) { Properties }
    As many as you want
    Type:
    Person
    Nick:
    geekie
    Type:
    Person
    Nick:
    WyriHaximus
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

37. @michellesanver
    (Node) [ Relationships ]
    As many as you want
    Type:
    Person
    Nick:
    geekie
    Type:
    Person
    Nick:
    WyriHaximus
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

38. @michellesanver
    [Relationship { properties } ]
    As many as you want
    Nick:
    geekie
    Nick:
    WyriHaximus
    knows
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

39. @michellesanver
    Labels
    Type:
    Person
    Nick:
    geekie
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

40. @michellesanver
    Indexes for easy lookup
    Type:
    Person
    Nick:
    geekie
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

41. @michellesanver
    Common named graphs
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

42. @michellesanver
    Common named graphs
    Local McLaughlin graph
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

43. @michellesanver
    Diamond Butterfly Bull Franklin Tutte
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    Common named graphs
    

    View Slide

44. @michellesanver
    You can make art out of your DB.
    (Don’t)
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    

    View Slide

45. OmNomHub
    https://github.com/Omnomhub
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

46. OmNomHub
    @michellesanver
    Like GitHub but for recipes!
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

47. OmNomHub
    @michellesanver
    Fork a recipe
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

48. @michellesanver
    See all forks
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub
    

    View Slide

49. @michellesanver
    Join groups
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub
    

    View Slide

50. @michellesanver
    Search similar recipes
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub
    

    View Slide

51. @michellesanver
    Have fun! ;-)
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub
    

    View Slide

52. Neo4j architecture
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

53. Neo4j architecture
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Disks
    File System Cache
    Record Files
    Object Cache
    Cypher
    Core API
    Traversal API
    Transaction Management
    Transaction Log
    

    View Slide

54. Hang in there
    (Michelle)-[:LOVES]->(Kittens) @michellesanver
    

    View Slide

55. Neo4j architecture: Traversals
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

56. @michellesanver
    Pathexpanders: Define what to traverse
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals
    

    View Slide

57. @michellesanver
    Order: Depth-first or breadth-first.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals
    

    View Slide

58. @michellesanver
    Uniqueness : Visit once.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals
    

    View Slide

59. @michellesanver
    Evaluator : Should I stay or should I go?
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals
    

    View Slide

60. @michellesanver
    Starting nodes: And so it begins.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals
    

    View Slide

61. Neo4j architecture: Cypher and Browser
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

62. Cypher
    @michellesanver
    Cypher: SQL for graph databases.
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

63. @michellesanver
    )
    (Michelle)-[:LOVES]->(Neo4j)
    Parentheses means nodes
    (Or hugs)
    (
    Cypher
    

    View Slide

64. @michellesanver
    }
    (Michelle)-[:LOVES]->(Neo4j)
    Curly braces means
    properties
    {
    Cypher
    

    View Slide

65. @michellesanver
    ]
    (Michelle)-[:LOVES]->(Neo4j)
    Square brackets means
    relationships
    [
    Cypher
    

    View Slide

66. @michellesanver
    Makes it easy to visualise and query the data.
    (Michelle)-[:LOVES]->(Neo4j)
    Browser
    

    View Slide

67. @michellesanver
    Let’s learn cypher in the browser!
    (Michelle)-[:LOVES]->(Neo4j)
    Browser
    

    View Slide

68. DEMO: Cypher and Browser
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

69. Neo4j architecture
    Neostore File Storage
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

70. Neostore
    @michellesanver
    Several different store files
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

71. @michellesanver
    Each store has specific data
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore
    

    View Slide

72. @michellesanver
    Nodes
    neostore.nodestore.db
    
    9 bytes
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore
    

    View Slide

73. @michellesanver
    Nodes
    in-use key: 1 byte
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore
    

    View Slide

74. Neostore
    Fixed size == FAST
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

75. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Relationships
    neostore.relationshipstore.db
    
    33 bytes
    Neostore
    

    View Slide

76. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Properties
    neostore.propertystore.db
    
    neostore.propertystore.db.index
    
    neostore.propertystore.db.strings
    
    neostore.propertystore.db.arrays
    Neostore
    

    View Slide

77. Hardware matters!
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

78. Cache
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

79. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Two-tiered Cache
    Cache
    

    View Slide

80. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Filesystem cache
    Cache
    

    View Slide

81. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Filesystem cache
    Can be fine-tuned
    Cache
    

    View Slide

82. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Object Cache
    Properties and references to their relationships
    Cache
    

    View Slide

83. OmNomHub: Beyond Pizza
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

84. OmNomHub: Beyond Pizza
    @michellesanver
    Connecting users
    “You both like”
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

85. @michellesanver
    Connecting recipes
    “These have similar ingredients and user base”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza
    

    View Slide

86. @michellesanver
    Being smart
    “You might like”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza
    

    View Slide

87. @michellesanver
    Being smart
    Smart recipe collections!
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza
    

    View Slide

88. @michellesanver
    Being creepy
    “Don’t like meat huh?”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza
    

    View Slide

89. Wrapup
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

90. Wrapup
    @michellesanver
    Graphs are everywhere
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

91. @michellesanver
    They make it easy to connect data
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup
    

    View Slide

92. @michellesanver
    Easier to be creepy <3
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup
    

    View Slide

93. @michellesanver
    OmNomHub will be EPIC :D
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup
    

    View Slide

94. @michellesanver
    Open Source!
    https://github.com/Omnomhub
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup
    

    View Slide

95. Resources
    @michellesanver
    docs.neo4j.org
    Graph Databases - Ian Robinson, 

    neo technology gives you the e-book version for free.
    neo4j.org/learn
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

96. Pleaaaaase….
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

97. Please
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

98. Please
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

99. Give me feedback
    https://joind.in/12745
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    

    View Slide

100. Thank you
     NoSQL Day 2014
     @michellesanver
     (We)-[:LOVE]->(Neo4j)
     

     View Slide

101. Questions?
     @michellesanver
     (Michelle)-[:LOVES]->(Neo4j)
     

     View Slide

102. How would you use it?
     @michellesanver
     (Michelle)-[:LOVES]->(Neo4j)
     

     View Slide