$30 off During Our Annual Pro Sale. View Details »

Graph Databases: The internal works of Neo4j

Graph Databases: The internal works of Neo4j

Relational databases (e.g. MySQL, PostgreSQL) — ironically — are not that good at the complex relationships some modern applications need. Multiple joins and complex sub-queries can take a toll on performance. Graph Databases are all about relationships, but how do they handle them internally? In this talk I will go through basic graph theory and explain how the popular graph database 'Neo4j' implements these concepts to handle connected data.

Michelle Sanver

November 21, 2014
Tweet

More Decks by Michelle Sanver

Other Decks in Programming

Transcript

  1. @michellesanver
    Neo4J: Graph Database
    NoSQL Day 2014

    View Slide

  2. About: Michelle Sanver
    @michellesanver

    View Slide

  3. About: Michelle Sanver
    @michellesanver
    President
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  4. @michellesanver
    Code Addict
    (Michelle)-[:LOVES]->(Neo4j)
    About: Michelle Sanver

    View Slide

  5. @michellesanver
    Zürich, Switzerland
    (Michelle)-[:LOVES]->(Neo4j)
    About: Michelle Sanver

    View Slide

  6. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Michelle
    is_from
    is_from lives_in
    lived_in
    About: Michelle Sanver

    View Slide

  7. Setting expectations.
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  8. Graphs \o/
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  9. Graphs \o/
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Graph theory has been
    studied since Leonard
    Euler’s Bridges 1736

    View Slide

  10. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/

    View Slide

  11. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/

    View Slide

  12. Graphs \o/
    @michellesanver
    Graphs are really just connected data… They are
    everywhere
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  13. Graphs \o/
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  14. @michellesanver
    It’s modern.
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/

    View Slide

  15. @michellesanver
    Facebook open graph
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/

    View Slide

  16. @michellesanver
    A graph is an easy way to visualise connected data.
    Michelle Graphs
    likes
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs \o/

    View Slide

  17. Graphs vs. Relational databases
    @michellesanver
    Relational question: Average age of everyone in this list?
    Graph question: Who knows me from third parties?
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  18. @michellesanver
    Relational databases have tables
    Recipes
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  19. @michellesanver
    Tables have relationships
    Recipes Groups
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  20. @michellesanver
    Which causes a join table…
    Recipes Groups
    RecipeToGroup
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  21. @michellesanver
    You query via the table
    Recipes Groups
    RecipeToGroup
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  22. @michellesanver
    Imagine having *actual* relations
    Pasta group Spaghetti Italian group
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  23. @michellesanver
    Recipes Groups
    RecipeToGroup
    Pasta group Spaghetti Italian group
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  24. @michellesanver
    A big query.
    The EAV model.
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs vs. Relational databases

    View Slide

  25. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    SET @entityid = '3';
    SELECT ea.attribute_id,
    ea.attribute_code, eav.value AS 'value',
    'varchar' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_varchar
    eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    SELECT ea.attribute_id,
    ea.attribute_code, eav.value AS
    'value', 'int' AS 'type'
    FROM catalog_category_entity
    e
    JOIN
    catalog_category_entity_int eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id =
    ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    Graphs vs. Relational databases

    View Slide

  26. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    SELECT ea.attribute_id,
    ea.attribute_code,
    eav.value AS 'value',
    'decimal' AS 'type'
    FROM
    catalog_category_entity e
    JOIN
    catalog_category_entity_d
    ecimal eav
    ON e.entity_id =
    eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id =
    ea.attribute_id
    WHERE e.entity_id =
    @entityid
    UNION
    SELECT ea.attribute_id, ea.attribute_code,
    eav.value AS 'value', 'datetime' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_datetime eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    UNION
    SELECT ea.attribute_id, ea.attribute_code,
    eav.value AS 'value', 'text' AS 'type'
    FROM catalog_category_entity e
    JOIN catalog_category_entity_text eav
    ON e.entity_id = eav.entity_id
    JOIN eav_attribute ea
    ON eav.attribute_id = ea.attribute_id
    WHERE e.entity_id = @entityid
    Graphs vs. Relational databases

    View Slide

  27. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Having a flexible schema database costs a lot.
    Graphs vs. Relational databases

    View Slide

  28. Neo4j
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  29. neo4j.org
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  30. neotechnology.com
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  31. Java Based
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  32. Open Source!
    github.com/neo4j
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  33. Meetups everywhere
    neo4j.meetup.com
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  34. Graphs and Neo4j
    @michellesanver
    Graphs have…
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  35. @michellesanver
    ( Nodes )
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  36. @michellesanver
    (Node) { Properties }
    As many as you want
    Type:
    Person
    Nick:
    geekie
    Type:
    Person
    Nick:
    WyriHaximus
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  37. @michellesanver
    (Node) [ Relationships ]
    As many as you want
    Type:
    Person
    Nick:
    geekie
    Type:
    Person
    Nick:
    WyriHaximus
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  38. @michellesanver
    [Relationship { properties } ]
    As many as you want
    Nick:
    geekie
    Nick:
    WyriHaximus
    knows
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  39. @michellesanver
    Labels
    Type:
    Person
    Nick:
    geekie
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  40. @michellesanver
    Indexes for easy lookup
    Type:
    Person
    Nick:
    geekie
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  41. @michellesanver
    Common named graphs
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  42. @michellesanver
    Common named graphs
    Local McLaughlin graph
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  43. @michellesanver
    Diamond Butterfly Bull Franklin Tutte
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j
    Common named graphs

    View Slide

  44. @michellesanver
    You can make art out of your DB.
    (Don’t)
    (Michelle)-[:LOVES]->(Neo4j)
    Graphs and Neo4j

    View Slide

  45. OmNomHub
    https://github.com/Omnomhub
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  46. OmNomHub
    @michellesanver
    Like GitHub but for recipes!
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  47. OmNomHub
    @michellesanver
    Fork a recipe
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  48. @michellesanver
    See all forks
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub

    View Slide

  49. @michellesanver
    Join groups
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub

    View Slide

  50. @michellesanver
    Search similar recipes
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub

    View Slide

  51. @michellesanver
    Have fun! ;-)
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub

    View Slide

  52. Neo4j architecture
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  53. Neo4j architecture
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Disks
    File System Cache
    Record Files
    Object Cache
    Cypher
    Core API
    Traversal API
    Transaction Management
    Transaction Log

    View Slide

  54. Hang in there
    (Michelle)-[:LOVES]->(Kittens) @michellesanver

    View Slide

  55. Neo4j architecture: Traversals
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  56. @michellesanver
    Pathexpanders: Define what to traverse
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals

    View Slide

  57. @michellesanver
    Order: Depth-first or breadth-first.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals

    View Slide

  58. @michellesanver
    Uniqueness : Visit once.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals

    View Slide

  59. @michellesanver
    Evaluator : Should I stay or should I go?
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals

    View Slide

  60. @michellesanver
    Starting nodes: And so it begins.
    (Michelle)-[:LOVES]->(Neo4j)
    Traversals

    View Slide

  61. Neo4j architecture: Cypher and Browser
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  62. Cypher
    @michellesanver
    Cypher: SQL for graph databases.
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  63. @michellesanver
    )
    (Michelle)-[:LOVES]->(Neo4j)
    Parentheses means nodes
    (Or hugs)
    (
    Cypher

    View Slide

  64. @michellesanver
    }
    (Michelle)-[:LOVES]->(Neo4j)
    Curly braces means
    properties
    {
    Cypher

    View Slide

  65. @michellesanver
    ]
    (Michelle)-[:LOVES]->(Neo4j)
    Square brackets means
    relationships
    [
    Cypher

    View Slide

  66. @michellesanver
    Makes it easy to visualise and query the data.
    (Michelle)-[:LOVES]->(Neo4j)
    Browser

    View Slide

  67. @michellesanver
    Let’s learn cypher in the browser!
    (Michelle)-[:LOVES]->(Neo4j)
    Browser

    View Slide

  68. DEMO: Cypher and Browser
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  69. Neo4j architecture
    Neostore File Storage
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  70. Neostore
    @michellesanver
    Several different store files
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  71. @michellesanver
    Each store has specific data
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore

    View Slide

  72. @michellesanver
    Nodes
    neostore.nodestore.db

    9 bytes
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore

    View Slide

  73. @michellesanver
    Nodes
    in-use key: 1 byte
    (Michelle)-[:LOVES]->(Neo4j)
    Neostore

    View Slide

  74. Neostore
    Fixed size == FAST
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  75. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Relationships
    neostore.relationshipstore.db

    33 bytes
    Neostore

    View Slide

  76. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Properties
    neostore.propertystore.db

    neostore.propertystore.db.index

    neostore.propertystore.db.strings

    neostore.propertystore.db.arrays
    Neostore

    View Slide

  77. Hardware matters!
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  78. Cache
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  79. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Two-tiered Cache
    Cache

    View Slide

  80. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Filesystem cache
    Cache

    View Slide

  81. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Filesystem cache
    Can be fine-tuned
    Cache

    View Slide

  82. @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)
    Object Cache
    Properties and references to their relationships
    Cache

    View Slide

  83. OmNomHub: Beyond Pizza
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  84. OmNomHub: Beyond Pizza
    @michellesanver
    Connecting users
    “You both like”
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  85. @michellesanver
    Connecting recipes
    “These have similar ingredients and user base”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza

    View Slide

  86. @michellesanver
    Being smart
    “You might like”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza

    View Slide

  87. @michellesanver
    Being smart
    Smart recipe collections!
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza

    View Slide

  88. @michellesanver
    Being creepy
    “Don’t like meat huh?”
    (Michelle)-[:LOVES]->(Neo4j)
    OmNomHub: Beyond Pizza

    View Slide

  89. Wrapup
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  90. Wrapup
    @michellesanver
    Graphs are everywhere
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  91. @michellesanver
    They make it easy to connect data
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup

    View Slide

  92. @michellesanver
    Easier to be creepy <3
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup

    View Slide

  93. @michellesanver
    OmNomHub will be EPIC :D
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup

    View Slide

  94. @michellesanver
    Open Source!
    https://github.com/Omnomhub
    (Michelle)-[:LOVES]->(Neo4j)
    Wrapup

    View Slide

  95. Resources
    @michellesanver
    docs.neo4j.org
    Graph Databases - Ian Robinson, 

    neo technology gives you the e-book version for free.
    neo4j.org/learn
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  96. Pleaaaaase….
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  97. Please
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  98. Please
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  99. Give me feedback
    https://joind.in/12745
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  100. Thank you
    NoSQL Day 2014
    @michellesanver
    (We)-[:LOVE]->(Neo4j)

    View Slide

  101. Questions?
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide

  102. How would you use it?
    @michellesanver
    (Michelle)-[:LOVES]->(Neo4j)

    View Slide