Dgraph: the Graph Database written in Go

In this talk, Francesc - VP of Product at Dgraph Labs - gives an introduction to Dgraph and Graph Databases in general. The talk includes a live demo and Q&A session covering many aspects of the query language and fun facts about Pokémon.

Dgraph is an open-source, fast, feature-rich and horizontally scalable graph database. It's designed from the ground up to run for web-scale, and achieve high throughput and low latency for arbitrarily complex queries.

Written entirely in Go, it embraces simplicity and robustness. It provides a very readable and powerful query language derived from GraphQL: https://docs.dgraph.io/query-language/.

For more, check out Dgraph website: https://dgraph.io.

Transcript

1. The most advanced graph database in the world
2. Dgraph The graph database written in Go
3. Francesc Campoy VP of Product at Dgraph Labs @francesc Campoy You might know me from: justforfunc.com Google Cloud Platform Podcast About me
4. - Graphs and databases - What’s in a Graph? - Graph Databases - Dgraph - Dgraph Architecture - The Dgraph Query Language - Live Demo! - Q&A Agenda
5. We are hiring! dgraph.io/careers
6. Graphs
7. Steven Spielberg Jaws Jurassic Park genre directed Comedy Thriller Science Fiction directed genre genre genre A movie graph node relationship Legend
8. name: Steven Spielberg name: Jaws year: 1975 name: Jurassic Park year: 1993 genre directed name: Comedy name: Thriller name: Science Fiction directed genre genre genre A movie graph with properties node relationship Legend
9. Graphs in Databases
10. name: Steven Spielberg name: Jaws year: 1975 name: Jurassic Park year: 1993 genre directed name: Comedy name: Thriller name: Science Fiction directed genre genre genre How would you store this in your database?
11. The mapping process can be complex: - one-to-one relationships become foreign keys - one-to-many relationships become foreign keys (repeated foreign keys if reversed) - many-to-many become rows in a new table with multiple foreign keys Traversals require joins which become very expensive quick. A graph in a relational database
12. ID int Name string The “Movie - Director” on a relational DB Movie ID int Name string Year int … DirectorID int FK Director Note: Fetching all the movies directed by a director requires an index for performance.
13. Twilight Zone: The Movie Directed by: - Steven Spielberg - John Landis - Joe Dante - George Miller Good luck migrating the schema! You thought a movie had one director?
14. What is a MovieDirector? We had to modify our logical model to fit the technology, bringing in unnecessary complexity. MovieID int FK DirectorID int FK ID int Name string Year int … ID int Name string The “Movie - Director” on a relational DB Movie Director MovieDirector
15. No foreign keys - You will need to keep many copies of your information. - You will need to keep them all up to date. - At that point, why do you even have a database? Traversals require multiple queries: get element, find property, get next element, etc. A graph in a non-relational database
16. Fetching all the names of the movies directed by a director requires n+1 queries. The “Movie - Director” on a no-SQL DB (A) Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ 123, 234, 345 ] } Movie document { “_id”: 123, “name”: “Jaws”, “year”: ... } Movie document { “_id”: 234, “name”: “E.T.”, “year”: ... } Movie document { “_id”: 345, “name”: “Jurassic Park”, “year”: ... }
17. So … what if a movie has multiple directors? The “Movie - Director” on a no-SQL DB (B) Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ { “_id”: 123, “name”: “Jaws”, “year”: … }, { “_id”: 234, “name”: “E.T.”, “year”: … }, { “_id”: 345, “name”: “Jurassic Park”, “year”: … } ] }
18. Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ { “_id”: 123, “name”: “Jaws”, “year”: … }, { “_id”: 234, “name”: “E.T.”, “year”: … }, { “_id”: 345, “name”: “Jurassic Park”, “year”: … } ] } The “Movie - Director” on a no-SQL DB (C) Movie document { “_id”: 123, “name”: “Jaws”, “year”: ... } Movie document { “_id”: 234, “name”: “E.T.”, “year”: ... } Movie document { “_id”: 345, “name”: “Jurassic Park”, “year”: ... } Now we can fetch all movies for a director in a query … but we might easily lose consistency.
19. Graph Databases
20. No need for mapping, the “whiteboard” model is your model. No need for “joins”: - traversals are fast since relationships point directly to nodes, not keys - “Index-free adjacency” - deep traversals are possible (and efficient) Graph Databases
21. Dgraph Data Modeling
22. Subject-Predicate-Value: Subject Predicate Value Jaws <was recorded in the year> 1975 Subject-Predicate-Object: Subject Predicate Object Jaws <was directed by> Steven Spielberg Subject-Predicate-Value name: Steven Spielberg name: Jaws was directed by name: Jaws year: 1975
23. The previous slide is not 100% accurate, as nodes have their own identifiers. So instead of using strings as identifiers: “Jaws” <was recorded in the year> 1975 “Jaws” <was directed by> “Steven Spielberg” We have Universal Identifiers (UIDs): 0x1 <has name> “Jaws” 0x1 <was recorded in the year> 1975 0x1 <was directed by> 0x2 0x2 <has name> “Steven Spielberg” Dgraph data modeling
24. Given the data from before: 0x1 <has name> “Jaws” 0x1 <was recorded in the year> 1975 0x1 <was directed by> 0x2 0x2 <has name> “Steven Spielberg” - 0x1 and 0x2 are UIDs (Universal IDentifiers). - <has name>, <was recorded in the year>, etc. are predicates. - “Jaws”, “Steven Spielberg”, and 1975 are values. Dgraph data modeling
25. Predicates are always attached to UIDs. We associate values and objects to keys composed by UID + predicate Keys Values 0x1:<has name> “Jaws” 0x1:<was recorded in the year> 1975 0x1:<was directed by> 0x2 0x2:<has name> “Steven Spielberg” Sometimes a value can be an array of UIDs or values. Dgraph data modeling
26. 1. Find the starting nodes for the traversal. 2. Append the predicate name to the UIDs we have so far. 3. Find the values associated to the UID:Predicate pairs. 4. Repeat (2) until the associated values are non UID or query is done. Benefit: values are not involved, keeping memory requirements low. Life of a query
27. Life of a query Example: give me the name of the friends of 0x1234. 0x1234 <is_friends_with> _ <has_name> X 1. Find the node with UID 0x1234 2. Append <is_friends_with> 0x1234 3. Retrieve values from 0x1234:<is_friends_with> [0xABCD, 0xBCDE] a. 0xABCD <has_name> “Diggy” | 0xBCDE <has_name> “Augie” 4. Return [“Diggy”, “Augie”]
28. How do we find the first nodes? We don’t always have the UID of the first node of our traversal. We can find them by the value of one of its predicates! - Node with name “Augie”. - All nodes with a predicate <has_age> larger than 18. - All nodes with a predicate <location> 20mi around SLC. These searches could be very expensive, so we use indices.
29. Indexing in Dgraph Dgraph provides indices on: - Strings: hash, exact, term, fulltext, trigram. - DateTime: year, month, day, hour. - Int, Float, Bool: default value index. - Geo properties : default value index.
30. - Schemas are not required in general. - But indices can only be defined on schema fields. - Be aware of the space requirements of the indices. Example, and indexed name predicate: <name>: string @index(fulltext, hash, term, trigram) . Dgraph schemas and indices
31. 1. Find the starting nodes for the traversal using UIDs or indexes. 2. Append the predicate ID (int) to the UIDs we have so far. 3. Find the values associated to the UID:Predicate pairs. 4. Repeat (2) until the associated values are non UID or query is done. Benefit: values are not involved, keeping memory requirements low. Updated life of a query
32. Dgraph Architecture
33. Dgraph Architecture at a glance Alpha Alpha Alpha Zero Zero Zero Ratel Zero: Cluster Management Alpha: Data Storage Ratel: Web UI
34. Dgraph Zero • Called Zero, because it used to be group zero. • Handles cluster membership, transactions and ID generation. • Each Dgraph cluster must have at least one Zero. • Participates in a Raft group for HA.
35. Dgraph Alpha • Called Alpha, because it runs group 1 and higher. • Stores data, serves queries. • Each Dgraph cluster must have at least one Alpha. • Participates in a Raft group for HA.
36. Still hiring! dgraph.io/careers
37. • Not part of the cluster, but useful for exploration. • Connects to Alphas to respond to queries. • Completely optional, but definitely useful. Dgraph Ratel
38. Dgraph Architecture at a glance Alpha Alpha Alpha Zero Zero Zero Ratel Zero: Cluster Management Alpha: Data Storage Ratel: Web UI
39. - Graph Databases make storing and retrieving graphs efficient. - Dgraph provides Subject-Predicate-(Object/Value) - Traversals are very efficient thanks to optimizing for low-latency disk seeks and minimizing network calls. - Dgraph architecture provides high availability and horizontal scalability. Conclusion
40. GraphQL+-
41. - Heavily inspired by GraphQL. - Modified to serve as a better DB language. - Pure GraphQL native support is coming up soon™ - Try it on playground instance of Ratel: play.dgraph.io GraphQL+-
42. Create a new UID and associate it to value “Alice” with predicate <name>. { set { _:alice <name> "Alice" . } } GraphQL+- mutations
43. Output: { "data": { "code": "Success", "message": "Done", "uids": { "alice": "0x1" } }, "extensions": {...} } GraphQL+- mutations
44. Create a new UID and associate it to value “Alice” with predicate <name>. { set { _:bob <name> "Bob" . _:bob <knows> <0x1> . } } GraphQL+- mutations
45. Output: { "data": { "code": "Success", "message": "Done", "uids": { "bob": "0x2" } }, "extensions": {...} } GraphQL+- mutations
46. Current dataset 0x1 0x2 knows Alice Bob name name
47. Our schema Default schema: - default types - no indices
48. Let’s fetch people’s names Query: { q(func: uid(0x2, 0x1)) { uid name } }
49. Let’s fetch people’s names Response: { "data": { "q": [ { "uid": "0x1", "name": "Alice" }, { "uid": "0x2", "name": "Bob" }] }, "extensions": {...} } 0x1 0x2 Alice Bob name name
50. Query: { q(func: uid(0x2)) { uid name knows { uid name } } } Who does Bob know?
51. Response: {"data": {"q": [{ "uid": "0x2", "name": "Bob", "knows": [{ "uid": "0x1", "name": "Alice" }] }]}, "extensions": {...} } Who does Bob know? 0x2 Bob name 0x1 Alice name knows
52. Query: { q(func: eq(name, “Bob”)) { uid name knows { uid name } } } Who does “Bob” know?
53. Response: Error Name: t Message: : Attribute name is not indexed. URL: http://localhost:8080/query Who does “Bob” know?
54. Let’s index names! <knows>: [uid] . <name>: string @index(term) .
55. Response: { "data": {"q": [ { "uid": "0x2", "name": "Bob", "knows": [{ "uid": "0x1", "name": "Alice" }] }] }, "extensions": {...} } Who does “Bob” know? 0x2 Bob name 0x1 Alice name knows Index on <name> - Alice → 0x1 - Bob → 0x2
56. Types We heard the community and released the first features for our type system with Dgraph v1.1.0. - A node can be of zero or more types. - The relationship is stored in the dgraph.type predicate. - Types can be used to find nodes with the type function. - Types are also used by expand.
57. A type Person First, we defined a type: type Person { name: string knows: [uid] } Then, we can use it in our queries: { q(func: type(Person)) { expand(_all_) } }
58. Updating values with dgraph.type? We would like to say all the nodes with name are of type Person. How would you do it? a) You know all the UIDs already, then just send a mutation. b) You fetch the UIDs in a query, then send a mutation. c) You use … UPSERT!
59. Upsert provides a simple way to query the database and use the resulting data in a mutation. upsert { query { people as var(func: has(name)) {} } mutation { set { uid(people) <dgraph.type> "Person" . } } } Query + Mutation = Upsert
60. Much much more to see Query language: - Filter - Cascade - Normalize - Recurse Schema: - Indexes - Reversed edges ...
61. github.com/campoy/pokegraph Demo time!
63. Did we say we are hiring? dgraph.io/careers
64. Thanks!