Dgraph: the Graph Database written in Go

Slide 1

Slide 1 text

The most advanced graph database in the world

Slide 2

Slide 2 text

Dgraph The graph database written in Go

Slide 3

Slide 3 text

Francesc Campoy VP of Product at Dgraph Labs @francesc Campoy You might know me from: justforfunc.com Google Cloud Platform Podcast About me

Slide 4

Slide 4 text

- Graphs and databases - What’s in a Graph? - Graph Databases - Dgraph - Dgraph Architecture - The Dgraph Query Language - Live Demo! - Q&A Agenda

Slide 5

Slide 5 text

We are hiring! dgraph.io/careers

Slide 6

Slide 6 text

Graphs

Slide 7

Slide 7 text

Steven Spielberg Jaws Jurassic Park genre directed Comedy Thriller Science Fiction directed genre genre genre A movie graph node relationship Legend

Slide 8

Slide 8 text

name: Steven Spielberg name: Jaws year: 1975 name: Jurassic Park year: 1993 genre directed name: Comedy name: Thriller name: Science Fiction directed genre genre genre A movie graph with properties node relationship Legend

Slide 9

Slide 9 text

Graphs in Databases

Slide 10

Slide 10 text

name: Steven Spielberg name: Jaws year: 1975 name: Jurassic Park year: 1993 genre directed name: Comedy name: Thriller name: Science Fiction directed genre genre genre How would you store this in your database?

Slide 11

Slide 11 text

The mapping process can be complex: - one-to-one relationships become foreign keys - one-to-many relationships become foreign keys (repeated foreign keys if reversed) - many-to-many become rows in a new table with multiple foreign keys Traversals require joins which become very expensive quick. A graph in a relational database

Slide 12

Slide 12 text

ID int Name string The “Movie - Director” on a relational DB Movie ID int Name string Year int … DirectorID int FK Director Note: Fetching all the movies directed by a director requires an index for performance.

Slide 13

Slide 13 text

Twilight Zone: The Movie Directed by: - Steven Spielberg - John Landis - Joe Dante - George Miller Good luck migrating the schema! You thought a movie had one director?

Slide 14

Slide 14 text

What is a MovieDirector? We had to modify our logical model to ﬁt the technology, bringing in unnecessary complexity. MovieID int FK DirectorID int FK ID int Name string Year int … ID int Name string The “Movie - Director” on a relational DB Movie Director MovieDirector

Slide 15

Slide 15 text

No foreign keys - You will need to keep many copies of your information. - You will need to keep them all up to date. - At that point, why do you even have a database? Traversals require multiple queries: get element, ﬁnd property, get next element, etc. A graph in a non-relational database

Slide 16

Slide 16 text

Fetching all the names of the movies directed by a director requires n+1 queries. The “Movie - Director” on a no-SQL DB (A) Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ 123, 234, 345 ] } Movie document { “_id”: 123, “name”: “Jaws”, “year”: ... } Movie document { “_id”: 234, “name”: “E.T.”, “year”: ... } Movie document { “_id”: 345, “name”: “Jurassic Park”, “year”: ... }

Slide 17

Slide 17 text

So … what if a movie has multiple directors? The “Movie - Director” on a no-SQL DB (B) Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ { “_id”: 123, “name”: “Jaws”, “year”: … }, { “_id”: 234, “name”: “E.T.”, “year”: … }, { “_id”: 345, “name”: “Jurassic Park”, “year”: … } ] }

Slide 18

Slide 18 text

Director document { “_id”: 111, “name”: “Steven Spielberg”, “movies”: [ { “_id”: 123, “name”: “Jaws”, “year”: … }, { “_id”: 234, “name”: “E.T.”, “year”: … }, { “_id”: 345, “name”: “Jurassic Park”, “year”: … } ] } The “Movie - Director” on a no-SQL DB (C) Movie document { “_id”: 123, “name”: “Jaws”, “year”: ... } Movie document { “_id”: 234, “name”: “E.T.”, “year”: ... } Movie document { “_id”: 345, “name”: “Jurassic Park”, “year”: ... } Now we can fetch all movies for a director in a query … but we might easily lose consistency.

Slide 19

Slide 19 text

Graph Databases

Slide 20

Slide 20 text

No need for mapping, the “whiteboard” model is your model. No need for “joins”: - traversals are fast since relationships point directly to nodes, not keys - “Index-free adjacency” - deep traversals are possible (and efﬁcient) Graph Databases

Slide 21

Slide 21 text

Dgraph Data Modeling

Slide 22

Slide 22 text

Subject-Predicate-Value: Subject Predicate Value Jaws 1975 Subject-Predicate-Object: Subject Predicate Object Jaws Steven Spielberg Subject-Predicate-Value name: Steven Spielberg name: Jaws was directed by name: Jaws year: 1975

Slide 23

Slide 23 text

The previous slide is not 100% accurate, as nodes have their own identifiers. So instead of using strings as identifiers: “Jaws” 1975 “Jaws” “Steven Spielberg” We have Universal Identifiers (UIDs): 0x1 “Jaws” 0x1 1975 0x1 0x2 0x2 “Steven Spielberg” Dgraph data modeling

Slide 24

Slide 24 text

Given the data from before: 0x1 “Jaws” 0x1 1975 0x1 0x2 0x2 “Steven Spielberg” - 0x1 and 0x2 are UIDs (Universal IDentiﬁers). - , , etc. are predicates. - “Jaws”, “Steven Spielberg”, and 1975 are values. Dgraph data modeling

Slide 25

Slide 25 text

Predicates are always attached to UIDs. We associate values and objects to keys composed by UID + predicate Keys Values 0x1: “Jaws” 0x1: 1975 0x1: 0x2 0x2: “Steven Spielberg” Sometimes a value can be an array of UIDs or values. Dgraph data modeling

Slide 26

Slide 26 text

1. Find the starting nodes for the traversal. 2. Append the predicate name to the UIDs we have so far. 3. Find the values associated to the UID:Predicate pairs. 4. Repeat (2) until the associated values are non UID or query is done. Beneﬁt: values are not involved, keeping memory requirements low. Life of a query

Slide 27

Slide 27 text

Life of a query Example: give me the name of the friends of 0x1234. 0x1234 _ X 1. Find the node with UID 0x1234 2. Append 0x1234 3. Retrieve values from 0x1234: [0xABCD, 0xBCDE] a. 0xABCD “Diggy” | 0xBCDE “Augie” 4. Return [“Diggy”, “Augie”]

Slide 28

Slide 28 text

How do we find the first nodes? We don’t always have the UID of the first node of our traversal. We can find them by the value of one of its predicates! - Node with name “Augie”. - All nodes with a predicate larger than 18. - All nodes with a predicate 20mi around SLC. These searches could be very expensive, so we use indices.

Slide 29

Slide 29 text

Indexing in Dgraph Dgraph provides indices on: - Strings: hash, exact, term, fulltext, trigram. - DateTime: year, month, day, hour. - Int, Float, Bool: default value index. - Geo properties : default value index.

Slide 30

Slide 30 text

- Schemas are not required in general. - But indices can only be deﬁned on schema ﬁelds. - Be aware of the space requirements of the indices. Example, and indexed name predicate: : string @index(fulltext, hash, term, trigram) . Dgraph schemas and indices

Slide 31

Slide 31 text

1. Find the starting nodes for the traversal using UIDs or indexes. 2. Append the predicate ID (int) to the UIDs we have so far. 3. Find the values associated to the UID:Predicate pairs. 4. Repeat (2) until the associated values are non UID or query is done. Beneﬁt: values are not involved, keeping memory requirements low. Updated life of a query

Slide 32

Slide 32 text

Dgraph Architecture

Slide 33

Slide 33 text

Dgraph Architecture at a glance Alpha Alpha Alpha Zero Zero Zero Ratel Zero: Cluster Management Alpha: Data Storage Ratel: Web UI

Slide 34

Slide 34 text

Dgraph Zero ● Called Zero, because it used to be group zero. ● Handles cluster membership, transactions and ID generation. ● Each Dgraph cluster must have at least one Zero. ● Participates in a Raft group for HA.

Slide 35

Slide 35 text

Dgraph Alpha ● Called Alpha, because it runs group 1 and higher. ● Stores data, serves queries. ● Each Dgraph cluster must have at least one Alpha. ● Participates in a Raft group for HA.

Slide 36

Slide 36 text

Still hiring! dgraph.io/careers

Slide 37

Slide 37 text

● Not part of the cluster, but useful for exploration. ● Connects to Alphas to respond to queries. ● Completely optional, but deﬁnitely useful. Dgraph Ratel

Slide 38

Slide 38 text

Dgraph Architecture at a glance Alpha Alpha Alpha Zero Zero Zero Ratel Zero: Cluster Management Alpha: Data Storage Ratel: Web UI

Slide 39

Slide 39 text

- Graph Databases make storing and retrieving graphs efﬁcient. - Dgraph provides Subject-Predicate-(Object/Value) - Traversals are very efﬁcient thanks to optimizing for low-latency disk seeks and minimizing network calls. - Dgraph architecture provides high availability and horizontal scalability. Conclusion

Slide 40

Slide 40 text

GraphQL+-

Slide 41

Slide 41 text

- Heavily inspired by GraphQL. - Modiﬁed to serve as a better DB language. - Pure GraphQL native support is coming up soon™ - Try it on playground instance of Ratel: play.dgraph.io GraphQL+-

Slide 42

Slide 42 text

Create a new UID and associate it to value “Alice” with predicate . { set { _:alice "Alice" . } } GraphQL+- mutations

Slide 43

Slide 43 text

Output: { "data": { "code": "Success", "message": "Done", "uids": { "alice": "0x1" } }, "extensions": {...} } GraphQL+- mutations

Slide 44

Slide 44 text

Create a new UID and associate it to value “Alice” with predicate . { set { _:bob "Bob" . _:bob <0x1> . } } GraphQL+- mutations

Slide 45

Slide 45 text

Output: { "data": { "code": "Success", "message": "Done", "uids": { "bob": "0x2" } }, "extensions": {...} } GraphQL+- mutations

Slide 46

Slide 46 text

Current dataset 0x1 0x2 knows Alice Bob name name

Slide 47

Slide 47 text

Our schema Default schema: - default types - no indices

Slide 48

Slide 48 text

Let’s fetch people’s names Query: { q(func: uid(0x2, 0x1)) { uid name } }

Slide 49

Slide 49 text

Let’s fetch people’s names Response: { "data": { "q": [ { "uid": "0x1", "name": "Alice" }, { "uid": "0x2", "name": "Bob" }] }, "extensions": {...} } 0x1 0x2 Alice Bob name name

Slide 50

Slide 50 text

Query: { q(func: uid(0x2)) { uid name knows { uid name } } } Who does Bob know?

Slide 51

Slide 51 text

Response: {"data": {"q": [{ "uid": "0x2", "name": "Bob", "knows": [{ "uid": "0x1", "name": "Alice" }] }]}, "extensions": {...} } Who does Bob know? 0x2 Bob name 0x1 Alice name knows

Slide 52

Slide 52 text

Query: { q(func: eq(name, “Bob”)) { uid name knows { uid name } } } Who does “Bob” know?

Slide 53

Slide 53 text

Response: Error Name: t Message: : Attribute name is not indexed. URL: http://localhost:8080/query Who does “Bob” know?

Slide 54

Slide 54 text

Let’s index names! : [uid] . : string @index(term) .

Slide 55

Slide 55 text

Response: { "data": {"q": [ { "uid": "0x2", "name": "Bob", "knows": [{ "uid": "0x1", "name": "Alice" }] }] }, "extensions": {...} } Who does “Bob” know? 0x2 Bob name 0x1 Alice name knows Index on - Alice → 0x1 - Bob → 0x2

Slide 56

Slide 56 text

Types We heard the community and released the ﬁrst features for our type system with Dgraph v1.1.0. - A node can be of zero or more types. - The relationship is stored in the dgraph.type predicate. - Types can be used to ﬁnd nodes with the type function. - Types are also used by expand.

Slide 57

Slide 57 text

A type Person First, we deﬁned a type: type Person { name: string knows: [uid] } Then, we can use it in our queries: { q(func: type(Person)) { expand(_all_) } }

Slide 58

Slide 58 text

Updating values with dgraph.type? We would like to say all the nodes with name are of type Person. How would you do it? a) You know all the UIDs already, then just send a mutation. b) You fetch the UIDs in a query, then send a mutation. c) You use … UPSERT!

Slide 59

Slide 59 text

Upsert provides a simple way to query the database and use the resulting data in a mutation. upsert { query { people as var(func: has(name)) {} } mutation { set { uid(people) "Person" . } } } Query + Mutation = Upsert

Slide 60

Slide 60 text

Much much more to see Query language: - Filter - Cascade - Normalize - Recurse Schema: - Indexes - Reversed edges ...