Graph vs Graph - Speaker Deck

Slide 1

Slide 1 text

Graph vs Graph GraphQL as the API for your graph database Rhys Evans, @wheresrhys

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

If your data is a graph, and we’re all thinking in graphs, why is it that only one thin layer actually is a graph?

Slide 6

Slide 6 text

Rhys Evans Principal Engineer Reliability Engineering Financial Times @wheresrhys

Slide 7

Slide 7 text

Why graphs interest us at the FT

Slide 8

Slide 8 text

Why we use neo4j graph database

Slide 9

Slide 9 text

Combining GraphQL and neo4j

Slide 10

Slide 10 text

Principles for easy(ish) graph evolution

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Just one system

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Some credentials have been leaked – could any user data have been compromised?

Slide 17

Slide 17 text

System Data Store Credentials Data Type

Slide 18

Slide 18 text

We’ve changed our T&C’s – which affiliates do we ask to update their websites?

Slide 19

Slide 19 text

Website Affiliate Smallprint

Slide 20

Slide 20 text

Which products cost more to run than they raise in revenue?

Slide 21

Slide 21 text

Product Infrastructure Revenue Stream Budget Stream

Slide 22

Slide 22 text

www.ft.com front page is returning a 404 – who can I contact to fix it at 3am?

Slide 23

Slide 23 text

System User Journey Monitoring Maintainer

Slide 24

Slide 24 text

“Wir müssen wissen, wir werden wissen.” ― David Hilbert

Slide 25

Slide 25 text

We come to understand things by constructing a model

Slide 26

Slide 26 text

Systems Teams System Team Join People Team Person Join System Person Join Budget lines Modules Data stores System Data Read Join System Module Join System Data Write Join

Slide 27

Slide 27 text

“Recursive dependents of systems reading from a DB” = Table x Table x Table x ...

Slide 28

Slide 28 text

All the Systems All the System joins All the Systems All the System joins All the Systems All the System joins ...

Slide 29

Slide 29 text

O(no)

Slide 30

Slide 30 text

Graph databases such as neo4j optimise for data which is highly connected

Slide 31

Slide 31 text

No global lookup by ID Uses a local pointer to the exact physical location of the data

Slide 32

Slide 32 text

Record Linked list of relationships Related records

Slide 33

Slide 33 text

“Recursive dependents of systems reading from a DB” = record –> siblings –> cousins -> ...

Slide 34

Slide 34 text

O(k)

Slide 35

Slide 35 text

(This)-[:IS_CONNECTED_TO]->(That) MATCH(s:System)-[:DEPENDS_ON*]->(:System) -[:READS_FROM]->(d:Database) WHERE d.code = "credit-cards" RETURN s.accessLogsUrl Cypher ≃ ASCII art + SQL

Slide 36

Slide 36 text

CREATE CONSTRAINT ON (s:System) ASSERT s.code IS UNIQUE CREATE (s:System) SET s = $properties MERGE (s)-[r:DEPENDS_ON]-(s2) WHERE s2.code = “dependency” RETURN s, r, s2 Constructing the graph

Slide 37

Slide 37 text

Our graph model

Slide 38

Slide 38 text

Where does GraphQL fit in?

Slide 39

Slide 39 text

1. Self-service 2. Everyone a power user 3. Low effort extensibility 4. API and UI for everything

Slide 40

Slide 40 text

1. Self-service 2. Everyone a power user 3. Low effort extensibility 4. API and UI for everything

Slide 41

Slide 41 text

We need an API that can represent graphs and allows users to define the data they need

Slide 42

Slide 42 text

Slide 43

Slide 43 text

neo4j-graphql-js - converts GraphQL queries to cypher - resolves with a single database query

Slide 44

Slide 44 text

Cypher: (This)-[:RELATED_TO]->(That) GraphQL: type This { relatedThats: [That] } type That { relatedThiss: [This] } Different semantics

Slide 45

Slide 45 text

@relation directive type System { code: String sla: SLA dependencies: [System] @relation(name: "DEPENDS_ON", direction: "OUT") dependents: [System] @relation(name: "DEPENDS_ON", direction: "IN") }

Slide 46

Slide 46 text

import { v1 as neo4j } from 'neo4j-driver'; import { makeAugmentedSchema } from 'neo4j-graphql-js'; const driver = neo4j.driver(...); router.use('/graphql', graphqlExpress(() => ({ schema: makeAugmentedSchema({typeDefs}), context: { driver } }) ) Resolver generation

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

N + 1 problem goes away N + 1 → 1 N(M + 1) + 1 → 1 N(M(P + 1) + 1) + 1 → 1

Slide 51

Slide 51 text

{ Systems(filter: { knownAboutBy_every:{isActive:false} }){ code } } Filters

Slide 52

Slide 52 text

type Team { stakeholderTeams: [Team] @cypher( statement: "MATCH (this)<-[:DELIVERED_BY]-(:System) <-[:DEPENDS_ON*]-(:System)<-[:DELIVERED_BY]- (t:Team) RETURN t" ) } @cypher directive

Slide 53

Slide 53 text

CALL algo.pageRank('System', 'DEPENDS_ON', {iterations:20, dampingFactor:0.85, write: true, writeProperty:"criticality"}) type System { code: String criticality: Float } AI - Graph algorithms

Slide 54

Slide 54 text

#GRANDstack GraphQL + React + Apollo + Neo4j Database https://grandstack.io/

Slide 55

Slide 55 text

1. Self-service 2. Everyone a power user 3. Low effort extensibility 4. API and UI for everything

Slide 56

Slide 56 text

Say we want to add a new type and edge to the graph System Hosting Platform HOSTED_BY

Slide 57

Slide 57 text

How do we add this to the data layer? System Hosting Platform HOSTED_BY

Slide 58

Slide 58 text

CREATE CONSTRAINT ON (h:HostingPlatform) ASSERT h.code IS UNIQUE We just create an index

Slide 59

Slide 59 text

How do we add this to the API layer? System Hosting Platform HOSTED_BY

Slide 60

Slide 60 text

type HostingPlatform { code: String hostsSystems: [System] @relation(name: "HOSTED_ON", direction: "IN") } extend type System { hostedOn: [HostingPlatform] @relation(name: "DEPENDS_ON", direction: "OUT") } Add it to the schema

Slide 61

Slide 61 text

Schema first vs Code first This is schema first… right?

Slide 62

Slide 62 text

Why prefer Code First? ‑ DRY ‑ Declarative (via static analysis/coding patterns)

Slide 63

Slide 63 text

neo4j-graphql-js: - DRY - only write schema - Static analysis of schema generates resolvers

Slide 64

Slide 64 text

Hot reloading Updating the schema and API without redeploying

Slide 65

Slide 65 text

// function that returns GraphQL middleware const constructAPI = schema => { api = graphqlExpress(() => ({ schema: makeAugmentedSchema({schema}) }) } let api; constructAPI(initialSchema) schemaFilePoller.on('change', constructAPI); app.post('/graphql', (...args) => api(...args)); Schema hot reloading

Slide 66

Slide 66 text

How do we add this to the UI layer? System Hosting Platform HOSTED_BY

Slide 67

Slide 67 text

Our UI is a pretty standard set of JSX components

Slide 68

Slide 68 text

Each primitive type - String, Boolean etc - has a corresponding component

Slide 69

Slide 69 text

GraphQL types are rendered using combinations of these primitive components

Slide 70

Slide 70 text

Relationship editor Boolean editor Enum editor

Slide 71

Slide 71 text

Metadata available in GraphQL - Type - Property name - Description

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

also - Handling inactive records - Required fields - Validation patterns ...

Slide 76

Slide 76 text

Defining these in a different location to the schema is hard to maintain

Slide 77

Slide 77 text

Custom yaml schema name: System description: Any combination of … properties: code: type: String required: true useInSummary: true pattern:^(?=.{2,64}$)[a-z0-9]+(?:-[a-z0-9]+)*$ label: Code description: The unique id …

Slide 78

Slide 78 text

const types = schema.getTypes().map(defineType); const enums = schema.getEnums().map(defineEnum); const queries = schema.getTypes().map(defineQueries) return [].concat( types, 'type Query {\n', ...queries, '}', enums, ); Transform to GraphQL SDL

Slide 79

Slide 79 text

Schema files GraphQL API Admin UI API Search index REST API

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

CFECSWYD

Slide 82

Slide 82 text

Code first... except the code is a schema written in YAML... development

Slide 83

Slide 83 text

Because what really matters ‑ DRY ‑ Declarative (or static) ‑ Co-location

Slide 84

Slide 84 text

GraphQL feature request: Support front matter in field descriptions so we can all stop saying CFECSWYD!

Slide 85

Slide 85 text

GraphQL + neo4j is the one true path?

Slide 86

Slide 86 text

Of course not

Slide 87

Slide 87 text

neo4j a poor choice for: - Large documents/blobs - Time series - Other things SQL/NoSQL perform well at

Slide 88

Slide 88 text

neo4j-graphql-js only generates resolvers where your code does not already provide one