Going Full Circle with Virtual Graphs and Relational Databases

Many years ago I gave presentations based on my work on Spring Data Neo4j: Going from relational databases to databases with relations. Back then I had to choose: Having data in both relational databases and import them into a Graph database for more insights.

WIth Neo4j announcing Virtual Graphs in the summer of 2026, I do no longer have to chose: I can connect Neo4j to any relational databases and query my tables like I would query a graph.

In this session I pick up my favourite topic doing database presentations: Musical habits and statistics. I will demonstrate kind of a medallion architecture, in which a relational database receives raw listening data (scrobbels), basically the bronze level, and how it can aggregate it into silver level and eventually, connect Neo4j Virtual Graph to that dataset and create gold level value by connecting aggregated data with context only a graph can produce.

1. Neo4j Virtual Graph Going Full circle with graph- and relational databases
2. Can your relational database answer that? 2
3. The real question: Can it be queried like that? 3
4. SQL didn’t fit one screen. WITH RECURSIVE "qpp" AS ( SELECT "unnamed1"."id" AS "unnamed1_id_left_key", "unnamed1"."created_at" AS "unnamed1_created_at_left", "unnamed1"."id" AS "unnamed1_id_left", // ...several pages later 4
5. Who is that guy? • Joined Neo4j 2018 with a strong relational background to work on Neo4j-OGM (Object Graph Mapping) and SDN (Spring Data Integration) 5 Neo4j Inc. All rights reserved 2024 Michael Simons Java & Testcontainers Champion, Senior Staff Engineer bei Neo4j
6. Who is that guy? • • Joined Neo4j 2018 with a strong relational background to work on Neo4j-OGM (Object Graph Mapping) and SDN (Spring Data Integration) Brought his whole knowledge of ORMs (Object relational Mapping) and SQL 6 Neo4j Inc. All rights reserved 2024 Michael Simons Java & Testcontainers Champion, Senior Staff Engineer bei Neo4j
7. Who is that guy? • • • Joined Neo4j 2018 with a strong relational background to work on Neo4j-OGM (Object Graph Mapping) and SDN (Spring Data Integration) Brought his whole knowledge of ORMs (Object relational Mapping) and SQL Manages the Neo4j-JDBC driver since 2024 with an optional SQL to Cypher translation layer 7 Neo4j Inc. All rights reserved 2024 Michael Simons Java & Testcontainers Champion, Senior Staff Engineer bei Neo4j
8. Who is that guy? • • • • Joined Neo4j 2018 with a strong relational background to work on Neo4j-OGM (Object Graph Mapping) and SDN (Spring Data Integration) Brought his whole knowledge of ORMs (Object relational Mapping) and SQL Manages the Neo4j-JDBC driver since 2024 with an optional SQL to Cypher translation layer Thinks now Virtual Graph is a fantastic Object Mapping framework 🙃 8 Neo4j Inc. All rights reserved 2024 Michael Simons Java & Testcontainers Champion, Senior Staff Engineer bei Neo4j
9. Agenda 9 1 What is Virtual Graph? 4 Which tools does it work with? 2 How does it work? 5 Where is available? 3 Demo 6 Outlook
10. What is Virtual Graph? A high level overview and the big question: Is it deterministic? 10
11. What is Neo4j Virtual Graph? • • • • 11 A core functionality of Neo4j Connects to a remote (relational) datasource Creates a “virtual” graph based on your table definitions Let’s you query that graph like you can query a normal Neo4j graph
12. Example: Fraud detection Given the relational schema storing transactions between accounts: CREATE TABLE IF NOT EXISTS accounts ( id INTEGER PRIMARY KEY, name VARCHAR(64) NOT NULL, email VARCHAR(64) NOT NULL ); CREATE TABLE IF NOT EXISTS transactions ( id INTEGER PRIMARY KEY, from_id INTEGER NOT NULL, to_id INTEGER NOT NULL, created_at TIMESTAMP NOT NULL, amount DECIMAL(9,2) NOT NULL, CONSTRAINT fk_account_from FOREIGN KEY(from_id) REFERENCES accounts (id), CONSTRAINT fk_account_to FOREIGN KEY(to_id) REFERENCES accounts (id) ); 12
13. Graph model: Transaction data Defining the mapping 13 CREATE OR REPLACE VIRTUAL GRAPH DATABASE fraud_detection SET DATASOURCE {type: 'duckdb', path: ‘fraud_detection.db'} SET SECRET {type: 'anonymous'} SET MAPPING NODE TABLES ( accounts LABEL Account PROPERTIES ARE ALL COLUMNS EXCEPT (id), transactions LABEL Transaction PROPERTIES (CREATED_AT AS date, AMOUNT AS amount) ) RELATIONSHIP TABLES ( transactions KEY (id, to_id) SOURCE KEY (id) REFERENCES transactions (id) DESTINATION KEY (to_id) REFERENCES accounts (id) LABEL TO, transactions KEY (id, from_id) SOURCE KEY (id) REFERENCES transactions (id) DESTINATION KEY (from_id) REFERENCES accounts (id) LABEL FROM )
14. Graph model: UI based mapping in Aura 14
15. Deterministic graph model 15
16. Deterministic queries 16
17. How does it work? A peak under the hood so you get the ORM reference. 17
18. Architecture Cypher Stack Neo4j DBMS Parsing Relational Database Semantic Analysis MATCH (n:Movie) RETURN n; Query Graph Construction Logical Planning Physical Planning Pooled JDBC connections prepared statements Execution Result 18
19. Cypher results Cypher SQL Relational results mapping: Cypher × Graph model → SQL × Projections
20. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) RETURN tom_hanks .born_year AS tom_hanks_yob
21. Cypher MATCH (tom_hanks:Person {name: 'Tom Hanks'}) RETURN tom_hanks.born_year AS tom_hanks_yob SQL FROM people
22. Cypher MATCH (tom_hanks:Person {name: 'Tom Hanks'}) RETURN tom_hanks.born_year AS tom_hanks_yob SQL FROM people AS tom_hanks
23. Cypher MATCH (tom_hanks:Person {name: 'Tom Hanks'}) RETURN tom_hanks.born_year AS tom_hanks_yob SQL FROM people AS tom_hanks WHERE tom_hanks.name = 'Tom Hanks'
24. Cypher MATCH (tom_hanks:Person {name: 'Tom Hanks'}) RETURN tom_hanks.born_year AS tom_hanks_yob SQL SELECT tom_hanks.born AS tom_hanks_yob FROM people AS tom_hanks WHERE tom_hanks.name = 'Tom Hanks'
25. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) RETURN tom_hanks .born_year AS tom_hanks_yob +-----------------+----+------------------------------------------------+ | Operator | Id | Details | +-----------------+----+------------------------------------------------+ | +ProduceResults | 1 | tom_hanks_yob | | | +----+------------------------------------------------+ | +External | 0 | SELECT | | | | "tom_hanks"."born" AS "tom_hanks_yob" | | | | FROM "main"."people" AS "tom_hanks" | | | | WHERE ("tom_hanks"."name" = ? /*autostring0*/ ) | +-----------------+----+------------------------------------------------+
26. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) RETURN tom_hanks +-----------------+----+--------------------------------------------------------------------------------------------+ | Operator | Id | Details | +-----------------+----+--------------------------------------------------------------------------------------------+ | +ProduceResults | 2 | tom_hanks | | | +----+--------------------------------------------------------------------------------------------+ | +Projection | 1 | (:Person {id: tom_hanks_id, born_year: tom_hanks_born, name: tom_hanks_name}) AS tom_hanks | | | +----+--------------------------------------------------------------------------------------------+ | +External | 0 | SELECT | | | | "tom_hanks"."id" AS "tom_hanks_id", | | | | "tom_hanks"."born" AS "tom_hanks_born", | | | | "tom_hanks"."name" AS "tom_hanks_name" | | | | FROM "main"."people" AS "tom_hanks" | | | | WHERE ("tom_hanks"."name" = ? /*autostring0*/) | +-----------------+----+--------------------------------------------------------------------------------------------+
27. MATCH (tom_hanks :Person {name: 'Tom Hanks'})-[acted_in :ACTED_IN]->(movie) RETURN movie FROM people AS tom_hanks JOIN movie_actors AS acted_in ON acted_in.person_id = tom_hanks.id JOIN movies AS movie ON movie.id = acted_in.movie_id
28. MATCH (tom_hanks :Person {name: 'Tom Hanks'})-[acted_in :ACTED_IN]->(movie) RETURN movie +-----------------+----+------------------------------------------------------------------------------------------------------+ | Operator | Id | Details | +-----------------+----+------------------------------------------------------------------------------------------------------+ | +ProduceResults | 2 | movie | | | +----+------------------------------------------------------------------------------------------------------+ | +Projection | 1 | (:Movie {id: movie_id, release_year: movie_released, tagline: movie_tagline, title: movie_title}) AS | | | | | movie | | | +----+------------------------------------------------------------------------------------------------------+ | +External | 0 | SELECT | | | | "movie"."id" AS "movie_id", | | | | "movie"."released" AS "movie_released", | | | | "movie"."tagline" AS "movie_tagline", | | | | "movie"."title" AS "movie_title" | | | | FROM "main"."people" AS "tom_hanks" | | | | JOIN "main"."movie_actors" AS "acted_in" ON "acted_in"."person_id" = "tom_hanks"."id" | | | | JOIN "main"."movies" AS "movie" ON "movie"."id" = "acted_in"."movie_id" | | | | WHERE ("tom_hanks"."name" = ? /*autostring0*/) | +-----------------+----+------------------------------------------------------------------------------------------------------+
29. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) MATCH p = (tom_hanks)-[acted_in :ACTED_IN]-> (movie)<-[r](person_1)-[follows :FOLLOWS](person_2) WHERE 2000 <= movie.release_year < 2010 RETURN movie.title AS movie_title, person_1 .name, person_2, p
30. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) MATCH p = (tom_hanks)-[acted_in :ACTED_IN]-> (movie :Movie)<-[r](person_1 :Person)-[follows :FOLLOWS](person_2 :Person) WHERE 2000 <= movie.release_year < 2010 RETURN movie.title AS movie_title, person_1 .name, person_2, p
31. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) MATCH p = (tom_hanks)-[acted_in :ACTED_IN]-> (movie:Movie)<-[r:ACTED_IN:DIRECTED:PRODUCED:REVIEWED:WROTE](person_1 :Person)-[follows :FOLLOWS](person_2 :Person) WHERE 2000 <= movie.release_year < 2010 RETURN movie.title AS movie_title, person_1 .name, person_2, p
32. MATCH (tom_hanks :Person {name: 'Tom Hanks' }) MATCH p = (tom_hanks)-[acted_in :ACTED_IN]-> Relationship type union (movie:Movie)<-[r:ACTED_IN:DIRECTED:PRODUCED:REVIEWED:WROTE](person_1 :Person)-[follows :FOLLOWS]Undirected relationship (person_2 :Person) WHERE 2000 <= movie.release_year < 2010 RETURN movie.title AS movie_title, person_1 .name, person_2, p Path projection Relationship uniqueness
33. Recap • Decompose the Cypher AST ◦ • Figure out what tables back a Node or a relationship ◦ • In the ORM world, the types represent an AST and annotations augment it In the ORM world that would be a type or member name, or the value of an annotation “We need joins, lots of them” ◦ (Well, not when we can deduce from the Cypher AST that they are superfluous) 33
34. Demo “Play It Again, Sam.” 34
35. Transactions example in Snowflake 35
36. Related music vs similar music 36
37. Related music vs similar music 37
38. What I used to map this to: @Entity @Table(name = "tracks") class Track { @OneToMany(mappedBy = "track") private Set<PlayedOn> days = new HashSet<>(); private String name; } @Entity @Table(name = "played_on") public class PlayedOn { @EmbeddedId private PlayedOnInd id; @ManyToOne @MapsId("trackId") private Track track; @ManyToOne @MapsId("dayId") private Day day; private LocalDateTime playedOn; } @Entity @Table(name = "tracks") class Day { private String name; } 38
39. What I map this now to CREATE OR REPLACE VIRTUAL GRAPH DATABASE music SET DATASOURCE {type: 'duckdb', path: 'music.db'} SET SECRET {type: 'anonymous'} SET MAPPING NODE TABLES ( tracks LABEL Track PROPERTIES ARE ALL COLUMNS EXCEPT (id), days LABEL Day PROPERTIES ARE ALL COLUMNS EXCEPT (id) ) RELATIONSHIP TABLES ( played_on KEY (track_id, day_id) SOURCE KEY (track_id) REFERENCES track (id) DESTINATION KEY (day_id) REFERENCES day (id) LABEL PLAYED_ON ) // Some more… 39
40. What I map this now to CREATE OR REPLACE VIRTUAL GRAPH DATABASE music SET DATASOURCE {type: 'duckdb', path: 'music.db'} SET SECRET {type: 'anonymous'} SET MAPPING NODE TABLES ( tracks LABEL Track PROPERTIES ARE ALL COLUMNS EXCEPT (id), days LABEL Day PROPERTIES ARE ALL COLUMNS EXCEPT (id) ) RELATIONSHIP TABLES ( played_on KEY (track_id, day_id) SOURCE KEY (track_id) REFERENCES track (id) DESTINATION KEY (day_id) REFERENCES day (id) LABEL PLAYED_ON ) // Some more… 40
41. What I can based on it: Project into GDS MATCH (g:Genre)<-[:HAS]-(al:Album)-[:CONTAINS]->(t:Track)-[p:PLAYED_ON]->(d:Day), (al)-[:RELEASED_BY]->(ar:Artist) MATCH (d)<-[q:PLAYED_ON]-(:Track) WITH t,p.playedOn AS played, q.playedOn AS windowStart, id(ar) AS artistId,id(g) AS genreId UNWIND CASE WHEN windowStart <= played AND played < windowStart + duration('PT1H') THEN [windowStart] ELSE [] END AS w WITH DISTINCT t, datetime({datetime: w, timezone: 'UTC'}).epochSeconds AS window, artistId, genreId RETURN gds.graph.project( $graphName, t, window, { sourceNodeLabels: 'Track', targetNodeLabels: 'Window', relationshipType: 'PLAYED_IN', sourceNodeProperties: {refs: [artistId, genreId]}, targetNodeProperties: null }, {undirectedRelationshipTypes: ['*']} ).nodeCount AS nodeCount 41
42. Compute similarity, cluster and rank CALL gds.nodeSimilarity.filtered.mutate($graphName, { nodeLabels: ['Track', 'Window'], relationshipTypes: ['PLAYED_IN'], sourceNodeFilter: 'Track', targetNodeFilter: 'Track', degreeCutoff: 2, topK: 5, similarityCutoff: 0.1, mutateRelationshipType: 'CO_PLAYED', mutateProperty: 'score'}) YIELD nodesCompared, relationshipsWritten RETURN nodesCompared, relationshipsWritten Two tracks are similar when they have at least two shared windows in common -> Marked with a new relationship CO_PLAYED 42
43. Compute similarity, cluster and rank CALL gds.leiden.mutate($graphName, { nodeLabels: ['Track'], relationshipTypes: ['CO_PLAYED_UNDIRECTED'], relationshipWeightProperty: 'score', gamma: 8.0, randomSeed: 42, mutateProperty: 'community'}) YIELD communityCount, modularity RETURN communityCount, modularity Create clusters of tracks co played together 43
44. Compute similarity, cluster and rank CALL gds.pageRank.mutate($graphName, { nodeLabels: ['Track'], relationshipTypes: ['CO_PLAYED'], relationshipWeightProperty: 'score', mutateProperty: 'centrality'}) YIELD ranIterations RETURN ranIterations 44
45. What I get: A recommendation graph 45
47. Which tools does it work with? On the Neo4j side of things, but also on the relational one. 47
48. Datasources • 48 Official support for driver and the SQL dialects of (Aura and Self-managed) ◦ Big Query ◦ Databricks ◦ S3 Tables with Iceberg catalog ◦ Snowflake
49. Datasources • • 49 Official support for driver and the SQL dialects of (Aura and Self-managed) ◦ Big Query ◦ Databricks ◦ S3 Tables with Iceberg catalog ◦ Snowflake Official support for driver and SQL dialects of (Self-managed only) ◦ Clickhouse ◦ DuckDB ◦ Motherduck ◦ Postgres ◦ SQLite
50. Datasources • • 50 Generic driver: Can use any JDBC driver with a generic dialect ◦ Advanced translations (QPP and such) won’t work ◦ Not 100% type fidelity Can be graduated with full support through a product request via support
51. All Neo4j drivers are supported • • • 51 Idiomatic language drivers (Java, .Net, Go, JavaScript, Python) Neo4j JDBC Driver Query API
52. Mapping format • • • 52 Graphic modelling in Aura ◦ Automatic creation via assistant ◦ Derived from relational schema ◦ Manual SQL/PG compatible syntax Ontology spec (in development)
53. Neo4j tools supported • • • • Browser (Query) Bloom (Explore) Dashboards GDS (some restrictions apply, will be lifted in a future release) Essentially anything that does not write back and does not require stable long ids 53
54. Composite support Full support for Composite (local and remote) and aliases on self-managed: neo4j@neo4j> show databases yield name, type, aliases, constituents; +----------------------------------------------------------------------------------+ | name | type | aliases | constituents | +----------------------------------------------------------------------------------+ | "comp" | "composite" | [] | ["comp.movies", "comp.x"] | | "neo4j" | "virtual graph" | ["comp.movies"] | [] | | "system" | "system" | [] | [] | | "anotherDb" | "standard" | ["comp.x"] | [] | +----------------------------------------------------------------------------------+ Fully supported as constituent in Composite on Aura later. 54
55. Where is it available? Pick your destiny. 55
56. Where is it available? • • 56 Aura GA this month Self-managed preview in December, GA early Q1 2027 ◦ Can be enabled via feature flag today, Cypher syntax not ready yet
57. Outlook What are we working on next? 57
58. Outlook • • • 58 More data sources (demand based) Extended Cypher support Graph materialisation
59. Thank you. Repository for the demo: https://github.com/michael-simons/nodes2026 59