Neo4j-Migrations: The lean way of applying transformations to a Graph Database @ JavaSummit 2023

Slide 1

Slide 1 text

© 2023 Neo4j, Inc. All rights reserved. © 2023 Neo4j, Inc. All rights reserved. Neo4j-Migrations: The lean way of applying refactorings transformations to a Graph Database Michael Simons @rotnroll666(@mastodon.social) Java Champion & Staff Software Engineer at Neo4j 1

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2023 Neo4j, Inc. All rights reserved. Entity relationship diagrams (ERD) They come in different forms and levels of abstraction ● Conceptual diagrams ● Logical diagrams ● Physical diagrams While mostly used for relational DBMS, they help also with ● General database design and debugging ● Some aid in requirements engineering ● Creating and patching database schema and content 3 Michael Simons at Java Summit 2023

Slide 4

Slide 4 text

© 2023 Neo4j, Inc. All rights reserved. Conceptual ERD Definition of what exists in a system, not how it manifests itself 4 Michael Simons at Java Summit 2023 https://www.visual-paradigm.com/guide/data-modeling/what-is-entity-relationship-diagram/

Slide 5

Slide 5 text

© 2023 Neo4j, Inc. All rights reserved. Logical ERD Enrichment of the conceptual model 5 Michael Simons at Java Summit 2023 https://www.visual-paradigm.com/guide/data-modeling/what-is-entity-relationship-diagram/

Slide 6

Slide 6 text

© 2023 Neo4j, Inc. All rights reserved. Physical ERD Concrete implementation details 6 Michael Simons at Java Summit 2023 https://www.visual-paradigm.com/guide/data-modeling/what-is-entity-relationship-diagram/

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

© 2023 Neo4j, Inc. All rights reserved. Forms of physical models in Neo4j CREATE CONSTRAINT customer_name FOR (c:Customer) REQUIRE c.name IS NOT NULL; CREATE CONSTRAINT bike_model FOR (b:Bike) REQUIRE b.model IS UNIQUE; 10 Michael Simons at Java Summit 2023

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© 2023 Neo4j, Inc. All rights reserved. History: Refactoring Databases by Ambler & Sadalage ● Published in 2006 ● Evolutionary database design to unblock continuous delivery ● Categorizes refactorings ● Core concepts: ○ Versioning of migrations ○ Checksumming ○ Tracking migrations (dedicated config table) 15 Michael Simons at Java Summit 2023

Slide 16

Slide 16 text

© 2023 Neo4j, Inc. All rights reserved. Why evolutionary database design? ● “Change management” is often more a euphemism for change prevention ● “The big schema design” seldom (never?) works out unchanged ➜ Avoid or minimize waste from upfront design and significant rework efforts 16 Michael Simons at Java Summit 2023

Slide 17

Slide 17 text

© 2023 Neo4j, Inc. All rights reserved. 17 If at some point roughly 20 years ago, DBAs sitting on top of the databases wouldn't have been so stuck in the 1990s and earlier processes, #NoSQL wouldn't have been such a big movement at all. Michaels Hot-Take #2 17 Michael Simons at Java Summit 2023

Slide 18

Slide 18 text

© 2023 Neo4j, Inc. All rights reserved. To be fair: Database refactorings are hard ● Must maintain behavioral semantics ● Must maintain informational semantics ● Touches usually a lot more than just a schema ○ Most often one or more applications ○ Some reports in a team far, far away 18 Michael Simons at Java Summit 2023

Slide 19

Slide 19 text

© 2023 Neo4j, Inc. All rights reserved. What is applicable to Neo4j? (Refactorings) ● Relevant categories ○ (Some) data quality refactorings ○ (Some) Referential integrity refactorings ○ All possible transformations All of them can be expressed with Cypher! 19 Michael Simons at Java Summit 2023

Slide 20

Slide 20 text

© 2023 Neo4j, Inc. All rights reserved. What is applicable to Neo4j? (Core concepts) ● Versioning of migrations (refactorings / changesets) ○ Not applying them multiple times ● Put them into actual version control ● Checksumming ○ Making sure changesets are not changed after the fact ● Make them validatable ● Track them in a schema / configuration table 20 Michael Simons at Java Summit 2023

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

© 2023 Neo4j, Inc. All rights reserved. For what did we need a database refactoring tool? 1. Spring Data Neo4j (SDN) and JHipster collaboration back in late 2019 (Needed a simple way to create a couple of Nodes and Relationships, basic inventory data) 2. Deriving index and constraint definitions from Neo4j-OGM and applying them to at least 4 different versions of Neo4j 23 Michael Simons at Java Summit 2023

Slide 24

Slide 24 text

© 2023 Neo4j, Inc. All rights reserved. Existing Java based tooling 24 ● Flyway (Relational) ● Liquibase (Relational + Plugins) ● Liquigraph (Graph) ○ Now obsolete, became Liquibase-Neo4j-Plugin Michael Simons at Java Summit 2023 Conceptual main difference: ● Free form scripts vs ● Built-in set of refactorings

Slide 25

Slide 25 text

© 2023 Neo4j, Inc. All rights reserved. 25 We do prefer the free form approach, it does align much better with what is necessary and possible within current Neo4j. 25 Michael Simons at Java Summit 2023

Slide 26

Slide 26 text

© 2023 Neo4j, Inc. All rights reserved. Our requirements ● Dependency free or as few as possible ○ Target frameworks often have many things in place, don’t want to bring them more dependency hell ○ Connection layer of course necessary and ok ● Not wrapping Neo4j, the bolt model, its sessions and concepts in JDBC ● As little abstractions as possible ● As little overhead as possible ● Liquigraph / Liquibase-Neo4j felt too alien for Neo4j => Let’s create our own thing! https://github.com/michael-simons/neo4j-migrations 26 Michael Simons at Java Summit 2023

Slide 27

Slide 27 text

© 2023 Neo4j, Inc. All rights reserved. Why Java?! 27 ● I do know Java inside / out ○ Less needed experiments ○ Well known build systems ○ Boring, in a good way ○ Modern and versatile, even more so with Java 17 and higher ● The driver / connector support for Java is IMHO the most mature Neo4j driver ● Fantastic CLI support via PicoCLI ● Fantastic native support via GraalVM Michael Simons at Java Summit 2023

Slide 28

Slide 28 text

© 2023 Neo4j, Inc. All rights reserved. History and feature overview ● January 2020: ○ 0.0.1 preview release for JHipster ○ 0.0.8 Spring Boot starter (needed in JHipster) ● February 2021: First external contributor ● July 2021: Make validation optional, transactional functions everywhere, full AuraDB support ● October / November 2021: Collaboration with JReleaser / Andres Almiray ● November 2021: 1.0.0 release published with JReleaser ● December 2021: Official Neo4j-Labs project Neo4j-Labs Project ● February 2022: Full support for Quarkus, including native mode 28

Slide 29

Slide 29 text

© 2023 Neo4j, Inc. All rights reserved. History and feature overview 29 ● June 2022: Catalogs ○ Primary for indexes and constraints, more to come ● August 2022: Built-In database refactorings ○ Renaming labels / types ○ Normalizing properties ○ Merging nodes ● September 2022: Annotation processor ● November 2022: Repeatable migrations ● Future: ○ Undo for catalog items ○ Undo with manual scripts Michael Simons at Java Summit 2023

Slide 30

Slide 30 text

© 2023 Neo4j, Inc. All rights reserved. Distribution https://github.com/michael-simons/neo4j-migrations ● Core API ● Spring Boot Starter ● Quarkus Extension ● Maven Plugin ● CLI (native binaries for Linux, macOS, Windows, noarch (JVM) for everything) ○ No JVM or Jars required, just one native binary! Read more: Modules 30 Michael Simons at Java Summit 2023

Slide 31

Slide 31 text

© 2023 Neo4j, Inc. All rights reserved. Installation (CLI only) ● Via SDKMan! since 1.5.1 sdk install neo4jmigrations ● Via Homebrew for macOS ● brew install michael-simons/homebrew-neo4j-migrations/neo4j-migrations Read more: Installation; Core-API, Quarkus Extension, Spring Boot Starter and Maven-Plugin are on Maven-Central! 31 Michael Simons at Java Summit 2023

Slide 32

Slide 32 text

© 2023 Neo4j, Inc. All rights reserved. How to run this? ● Inside applications ○ Only inventory data, please ○ Moving data for tests (resources will be aggregated) ○ Modern applications are usually running more than one instance: Which one should trigger migrations? ● CI/CD (No JVM necessary, self-contained, native binary) Safe passwords (files or environment) ● Sidecar ● Init container 32 Michael Simons at Java Summit 2023

Slide 33

Slide 33 text

Slide 34

Slide 34 text

© 2023 Neo4j, Inc. All rights reserved. © 2023 Neo4j, Inc. All rights reserved. 35 Email [email protected] Social media @rotnroll666(@mastodon.social) Profile https://github.com/michael-simons Neo4j-Migrations https://neo4j.com/labs/neo4j-migrations Demos https://github.com/michael-simons/javasummit2023 (Quarkus) https://github.com/michael-simons/nodes2022 (Spring) Tools used Java 17, Quarkus 3 (REST service) Neo4j via Docker httpie for sending requests https://httpie.io jq for parsing JSON https://stedolan.github.io/jq/ Thank you!