Neo4j-Migrations: The lean way of applying database refactorings to Neo4j

Neo4j-Migrations gives you an easy way to apply schema changes to Neo4j. Unlike Liquibase, It is almost dependency free and runs right on the Neo4j Java Driver, with no need to work with JDBC. Neo4j-Migrations integrates Spring Data Neo4j with JHipster and has been in production since summer 2020.

Neo4j-Migrations:

* Runs Cypher scripts
* Runs Java-based migrations
* Records all migrations in a simple graph
* Works with multiple databases
* Has extensions for Spring Boot (included in Neo4j Ops Manager) and Quarkus

The most important feature? Neo4j-Migrations comes as a native binary CLI, perfect for use in CI/CD systems.The CLI has everything you need to set up safe CI scenarios. Come to this session to learn how you can put Neo4j-Migrations to work for you.

Transcript

1. 1
   Neo4j-Migrations:
   The lean way of refactoring Neo4j content
   By Michael Simons, Java Champion & Staff Software Engineer @ Neo4j
   

   View Slide

2. 2
   It takes a village to raise a child
   

   View Slide

3. What this is not about
   ● Migrating from one Neo4j version to another
   ● Neo4j-OGM
   ● Spring Data Neo4j
   (But I might speak about those things^^)
   3
   

   View Slide

4. 4
   What is this all about?
   You can’t have not a schema.
   

   View Slide

5. What can be part of a Neo4j-Schema
   ● Constraints
   ● Indexes
   ● A consistent set of labels & types
   ○ Consistent properties within them
   ○ Consistent values for properties with the same name
   5
   

   View Slide

6. 6
   The only constant is change
   Requirements, data itself, processes etc…
   

   View Slide

7. History
   Refactoring Databases published in 2006.
   • Versioning of migrations
   ◦ Repeating
   ◦ Maybe undoing
   • Improve the database design
   • Unblock continuous delivery
   • Very focussed on relational databases
   • “schema on write”
   7
   

   View Slide

8. Applicable concepts
   ● Versioning of migrations (refactorings / changesets)
   ○ Not applying them multiple times
   ● Put them into actual version control
   ● Checksumming
   ○ Making sure changesets are not changed after the fact
   ● Make them validatable
   

   View Slide

9. For what did we need a database refactoring tool?
   ● SDN6 and JHipster collaboration back in late 2019
   ○ Need for a simple way to create a couple of Nodes and Relationships
   ● We don’t want people to use Neo4j-OGM auto-index manager
   (automatically deriving indexes from the OGM schema and trying to get
   this right for more than 5 supported database versions)
   9
   

   View Slide

10. Existing Java based tooling
    10
    ● Liquibase (Relational)
    ● Flyway (Relational)
    ● Liquigraph (Graph)
    ○ Now obsolete, became Liquibase-Neo4j-Plugin
    

    View Slide

11. Our requirements
    ● Dependency free or as few as possible
    ○ Target frameworks often have many things in place, don’t wanna bring them
    more dependency hell
    ○ Connection layer of course necessary and ok
    ● Not wrapping Neo4j, the bolt model, its sessions and concepts in JDBC
    ● As little abstractions as possible
    ● As little overhead as possible
    ● Liquigraph / Liquibase-Neo4j felt too alien for Neo4j
    => Let’s create our own thing!
    https://github.com/michael-simons/neo4j-migrations
    11
    

    View Slide

12. Very much Flyway inspired 󰠗
    12
    

    View Slide

13. 13
    For what do you might need a database refactoring
    tool?
    https://github.com/meistermeier/migrate-to-neo4j-50
    

    View Slide

14. History and feature overview
    ● January 2020:
    ○ 0.0.1 preview release for JHipster
    ○ 0.0.4 full support for 4.0 multi database
    ○ 0.0.8 Spring Boot starter (needed in JHipster)
    ● February 2021: First external contributor
    ● July 2021: Make validation optional, transactional functions everywhere, full
    AuraDB support
    ● October / November 2021: Collaboration with JReleaser / Andres Almiray
    ● November 2021: 1.0.0 release published with JReleaser
    ● December 2021: Official Neo4j-Labs project Neo4j-Labs Project:
    https://neo4j.com/labs/neo4j-migrations/
    ● February 2022: Full support for Quarkus, including native mode
    ● March 2022:
    ○ SDKMan! Release
    ○ Preconditions
    14
    

    View Slide

15. History and feature overview
    15
    ● June 2022: Catalogs
    ○ Primary for indexes and constraints, more to come
    ● August 2022: Built-In database refactorings
    ○ Renaming labels / types
    ○ Normalizing properties
    ○ Merging nodes
    ● September 2022: Annotation processor
    ● November 2022: Repeatable migrations
    ● Future:
    ○ Undo for catalog items
    ○ Undo with manual scripts
    

    View Slide

16. Distribution
    https://github.com/michael-simons/neo4j-migrations
    ● Core API
    ● CLI (native binaries for Linux, macOS, Windows, noarch (JVM) for
    everything)
    ○ No plugins, no driver, just one binary!
    ● Spring Boot Starter
    ● Quarkus Extension
    ● Maven Plugin
    Read more: Modules
    

    View Slide

17. Installation (CLI only)
    ● Via SDKMan! since 1.5.1
    sdk install neo4jmigrations
    ● Via Homebrew for macOS
    ● brew install
    michael-simons/homebrew-neo4j-migrations/neo4j-migrations
    Read more: Installation;
    Core-API, Quarkus Extension, Spring Boot Starter
    and Maven-Plugin are on Maven-Central!
    

    View Slide

18. How to run this?
    ● Inside applications
    ○ Only inventory data, please
    ○ Moving data for tests (resources will be aggregated)
    ○ Modern applications are usually running more than one instance:
    Which one should trigger migrations?
    ● CI/CD (No JVM necessary, self-contained, native binary)
    Safe passwords (files or environment)
    

    View Slide

19. Our schema
    

    View Slide

20. Commands:
    clean Removes Neo4j-Migration specific data from the selected
    schema database
    help Display help information about the specified command.
    info Retrieves all applied and pending information, prints them
    and exits.
    init Creates a migration project inside the current folder.
    migrate, apply Retrieves all pending migrations, verify and applies them.
    run Resolves the specified migrations and applies them. Does not
    record any metadata.
    show-catalog Gets the local or remote catalog and prints it to standard
    out in the given format.
    validate Resolves all local migrations and validates the state of the
    configured database with them.
    Features / commands
    Feature parity for all distributions!
    20
    

    View Slide

21. View Slide

22. 22
    Email michael.simons@neo4j.com
    Twitter @rotnroll666
    Profile https://github.com/michael-simons
    Neo4j-Migrations https://neo4j.com/labs/neo4j-migrations
    Demo https://github.com/michael-simons/nodes2022
    Thank you!
    

    View Slide