Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Neo4j-Migrations: The lean way of applying database refactorings to Neo4j

Neo4j-Migrations: The lean way of applying database refactorings to Neo4j

Neo4j-Migrations gives you an easy way to apply schema changes to Neo4j. Unlike Liquibase, It is almost dependency free and runs right on the Neo4j Java Driver, with no need to work with JDBC. Neo4j-Migrations integrates Spring Data Neo4j with JHipster and has been in production since summer 2020.

Neo4j-Migrations:

* Runs Cypher scripts
* Runs Java-based migrations
* Records all migrations in a simple graph
* Works with multiple databases
* Has extensions for Spring Boot (included in Neo4j Ops Manager) and Quarkus

The most important feature? Neo4j-Migrations comes as a native binary CLI, perfect for use in CI/CD systems.The CLI has everything you need to set up safe CI scenarios. Come to this session to learn how you can put Neo4j-Migrations to work for you.

Michael Simons

November 17, 2022
Tweet

More Decks by Michael Simons

Other Decks in Programming

Transcript

  1. 1
    Neo4j-Migrations:
    The lean way of refactoring Neo4j content
    By Michael Simons, Java Champion & Staff Software Engineer @ Neo4j

    View Slide

  2. 2
    It takes a village to raise a child

    View Slide

  3. What this is not about
    ● Migrating from one Neo4j version to another
    ● Neo4j-OGM
    ● Spring Data Neo4j
    (But I might speak about those things^^)
    3

    View Slide

  4. 4
    What is this all about?
    You can’t have not a schema.

    View Slide

  5. What can be part of a Neo4j-Schema
    ● Constraints
    ● Indexes
    ● A consistent set of labels & types
    ○ Consistent properties within them
    ○ Consistent values for properties with the same name
    5

    View Slide

  6. 6
    The only constant is change
    Requirements, data itself, processes etc…

    View Slide

  7. History
    Refactoring Databases published in 2006.
    • Versioning of migrations
    ◦ Repeating
    ◦ Maybe undoing
    • Improve the database design
    • Unblock continuous delivery
    • Very focussed on relational databases
    • “schema on write”
    7

    View Slide

  8. Applicable concepts
    ● Versioning of migrations (refactorings / changesets)
    ○ Not applying them multiple times
    ● Put them into actual version control
    ● Checksumming
    ○ Making sure changesets are not changed after the fact
    ● Make them validatable

    View Slide

  9. For what did we need a database refactoring tool?
    ● SDN6 and JHipster collaboration back in late 2019
    ○ Need for a simple way to create a couple of Nodes and Relationships
    ● We don’t want people to use Neo4j-OGM auto-index manager
    (automatically deriving indexes from the OGM schema and trying to get
    this right for more than 5 supported database versions)
    9

    View Slide

  10. Existing Java based tooling
    10
    ● Liquibase (Relational)
    ● Flyway (Relational)
    ● Liquigraph (Graph)
    ○ Now obsolete, became Liquibase-Neo4j-Plugin

    View Slide

  11. Our requirements
    ● Dependency free or as few as possible
    ○ Target frameworks often have many things in place, don’t wanna bring them
    more dependency hell
    ○ Connection layer of course necessary and ok
    ● Not wrapping Neo4j, the bolt model, its sessions and concepts in JDBC
    ● As little abstractions as possible
    ● As little overhead as possible
    ● Liquigraph / Liquibase-Neo4j felt too alien for Neo4j
    => Let’s create our own thing!
    https://github.com/michael-simons/neo4j-migrations
    11

    View Slide

  12. Very much Flyway inspired 󰠗
    12

    View Slide

  13. 13
    For what do you might need a database refactoring
    tool?
    https://github.com/meistermeier/migrate-to-neo4j-50

    View Slide

  14. History and feature overview
    ● January 2020:
    ○ 0.0.1 preview release for JHipster
    ○ 0.0.4 full support for 4.0 multi database
    ○ 0.0.8 Spring Boot starter (needed in JHipster)
    ● February 2021: First external contributor
    ● July 2021: Make validation optional, transactional functions everywhere, full
    AuraDB support
    ● October / November 2021: Collaboration with JReleaser / Andres Almiray
    ● November 2021: 1.0.0 release published with JReleaser
    ● December 2021: Official Neo4j-Labs project Neo4j-Labs Project:
    https://neo4j.com/labs/neo4j-migrations/
    ● February 2022: Full support for Quarkus, including native mode
    ● March 2022:
    ○ SDKMan! Release
    ○ Preconditions
    14

    View Slide

  15. History and feature overview
    15
    ● June 2022: Catalogs
    ○ Primary for indexes and constraints, more to come
    ● August 2022: Built-In database refactorings
    ○ Renaming labels / types
    ○ Normalizing properties
    ○ Merging nodes
    ● September 2022: Annotation processor
    ● November 2022: Repeatable migrations
    ● Future:
    ○ Undo for catalog items
    ○ Undo with manual scripts

    View Slide

  16. Distribution
    https://github.com/michael-simons/neo4j-migrations
    ● Core API
    ● CLI (native binaries for Linux, macOS, Windows, noarch (JVM) for
    everything)
    ○ No plugins, no driver, just one binary!
    ● Spring Boot Starter
    ● Quarkus Extension
    ● Maven Plugin
    Read more: Modules

    View Slide

  17. Installation (CLI only)
    ● Via SDKMan! since 1.5.1
    sdk install neo4jmigrations
    ● Via Homebrew for macOS
    ● brew install
    michael-simons/homebrew-neo4j-migrations/neo4j-migrations
    Read more: Installation;
    Core-API, Quarkus Extension, Spring Boot Starter
    and Maven-Plugin are on Maven-Central!

    View Slide

  18. How to run this?
    ● Inside applications
    ○ Only inventory data, please
    ○ Moving data for tests (resources will be aggregated)
    ○ Modern applications are usually running more than one instance:
    Which one should trigger migrations?
    ● CI/CD (No JVM necessary, self-contained, native binary)
    Safe passwords (files or environment)

    View Slide

  19. Our schema

    View Slide

  20. Commands:
    clean Removes Neo4j-Migration specific data from the selected
    schema database
    help Display help information about the specified command.
    info Retrieves all applied and pending information, prints them
    and exits.
    init Creates a migration project inside the current folder.
    migrate, apply Retrieves all pending migrations, verify and applies them.
    run Resolves the specified migrations and applies them. Does not
    record any metadata.
    show-catalog Gets the local or remote catalog and prints it to standard
    out in the given format.
    validate Resolves all local migrations and validates the state of the
    configured database with them.
    Features / commands
    Feature parity for all distributions!
    20

    View Slide

  21. 21

    View Slide

  22. 22
    Email [email protected]
    Twitter @rotnroll666
    Profile https://github.com/michael-simons
    Neo4j-Migrations https://neo4j.com/labs/neo4j-migrations
    Demo https://github.com/michael-simons/nodes2022
    Thank you!

    View Slide