Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond Go migrate: Why We Built Our Own - Migra...

Beyond Go migrate: Why We Built Our Own - Migrations that never roll back

Why build your own database migration tool when so many already exist? In this talk, I’ll share why we moved away from Go migrate, how Google’s approach to database migrations inspired us, and why we chose to build our own solution rather than adopt an existing tool from the Go or Java ecosystem. We’ll explore the requirements and trade-offs that shaped our approach, giving Go developers a practical perspective on the build-versus-adopt decision.

Ladislav Prskavec

November 19, 2026

Transcript

15 slides
  1. GO MEETUP #26 · 19 NOVEMBER 2026 Migrations that never

    roll back Replacing golang-migrate so our Go services can roll back safely Ladislav Prskavec · Everpure krypton-db-management
  2. THE PROBLEM Our services could not be rolled back The

    migrator blocks it The code was never tested for it golang-migrate fails against a database already migrated to n+1. Rolling the service back meant rolling the schema back too. Nothing guaranteed the previous Go binary works against the n+1 schema. Backward compatibility was a convention, not a check. On top of that, several security findings against the migration stack were blocked with no upstream fix. Migrations that never roll back 02
  3. HOW GOLANG-MIGRATE WORKS One version, one dirty flag, three commits

    per migration COMMIT 1 COMMIT 2 COMMIT 3 SetVersion(N, dirty=true) run the migration SQL SetVersion(N, dirty=false) A crash or a failing statement between commit 1 and commit 3 leaves N | dirty=true behind. Every later run refuses until a human inspects the schema and runs force. Migrations that never roll back 03
  4. PITFALL 1 · DIRTY STATE The dirty flag is not

    hypothetical db/> select * from migrations; version | dirty ---------+------1 | t Migrations that never roll back A real table from one of our integration databases: a migration stream whose first migration failed and was abandoned there. Recovery is manual: work out which statements ran, fix the schema by hand, force the version that matches reality, then fix the file. 04
  5. PITFALL 2 · SILENT SKIPS A merged migration that never

    runs Two branches Alice adds 010. Bob adds 020. Bob merges first Production moves to 20. Alice merges 010 is below 20, so it is skipped. up still reports success. Only the latest version is stored, so there is no history to notice the gap. Timestamp filenames make collisions rarer; they do not prevent skips. A CI check against main’s highest version does. Migrations that never roll back 05
  6. HOW GOOGLE DOES IT · F1 Break a risky change

    into safe intermediate states absent delete-only write-only backfill public At most one version behind Slow on purpose Servers switch schema at different times. Correctness holds as long as no server is more than one version behind. Each step waits out a schema lease, so even a simple change takes 10–20 minutes. The team built a formal model because a bad change could lose data. Rae et al., “Online, Asynchronous Schema Change in F1”, VLDB 2013 06
  7. HOW GOOGLE DOES IT · SPANNER, DORA, SRE Schema changes

    are online, decoupled and canaried Background validation Expand, then contract Rolled out like code Spanner applies DDL as a long-running operation. Validation and backfill run below production traffic; a failed check fails the update. DORA’s guidance: never mutate existing objects; add new ones beside them, so app deploys and schema changes stay independent. A schema change once slowed Google Classroom for 1 h 53 min and was rolled back. The follow-up: incremental rollout. Sources: Spanner schema-update docs · dora.dev database change management · Google Workspace status dashboard 07
  8. OUR DESIGN · THE RULE Migrations ship first and never

    roll back, so the service always can. FAST TRAIN SLOW TRAIN component-migrations component Its own CI/CD component and image. Reaches production first, several times per sprint. Written against the schema already in production, so it can roll back to anything production runs.
  9. FAST TRAIN AND THE MERGE GATE Schema first, then the

    code that needs it FAST TRAIN · db-migrations, several times a sprint Migration PR Image 052_….up.sql tag 52-… INTEG QA PREPROD PROD schema = 52 check-compat reads the deployed prod version SLOW TRAIN · the service Service PR Manifest check-compat uses column min = 52 prod ≥ 52? Merge Rollout Start-up gate per site Merge blocked while prod is 51 Migrations that never roll back 09
  10. ENGINE · JACKC/TERN The migration and its version bump commit

    together begin; /- the migration SQL alter table resources add column owner text; reset all; update schema_version set version = 52; commit; Migrations that never roll back A failure rolls everything back. The version is unchanged and a rerun is clean. There is no dirty column and nothing to force. pgx-native. Our services already use pgx, so database/sql and lib/pq leave the dependency tree. Advisory lock built in. Concurrent runners cannot interleave. 10
  11. A DEPLOYMENT, NOT AN INIT CONTAINER Migrate on start, then

    stay up and report krypton_db_schema_version The applied version as a metric. The pipeline gates on it. GET /v1/schema “What schema is preprod on?” without a psql session. GET /healthz 503 while migrating or failed. A failure blocks the queue; nothing auto-rolls back. // in the service’s readiness check err /= kdbm.RequireMinVersion( ctx, pool, version.MinSchemaVersion) if err /= nil { // ErrSchemaTooOld{Required, Applied} return notReady(err) } Fail readiness, not the process: the rollout stalls with old pods serving, then heals when the migration lands. Migrations that never roll back 13
  12. STRICT BY DESIGN The loader refuses instead of guessing Rule

    Why No down migrations. A leftover .down.sql fails the build. A down verb that exists will eventually run against something that matters. No BEGIN or COMMIT in new files. The tool owns the transaction. Stripping them silently would hide the rule. -- krypton-db-management:disable-tx The one explicit exit, for CREATE INDEX CONCURRENTLY and similar. legacy_gomigrate_through: N Files from the go-migrate era run as written. No rewrite of 136 files. Append-only set, new prefixes above the high-water mark. Fixes the silent-skip problem from slide 5. Migrations that never roll back 14
  13. CUTTING OVER SAFELY Remove the old runner instead of feeding

    it 1 · BRIDGE A service release that keeps the go-migrate code but never runs it. 2 · HOLD 3 · ADVANCE --hold-at until every rollback target is past the bridge. Only now do new migrations apply at that site. Our first idea was to mirror the version back into the go-migrate table. Review killed it: a rolled-back binary still lacks the files above its embedded set, and the mirror write races a runner that takes no lock. Migrations that never roll back 15
  14. TAKEAWAYS Six things to apply on Monday 01 Commit each

    migration and its version bump in one transaction. A BEGIN/END lint rule only gets you halfway. 02 Ship the schema first; write code against what production already runs. 03 Expand, then contract. Roll forward; never roll the schema back. 04 Gate readiness on a minimum schema version, never a maximum. 05 Make the applied version a metric, not a log line. 06 Budget for integration, not just design. Each component's CI and test flows cost us more than building the service. Migrations that never roll back 17
  15. THANK YOU Questions? Further reading: Rae et al., “Online, Asynchronous

    Schema Change in F1”, VLDB 2013 · Corbett et al., “Spanner: Google’s Globally-Distributed Database”, OSDI 2012 · Cloud Spanner docs, “Make schema updates” · dora.dev, “Database change management” · github.com/jackc/tern · github.com/golang-migrate/migrate

More Decks by Ladislav Prskavec

Other Decks in Technology