Share
Embed
Beyond Go migrate: Why We Built Our Own - Migrations that never roll back
Why build your own database migration tool when so many already exist? In this talk, I’ll share why we moved away from Go migrate, how Google’s approach to database migrations inspired us, and why we chose to build our own solution rather than adopt an existing tool from the Go or Java ecosystem. We’ll explore the requirements and trade-offs that shaped our approach, giving Go developers a practical perspective on the build-versus-adopt decision.
Ladislav Prskavec
November 19, 2026
Transcript
15 slides
-
GO MEETUP #26 · 19 NOVEMBER 2026 Migrations that never
roll back Replacing golang-migrate so our Go services can roll back safely Ladislav Prskavec · Everpure krypton-db-management -
THE PROBLEM Our services could not be rolled back The
migrator blocks it The code was never tested for it golang-migrate fails against a database already migrated to n+1. Rolling the service back meant rolling the schema back too. Nothing guaranteed the previous Go binary works against the n+1 schema. Backward compatibility was a convention, not a check. On top of that, several security findings against the migration stack were blocked with no upstream fix. Migrations that never roll back 02 -
HOW GOLANG-MIGRATE WORKS One version, one dirty flag, three commits
per migration COMMIT 1 COMMIT 2 COMMIT 3 SetVersion(N, dirty=true) run the migration SQL SetVersion(N, dirty=false) A crash or a failing statement between commit 1 and commit 3 leaves N | dirty=true behind. Every later run refuses until a human inspects the schema and runs force. Migrations that never roll back 03 -
PITFALL 1 · DIRTY STATE The dirty flag is not
hypothetical db/> select * from migrations; version | dirty ---------+------1 | t Migrations that never roll back A real table from one of our integration databases: a migration stream whose first migration failed and was abandoned there. Recovery is manual: work out which statements ran, fix the schema by hand, force the version that matches reality, then fix the file. 04 -
PITFALL 2 · SILENT SKIPS A merged migration that never
runs Two branches Alice adds 010. Bob adds 020. Bob merges first Production moves to 20. Alice merges 010 is below 20, so it is skipped. up still reports success. Only the latest version is stored, so there is no history to notice the gap. Timestamp filenames make collisions rarer; they do not prevent skips. A CI check against main’s highest version does. Migrations that never roll back 05 -
HOW GOOGLE DOES IT · F1 Break a risky change
into safe intermediate states absent delete-only write-only backfill public At most one version behind Slow on purpose Servers switch schema at different times. Correctness holds as long as no server is more than one version behind. Each step waits out a schema lease, so even a simple change takes 10–20 minutes. The team built a formal model because a bad change could lose data. Rae et al., “Online, Asynchronous Schema Change in F1”, VLDB 2013 06 -
HOW GOOGLE DOES IT · SPANNER, DORA, SRE Schema changes
are online, decoupled and canaried Background validation Expand, then contract Rolled out like code Spanner applies DDL as a long-running operation. Validation and backfill run below production traffic; a failed check fails the update. DORA’s guidance: never mutate existing objects; add new ones beside them, so app deploys and schema changes stay independent. A schema change once slowed Google Classroom for 1 h 53 min and was rolled back. The follow-up: incremental rollout. Sources: Spanner schema-update docs · dora.dev database change management · Google Workspace status dashboard 07 -
OUR DESIGN · THE RULE Migrations ship first and never
roll back, so the service always can. FAST TRAIN SLOW TRAIN component-migrations component Its own CI/CD component and image. Reaches production first, several times per sprint. Written against the schema already in production, so it can roll back to anything production runs. -
FAST TRAIN AND THE MERGE GATE Schema first, then the
code that needs it FAST TRAIN · db-migrations, several times a sprint Migration PR Image 052_….up.sql tag 52-… INTEG QA PREPROD PROD schema = 52 check-compat reads the deployed prod version SLOW TRAIN · the service Service PR Manifest check-compat uses column min = 52 prod ≥ 52? Merge Rollout Start-up gate per site Merge blocked while prod is 51 Migrations that never roll back 09 -
ENGINE · JACKC/TERN The migration and its version bump commit
together begin; /- the migration SQL alter table resources add column owner text; reset all; update schema_version set version = 52; commit; Migrations that never roll back A failure rolls everything back. The version is unchanged and a rerun is clean. There is no dirty column and nothing to force. pgx-native. Our services already use pgx, so database/sql and lib/pq leave the dependency tree. Advisory lock built in. Concurrent runners cannot interleave. 10 -
A DEPLOYMENT, NOT AN INIT CONTAINER Migrate on start, then
stay up and report krypton_db_schema_version The applied version as a metric. The pipeline gates on it. GET /v1/schema “What schema is preprod on?” without a psql session. GET /healthz 503 while migrating or failed. A failure blocks the queue; nothing auto-rolls back. // in the service’s readiness check err /= kdbm.RequireMinVersion( ctx, pool, version.MinSchemaVersion) if err /= nil { // ErrSchemaTooOld{Required, Applied} return notReady(err) } Fail readiness, not the process: the rollout stalls with old pods serving, then heals when the migration lands. Migrations that never roll back 13 -
STRICT BY DESIGN The loader refuses instead of guessing Rule
Why No down migrations. A leftover .down.sql fails the build. A down verb that exists will eventually run against something that matters. No BEGIN or COMMIT in new files. The tool owns the transaction. Stripping them silently would hide the rule. -- krypton-db-management:disable-tx The one explicit exit, for CREATE INDEX CONCURRENTLY and similar. legacy_gomigrate_through: N Files from the go-migrate era run as written. No rewrite of 136 files. Append-only set, new prefixes above the high-water mark. Fixes the silent-skip problem from slide 5. Migrations that never roll back 14 -
CUTTING OVER SAFELY Remove the old runner instead of feeding
it 1 · BRIDGE A service release that keeps the go-migrate code but never runs it. 2 · HOLD 3 · ADVANCE --hold-at until every rollback target is past the bridge. Only now do new migrations apply at that site. Our first idea was to mirror the version back into the go-migrate table. Review killed it: a rolled-back binary still lacks the files above its embedded set, and the mirror write races a runner that takes no lock. Migrations that never roll back 15 -
TAKEAWAYS Six things to apply on Monday 01 Commit each
migration and its version bump in one transaction. A BEGIN/END lint rule only gets you halfway. 02 Ship the schema first; write code against what production already runs. 03 Expand, then contract. Roll forward; never roll the schema back. 04 Gate readiness on a minimum schema version, never a maximum. 05 Make the applied version a metric, not a log line. 06 Budget for integration, not just design. Each component's CI and test flows cost us more than building the service. Migrations that never roll back 17 -
THANK YOU Questions? Further reading: Rae et al., “Online, Asynchronous
Schema Change in F1”, VLDB 2013 · Corbett et al., “Spanner: Google’s Globally-Distributed Database”, OSDI 2012 · Cloud Spanner docs, “Make schema updates” · dora.dev, “Database change management” · github.com/jackc/tern · github.com/golang-migrate/migrate