Hasan's Journal

Stories, lessons, and scars from production.

Mehedi Hasan
Back to blog

Database Migrations Without Downtime

Schema changes don't have to mean maintenance windows. Here's the expand-and-contract pattern that lets you migrate without going offline.

#Database#Migrations#DevOps

The Problem

Database schema migrations are one of the few remaining operations that traditionally require downtime, and the reason is that most migration strategies involve a moment where the old code and the new schema are incompatible. If you change the schema first and then deploy the new code, the old code (still running during the deploy) fails against the new schema. If you deploy the new code first and then change the schema, the new code fails against the old schema. Either way, there's a window where something is broken, and the traditional solution is to take the application offline during that window — which is feasible for applications with low traffic at night but not for applications that need to be available 24/7.

The solution is the expand-and-contract pattern, which breaks the migration into three phases that each maintain compatibility. In the expand phase, you change the schema to be compatible with both the old and new code — typically by adding new columns or tables alongside the old ones, without removing anything. In the migrate phase, you deploy the new code, which writes to both the old and new schema (dual-writing) and reads from the new schema if it's populated, falling back to the old. In the contract phase, after the new code is fully deployed and the new schema is fully populated, you remove the old schema. Each phase is independently deployable and independently reversible, which means you can stop at any point without breaking anything.

The Expand Phase

The expand phase adds the new schema without removing the old, and the key is that the new schema must be additive — it adds columns or tables but doesn't change or remove existing ones. If you're renaming a column, the expand phase adds the new column alongside the old. If you're changing a column type, the expand phase adds a new column with the new type. If you're splitting a table, the expand phase adds the new table. The old code continues to work because the old schema is unchanged, and the new code (not yet deployed) is designed to work with both. The expand phase is the safest of the three, because it's purely additive — if something goes wrong, you can simply drop the new columns and be back where you started.

The subtlety in the expand phase is that adding columns to a large table can be slow, because most databases rewrite the entire table to add a column even when the column allows nulls. On a table with millions of rows, this can take hours, during which the table is locked and the application is effectively down. The fix is to use a database that supports fast column additions (PostgreSQL has supported non-locking column additions since version 11) or to add the column in a way that doesn't require a table rewrite (using a default that the database can apply lazily). Knowing your database's migration characteristics is essential for zero-downtime migrations, because a migration that's fast on one database may be catastrophically slow on another.

The Migrate and Contract Phases

The migrate phase is where the new code is deployed, and the key is dual-writing: every write to the old schema is also written to the new schema, so the new schema stays populated as the new code runs. Reads can come from either schema, but the safest approach is to read from the old schema during the migrate phase (since it's guaranteed to be complete) and switch to the new schema only in the contract phase. The migrate phase is the most complex, because the dual-writing logic has to be perfect — any write that goes to the old schema but not the new creates an inconsistency that surfaces later as a missing or stale value in the new schema. Backfilling the new schema for existing rows (rows that were written before the migrate phase started) is also part of the migrate phase, and it's typically done as a background job that iterates through the table and populates the new columns.

The contract phase removes the old schema, and it's the phase that makes the migration irreversible. Once you drop the old columns, you can't go back to the old code, because the old code depends on columns that no longer exist. That irreversibility is why the contract phase should be done last, after the new code has been running in production long enough to give you confidence that it works. The contract phase is also the simplest, because by the time you reach it, the new schema is fully populated, the new code is fully deployed, and dropping the old schema is a metadata operation that doesn't require rewriting any data. The entire expand-and-contract pattern is about deferring the irreversible step to the end, after everything else has been verified, so that you can abort at any point before then without consequence.