What the freeze is really for
A migration plan with a freeze in it is saying something specific, usually without meaning to: we do not know how to keep two systems agreeing with each other while both are moving, so we are going to stop one of them. That is an honest admission of a hard problem. It is also a plan that fails in a predictable way, because the business does not stop, the freeze gets extended, the queue of held-back changes grows, and the cutover eventually happens under more pressure than the day it was scheduled, not less.
We have watched the same sequence enough times to write it down. Week one of the freeze is calm. Week three, a customer commitment requires a change to the legacy system, and it gets made, because revenue outranks the migration plan. Week five, someone notices the new system does not have that change, so a second backlog opens: the list of things done to the old system during the freeze that must be redone in the new one. Week eight, the cutover date moves. By week twelve there are two backlogs, two teams that have stopped talking about the same system in the same terms, and a cutover nobody can rehearse because rehearsing it means running it.
The alternative is not heroics. It is a set of mechanics that let both systems keep taking traffic while the boundary between them moves, request by request, entity by entity, cohort by cohort. None of it is exotic. All of it has to be built before the first request is routed, which is the part most plans skip.
Where Migrations Actually Fail — Our Ranking by Frequency
Ordered by how often we find each one as the primary cause when a migration is stalled or reversed. The top two account for most of it.
You are probably here because
- The freeze has already been extended once, and the list of changes waiting behind it keeps growing.
- Both systems are supposed to agree, and nobody can tell you today whether they actually do.
- Nobody can produce a complete list of what is still reading the old database.
- The rollback plan is a sentence in a document that has never been run against real users.
These are one problem wearing four faces — nobody has decided how the two systems stay in agreement while both are still taking writes — and the sections below on the write path and on the reconciliation suite are where this article answers it.
Put the seam where the interface is stable
A migration that runs live needs one place where traffic can be sent to the old system or the new one. That place is the seam, and choosing it well decides how much of the rest is easy. The right seam is an interface that already exists and already has a contract: an HTTP boundary, a service facade, a message topic, a set of stored procedures, a database view. The wrong seam is inside application code, at the ORM layer or below, because that gives you a switch that is entangled with business logic and cannot be flipped per request.
In practice the seam is usually a routing layer in front of the API. It reads a decision from a flag store and forwards the request to the old service or the new one. Three properties matter and they are worth being strict about. The decision must be per request, not per deploy, so a rollback is a flag change and not a release. The decision must be deterministic for a given entity, so the same account does not bounce between systems on consecutive calls and see its own writes disappear. And the decision must be logged with the request, so that when something is wrong three days later you can tell which system produced it without guessing.
Route at the granularity of the work, not the granularity of the codebase. Migrations that move one endpoint at a time make progress that is visible and reversible. Migrations that move one service at a time tend to discover, at the moment of cutover, that the service had eleven callers and two of them were not in the inventory.
Writes are the hard half
Reads are a comparison problem. Writes are a correctness problem, and the entire difficulty of a live migration lives there. The rule that keeps it tractable: at any moment, every entity has exactly one system of record, and everything else is a replica. Not "both systems are authoritative during the transition." That sentence has no implementation.
Dual-write from the application is the approach teams reach for first, and it is the one that produces the ugliest failures. Two writes to two systems are not one transaction. The first succeeds and the second times out, and now the systems disagree with no record of which one is right. Retrying the second write can apply it twice. Concurrent requests can apply the two writes in different orders in the two systems, so both are internally consistent and permanently different. Wrapping both in a distributed transaction is possible in narrow cases and expensive in all of them, and it makes availability the product of the two systems rather than the better of them.
The approach that holds is to write to one system and replicate to the other from a durable log. If the legacy database is the system of record, run change data capture off its write-ahead log or binlog and apply the stream to the new one. Postgres logical replication and Debezium are the common paths; both read the same log the database uses for its own recovery, which means the replica sees every committed change exactly in commit order, including changes made by a batch job or by someone at a psql prompt. That last part matters more than it sounds, because legacy systems always have a second writer nobody mentioned.
When the new system takes over as the system of record for an entity, the direction reverses, and the cleanest mechanism is a transactional outbox: the service writes the business change and an event row in the same local transaction, and a relay reads the outbox and publishes it. The event cannot be lost if the write committed, and it cannot exist if the write rolled back, because they are the same commit. Everything downstream, including the legacy replica, becomes a consumer with an idempotent apply.
| Write strategy | How it behaves | What breaks | Use it when |
|---|---|---|---|
| Application dual-write | Service writes both systems inline | Partial writes, ordering differences under concurrency, no record of which side is right | Almost never. A short bridge for append-only data with no reads on the new side |
| CDC from legacy to new | New system is a replica fed by the old system's log | Schema translation bugs, replication slot growth if a consumer stalls | The whole build phase, and every entity not yet cut over |
| Outbox from new to legacy | New system is authoritative, publishes events in the write transaction | Consumers that are not idempotent, relay lag during bursts | After cutover, while the old system still has readers |
| Entity-scoped ownership | One system owns each entity, decided by the same flag as routing | Entities that reference each other across the boundary | The default. Combine with either replication direction above |
Backfill and the tail, and the moment they meet
The new system needs history, and history has to be loaded while the old system keeps changing. The shape that works is a snapshot plus a tail. Record the log position first. Take the snapshot, or read it in key ranges. Then apply the captured stream from the recorded position forward. If the tail starts where the snapshot began rather than where it ended, the overlap is replayed and no change is lost, which is the point.
This only works if every apply is idempotent and if the merge rule is explicit. The bug we find most often in a half-built migration is a backfill worker overwriting a newer live change with an older snapshot row, because the two paths were written by different people at different times and both used a plain upsert. The fix is a version column that is compared on write. Apply the incoming row only if its source version, log sequence number, or updated timestamp taken from the source database is greater than what is stored. Do not use the wall clock of the machine running the loader. Clock skew of a few hundred milliseconds between two workers is enough to silently reorder writes, and nothing in the pipeline will tell you it happened.
Make the backfill restartable by construction. Chunk by primary key range, record the completed ranges in a table, and let the worker be killed and restarted without losing position or repeating expensive work. A backfill that must run to completion in one pass will fail on the largest table at the worst time, and the recovery plan will be to start over. Chunking also gives you a throttle, which you will need the first time the backfill drives replica lag on a production database into the tens of seconds.
One operational detail that has caused more incidents than it should: a logical replication slot retains write-ahead log on the source until the consumer acknowledges it. Point a slot at a consumer, let the consumer crash on a weekend, and the source database fills its disk. Alert on slot lag from the first day the slot exists, not after the first outage.
Find every consumer of the old database, from the query log
Ask around and you will get the list of applications. Read a week of query logs, connection metadata and grant tables and you will get the real list: the nightly export to a partner, the BI tool connecting as a shared read-only user, a scheduled report someone built four years ago, a spreadsheet on a finance laptop with a live ODBC connection, and an internal tool whose owner left. Every one of them is a thing that breaks at cutover, and every one is cheap to handle if you know about it in month one. Group connections by user, host and application name, then chase down anything you cannot map to a system you already know.
Shadow reads and a diff budget
Before any read traffic is served by the new system, it should have answered the same requests in the dark for weeks. Mirror a percentage of production reads to the new implementation, discard its response, and compare the two. The comparison is where the engineering is. Raw response equality produces thousands of diffs on day one and everyone stops reading the report by day three.
So normalize before comparing. Sort collections that have no defined order. Round floating point to the precision the consumer actually uses. Drop or canonicalize timestamps generated at request time. Then classify each remaining diff into one of three buckets: the new system is wrong, the old system is wrong and we are keeping the behavior anyway, or the difference is intentional and should be documented. That middle bucket is the one that surprises people, and it is the single best argument for shadow reads. Legacy systems accumulate behavior that was never a requirement: a rounding quirk in a total, a filter that silently excludes soft-deleted rows, a sort that is stable only by accident. Some of it is depended on by customers, and the diff report is how you find out before they do.
Store diffs as samples with the full request and both responses, not as a count. A dashboard that says the diff rate is 0.4 percent tells you nothing you can act on. Fifty saved examples tell you there are three underlying causes. Then set a budget and ratchet it: the new system serves live reads for an entity type when the unexplained diff rate over a rolling window is under a number you wrote down in advance, and every diff above it has a ticket.
Cutover Readiness Gates — What Has To Be True Before Routing Real Users
Set the numbers before the pressure arrives. A gate negotiated the week of cutover is not a gate.
Cut over by cohort, not by calendar
A date-based cutover moves everyone at once and learns everything at once. A cohort cutover moves a group whose failure you can absorb, learns what is wrong, fixes it, and moves the next group. Order the cohorts by what they teach relative to what they cost: internal users first, then a low-volume account that has agreed to be early, then a segment with simple usage, then the long tail, then the accounts whose escalation path ends at your CEO.
Give each cohort a soak period long enough to include the slow signals. Most migration defects do not show up in the first hour. They show up on the first month-end close, the first invoice run, the first weekly export, the first customer who tries to edit something they created two years ago. A cohort that has been live for three days has proven that the write path works. It has not proven that the reporting path works.
Keep the reverse replication running for the entire soak. This is what makes rollback real rather than theoretical: while a cohort is on the new system, its changes are flowing back to the legacy database, so flipping the flag back returns those users to a system that is current, not to one that is a week stale. The moment reverse replication stops, rollback stops being a flag and becomes a data recovery project. Know which day that is, name it, and treat it as a decision rather than a side effect.
The failure modes we see most
- A freeze that starts before the seam exists. The freeze buys time to build the thing that would have made the freeze unnecessary, and the clock runs the whole time.
- Dual-write with no reconciliation. The systems diverge from week one, quietly, and the divergence is discovered by a customer.
- Backfill and live changes with last-writer-wins on wall-clock time. Older data overwrites newer data, sometimes, under load, with no error anywhere.
- Comparing systems only on aggregate counts. Two tables with identical row counts can disagree on every row that matters.
- Cutting over the write path and the schema redesign together. When something is wrong you cannot tell which change caused it, and you cannot roll back half.
- Treating the reporting layer as a later problem. Dashboards are consumers with owners and expectations, and they break loudly in front of executives.
- No named end date for the old system. Two systems running forever is the most expensive outcome available, and it is the default one.
Send it over and we will tell you what we would change.
Email your migration plan and its cutover sequence — including the freeze window it assumes and the entities that have to move first — to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comIdentifiers, enums, and the fields nobody owns
Data that has lived in one system for a decade has grown attachments to that system. Three of them cause most of the pain.
Identifiers. If the new system generates its own keys, keep a mapping table from old identifier to new, populate it during backfill, and treat it as production data with backups. Do not plan to throw it away. Old identifiers are printed on invoices, stored in partner systems, embedded in support tickets and pasted into emails, and someone will need to resolve one three years after the legacy database is gone. Keeping the old identifier as a queryable column on the new record is cheaper than every alternative.
Enumerations and status fields. The legacy status column has nine values in the schema, twelve in production, and two that mean the same thing for historical reasons. The new model has five clean states. That mapping is a business decision, not a data-engineering one, and it needs an owner who can say what happens to the rows that fit nowhere. Run the distribution query early and put the actual counts in front of that person.
The fields carrying meaning nobody documented. A nullable column where null means "not applicable" in one code path and "not yet entered" in another. A free-text note field that operations has been using as a workflow state for years. A date column stored without a timezone whose meaning depends on the office that entered it. These are found by profiling the data and by asking the people who use the system daily, and they are never found by reading the schema.
A reconciliation suite worth trusting
Reconciliation is what replaces the freeze. The freeze existed to guarantee the two systems agreed. Continuous reconciliation gives the same guarantee without stopping anything, and it produces evidence you can show to the people who wanted the freeze.
| Check | What it compares | Cadence | On failure |
|---|---|---|---|
| Replication lag | Log position applied against source head | Continuous | Page if above the rollback window; block new cohorts |
| Row-level checksum by key range | Hash of business fields per chunk, both sides | Hourly on hot ranges, nightly full | Re-sync the chunk, then find why it drifted |
| Financial and count totals | Sums and counts by period and account | Daily, plus at period close | Stop cutover progression until explained |
| Shadow read diffs | Live responses from both implementations | Continuous on a traffic sample | Classify, ticket, hold the gate |
| Orphan and referential scan | Cross-boundary references in both directions | Nightly | Fix the mapping, not the symptom |
| Late-arriving change replay | Records updated in the old system after cutover | Nightly during soak | Investigate the writer nobody knew about |
The last row is worth dwelling on. Once a cohort is on the new system, no writes should be reaching the old one for those entities. If the nightly scan finds some, you have discovered an undocumented writer, which is the single most valuable thing reconciliation produces. Every migration has one. Usually it is a scheduled job, an admin tool, or an integration that writes directly to the database because someone needed it done quickly in 2019.
A shape that does not require stopping
Migration Sequence
The overlaps are deliberate. Nothing here is a stage gate in the project-management sense, and the phases that look sequential are not. What is strict is the ordering of a few dependencies: the seam before any traffic decision, reconciliation before any read cutover, reverse replication before any write cutover, and consumer migration before the old database is turned off.
Decommission is a project, not a date
The last twenty percent of a migration is where the money is lost, because the interesting engineering is finished and attention moves on while both systems keep costing money. Two teams' worth of on-call, two sets of infrastructure, two places to check when something is wrong, and every new feature built twice or delayed until the migration finishes. That overhead is the real budget line, and the way to bound it is to make the end explicit.
Move the legacy system to read-only as soon as the last cohort is cut over and the soak has passed. This is a strong signal and a cheap one. Anything that was still writing fails immediately and visibly, instead of writing into a system nobody is watching. Keep it readable for a defined window while the remaining consumers are moved, and then decommission against criteria you wrote down: every consumer migrated or retired, an archive of the data in a portable format with a documented schema, the retention obligations satisfied, deletion requests resolvable in whatever remains, and the credentials revoked.
Retention deserves a real answer rather than a default. If your obligations are shaped by SOC 2 commitments, contractual terms, or a GDPR deletion path, the archive has to support them: an export nobody can query is not an archive, and a copy of a database that cannot honor an erasure request is a liability with storage costs. Decide during the migration which records the new system carries forward, which live in the archive, and which are deleted, and get that decision from the people who own the obligation.
One practical forcing function: tie the decommission date to a renewal. Licenses, support contracts and reserved capacity all have dates, and a date that costs real money when it passes is the only kind that reliably holds attention.
Before you route the first request
- A seam at a stable interface, with per-request routing and a flag store
- One documented system of record per entity, and the rule for when it changes
- Capture running with lag alerting, and a restartable, chunked backfill
- An explicit merge rule using source versions, never the loader's clock
- A reconciliation suite that runs unattended and pages when it disagrees
- Shadow reads with normalization, sampled diffs, and a written diff budget
- A rollback rehearsed on a real cohort, with reverse replication proven
- The consumer inventory from query logs, with an owner named for each entry
- An identifier mapping table treated as production data
- A decommission date attached to something that costs money when it slips
What this costs, and why it is still cheaper
Running this way is more work than a big-bang cutover on paper. You build a routing layer you will delete, a capture pipeline you will delete, a reconciliation suite you will mostly delete, and you carry two systems for months. A plan that freezes and cuts over in a weekend looks cheaper in the estimate and is cheaper in the cases where it works.
The comparison that matters is against what actually happens. A freeze that slips from four weeks to fourteen costs a quarter of engineering throughput and every commitment that depended on it. A cutover that goes wrong on a Saturday and cannot be reversed costs the weekend, the following week, and a measure of trust that takes much longer to earn back. The live-migration mechanics convert an unbounded tail risk into a known, bounded, scheduled cost, and they do it while the business keeps shipping. That trade is why we build it this way.
Bottom line
The freeze is a symptom. When a plan contains one, the useful question is not how long it needs to be, it is which unsolved problem it is standing in for. Usually the answer is that nobody has decided how the two systems will be kept in agreement while both are live. Solve that with a seam, one system of record per entity, replication from a durable log, and reconciliation that runs on its own, and the freeze stops being necessary. Then move users in cohorts, keep the reverse path open until you have decided to close it, and give the old system a decommission date that costs something when it slips.
Frequently asked questions
A short freeze on schema changes to the legacy database during the final cutover window is reasonable, because schema drift under an active capture pipeline is genuinely hard. A freeze on business changes for weeks or months is not, and it usually indicates that the reconciliation and routing work has not been done. Freeze the schema, not the company.
For a system of meaningful size, plan in quarters rather than weeks, with most of the calendar spent in cohort cutover and soak rather than in building. The build phase is bounded and estimable. The part that takes real time is proving agreement between two systems across every slow cycle: month-end, invoicing, reporting, and the customer who edits something created years ago.
Dual-write means the application writes both systems inline, which is two operations that can fail independently and can be applied in different orders. Change data capture means the application writes one system and a replica is fed from that database's own transaction log, which preserves commit order and catches writers your application does not control. For migrations, capture is the default and dual-write is a narrow exception.
From the data, not from a meeting. Read a week or more of query logs and group connections by database user, source host and application name, then map each group to a system you can name. Whatever does not map is the list worth chasing, and it is where the scheduled export, the BI connection and the tool with no current owner will be.
When every consumer is migrated or retired, the data is archived in a portable format with a documented schema, retention and deletion obligations can be met from what remains, identifiers from the old system are still resolvable, and the system has been read-only long enough that nothing has failed. Write those criteria down at the start of the migration, because at the end there will be pressure to declare victory without them.