The part of a migration everyone skips: the way back

We spent four years in technical leadership at a US marketplace, and in that time we replaced the language, the framework, the hosting and the mobile stack. Users never saw a page that didn’t work.

The interesting part isn’t that we migrated. Everyone migrates. It’s that we could have undone it in seconds, at any point, without a deploy — and that changed how the whole thing was done.

Three parts, and the one that’s missing

A migration needs three things:

1. Moving the data into the new shape
2. Keeping both systems consistent while they run together
3. A way back if the new one is wrong

Most plans have the first. Good ones have the second. The third is why migrations get scheduled for 2 a.m. on a Sunday — not because the work is different at night, but because everyone knows that if it goes wrong there is no plan, only a long evening.

Moving the data

We took a dump of the old database and wrote a script that loaded it into the new one, remapping columns and structure along the way.

That’s the ordinary part, with one decision worth naming: we deliberately kept the same database engine. The language changed, the framework changed, the hosting changed, the mobile stack changed. If the database had changed too, there would have been no fixed point against which to check that the new system behaved like the old one.

When you’re changing four things, keep the fifth still. It’s the only way to know which of the four broke something.

Keeping both consistent

This is where most of the thinking went, and the reason was not technical.

We controlled the web. A web user gets the new version on their next page load, so the web could be cut over. We did not control the phones. A mobile user updates when they feel like it, and the old apps had no mechanism to tell them a new one existed — we published new apps to both stores, but an app already installed on someone’s phone doesn’t know that.

Shutting down the old backend would have broken working software on a large installed base of devices. So the old system had to keep running, which meant old and new had to agree.

We put triggers on the old database that forwarded writes to the new one — and the same in reverse. An ad created in the new system appeared in the old, and an ad created in the old appeared in the new. Two schemas, kept in agreement, in production, for over a year.

   old mobile apps                        web + new apps
          │                                       │
          ▼                                       ▼
   ┌──────────────┐   triggers, both ways   ┌──────────────┐
   │ old database   │◄─────────────────────►│ new database   │
   └───────────────┘                         └───────────────┘
     ran for over a year, until traffic to the old side hit zero

One-directional sync would have been easier and would have been wrong. Mobile users were still creating and editing records; without the return path, the new system would have been immediately stale in exactly the places that mattered.

Knowing when to stop

We didn’t shut the old system down on a date. We shut it down when analytics showed traffic to it had reached zero — when the last user of the old mobile app had stopped.

That took over a year, and it was the right call. A date is a guess dressed as a plan. A measurement is a fact, and the fact was that people kept using the old app long after we’d have liked them to stop.

If your decommission has a date but no metric, you don’t have a plan, you have a hope.

The way back

Our rollback was a change of origin at the CDN. Seconds. No deploy, no build, no waiting.

That property is the one worth copying, and it’s not about the CDN specifically. It’s that the rollback path must not depend on the thing you’re rolling back. If getting back requires a deploy, and the deploy pipeline is part of what you changed, then your way back is exactly as reliable as the change you’re trying to undo.

Ask one question about your own plan: if the new system is wrong at 3 p.m. on a Tuesday, what is the sequence of actions that returns us to the old one, and how long does it take? If the answer involves a build, it’s too slow. If the answer involves a person who is on holiday, it isn’t an answer.

What having a way back changes

This is the part we didn’t expect.

When rollback is seconds away, you migrate during the day. You do it while everyone is awake, with the team watching, on a Tuesday, when a support person can pick up the phone and an engineer isn’t reasoning at 3 a.m.

The confidence isn’t bravado — it’s structural. A change you can undo instantly is a different class of risk from one you can’t, and it deserves a different class of ceremony.

What we’d do differently

The sync ran for over a year, which was correct, but we never wrote down what we’d do if the two systems disagreed. They didn’t, and that was luck as much as design. A reconciliation check — comparing counts and checksums between the two on a schedule — costs almost nothing and turns “they agree” from an assumption into a fact.

This came from four years of technical leadership at a US marketplace, where the same period also produced an analytics layer taking two million writes a day. Read the case study →