A rollback in seconds let us migrate PHP to Node during the day

I spent four years as technical lead on a US marketplace, and in that time we replaced the language, the framework, the hosting and the mobile stack. Users never saw a page that didn't work.

What was unusual is that we could have undone the migration in seconds, at any point, without a deploy. That changed how we ran the migration.

A migration needs three things:

1. Moving the data into the new shape
2. Keeping both systems consistent while they run together
3. A way back if the new one is wrong

Most plans cover the first, and good ones cover the second. The third is usually missing, which is why migrations get scheduled for 2 a.m. on a Sunday. The work is no different at night, but everyone knows that if it goes wrong there is no plan for getting back.

We took a dump of the old database and wrote a script that loaded it into the new one, remapping columns and structure along the way.

That part was ordinary, with one exception: we chose to keep the same database engine. The language, the framework, the hosting and the mobile stack all changed. If the database had changed too, we would have had nothing fixed to compare against when checking that the new system behaved like the old one.

With four things changing, keeping the fifth still was the only way to tell which of the four had broken something.

Most of the thinking went here, and the reason was not technical.

We controlled the web. A web user gets the new version on their next page load, so the web could be cut over. We did not control the phones. A mobile user updates when they feel like it, and the old apps had no way to tell their users that a new version existed. We published new apps to both stores, but an app already installed on someone's phone doesn't know that.

Shutting down the old backend would have broken working software on tens of thousands of devices. So the old system had to keep running, and the old and new systems had to agree.

We put triggers on the old database that forwarded its writes to the new one, and set up the same in the other direction. An ad created in the new system appeared in the old one, and an ad created in the old system appeared in the new. The two schemas stayed in agreement, in production, for over a year.

Syncing in one direction would have been easier, and it would have been wrong. Mobile users were still creating and editing records, so without the return path the new system would have gone stale at once, in the places that mattered.

We didn't pick a date for shutting the old system down. We shut it down when analytics showed that its traffic had reached zero, which meant the last user of the old mobile app had stopped.

That took over a year, and it was the right call. Any date we picked would have been a guess. The measurement showed that people kept using the old app long after we'd have liked them to stop.

If you are planning a decommission, tie it to a metric like that one.

Our rollback was a change of origin at the CDN, which took seconds and needed no deploy or build.

What is worth copying works with or without a CDN: the rollback path must not depend on the thing you are rolling back. If getting back requires a deploy, and the deploy pipeline is part of what you changed, your way back is no more reliable than the change you are trying to undo.

Ask one question about your own plan: if the new system is wrong at 3 p.m. on a Tuesday, what is the sequence of actions that returns us to the old one, and how long does it take? If the answer involves a build, it's too slow. If the answer involves a person who is on holiday, it isn't an answer.

I hadn't expected the rollback to change when we did the work. When rollback is seconds away, you migrate during the day: on a Tuesday, while everyone is awake and the team is watching, when someone in support can pick up the phone and no engineer has to reason through a problem at 3 a.m.

Working in daylight took confidence, and it came from how the migration was set up. A change you can undo instantly is far less risky than one you can't, so it needs less ceremony.

The sync ran for over a year, which was correct, but we never wrote down what we'd do if the two systems disagreed. They didn't, and that was luck as much as design. A scheduled reconciliation check that compares counts and checksums between the two costs almost nothing, and it would have turned "they agree" from an assumption into a fact.