How I Took a Legacy Production System Live on Node 20 and MongoDB 6: What Actually Happened (Part 3)

The preparation, the cutover night, the first day live, and why we didn’t roll back. Part 1 — Node 6 to 20 · Part 2 — MongoDB 3.4 to 6 · Part 3 — The cutover

If this is your first time coming across this series, this is Part 3 of my legacy code upgrade series. Part 1 covered how I upgraded the application from Node 6 to Node 20, and in Part 2 I covered how I upgraded MongoDB from 3.4 to 6. I went through the driver upgrade, the things that broke, and why I chose to migrate the data logically into a new MongoDB 6 instance instead of upgrading the old one in place. You can find Part 1 here and Part 2 here, and I suggest reading those first for the full context. This final part covers the night we dropped the database and went live, and everything after it.

For context, the system is a large monolithic Express.js REST API that has been in production for several years. It consists of roughly 140,000+ lines of JavaScript spread across 250+ files, with 67 production dependencies. The application serves multiple user roles and includes several third-party integrations (payment gateways, push notifications, etc.), backed by MongoDB. It is a complex, non-trivial legacy codebase.

A note on wording: in this part I say “we” a lot. Even though I laid out the plan, the cutover was a collaboration between our team, the DevOps and infrastructure teams, and the operations team.

Disclaimer: Same as the first two parts, every legacy codebase is different, and what worked for my project may not work exactly the same for yours. This part especially, because a cutover depends a lot on your infrastructure and on whether your system has a time when nobody is using it.

The rollback plan

I mentioned this in Part 2, and I will repeat it here because it is the first thing I would tell anyone doing this: have a rollback plan, communicate it with every stakeholder, and agree on the decision point before you do anything. That decision affects both the technical operation and the business, so it is not something you should take full responsibility for on your own. Get organized, agree on the plan with the team, then move to the next step.

For us, there were two options.

The first was to use a replica set. The idea was to keep the old MongoDB 3.4 instance and the final dump of the old database in place, while running MongoDB 6 as the new production database. With replication in place, we could capture the operations happening on MongoDB 6 after going live and use that information to reconcile the changes if we needed to switch back to the old system.

This mattered because rollback is not as simple as restoring the new database back into MongoDB 3.4. MongoDB is not backward-compatible in that direction, so once real users start writing to MongoDB 6, those new writes cannot simply be restored into the old 3.4 database. We would need a way to account for the changes that happened after the cutover.

The second option was much simpler: keep the old MongoDB 3.4 instance running, and if something went wrong within a short time of going live, switch the application back to the original setup before the new database accumulated too many changes.

I looked into both options and discussed the trade-offs with the team. We went with the second one because we were confident in the upgrade and in the amount of testing we had done, and the simpler rollback plan was enough for the window we had.

If you are in the same place, look into every realistic way you could roll back, then document the plan and discuss it with the people responsible for the system and the business. One thing to be aware of is that a clean rollback only lasts until real users start writing to the new database. Before that point, going back costs you a restart. After that point, going back costs you data. You should know exactly where that line is in your own plan.

How we got to the cutover

Looking back, one thing we did made the night much easier than it would have been.

We started the process four days before the cutover.

Not the migration itself, that was already done and tested. The setup. The idea was that on the actual night we should have as little to do as possible. Anything that could be done in advance, we did in advance.

In those four days we:

• moved the code base to the new server
• set up the CI/CD on the server
• put the backup and the recovery plan in place
• moved the large files ahead of time, so that during the cutover we would only have to move what had changed since
• restored the real production data into the new MongoDB 6 instance
• ran one full round of testing on all the critical flows against that restored real data

The last two are the important ones. By the time the cutover night came, we had already done the whole restore once with real data, and already tested the application against it. So nothing on the night was new to us. We were repeating something we had already done, just with a fresh copy of the data.

Then there was the timing. Our product closes to customers overnight, so we planned the cutover for that window. That gave us enough time to do everything overnight without any customer being affected.

If your system has a window like that, plan around it first, before anything else. It changes the cutover from something you have to survive into something you have time to do properly.

The cutover night

We got notified that the product was closed to customers at 11:30 PM, and we started work.

The gap between the dump and the restore is the time it took to move the dump file over to the new server.

The two restores were not planned. After the first one we realised we had forgotten to run the script that updates the IDs, the issue I covered in Part 2. It has to run as part of the restore, so we had to do the whole thing again. It cost us around 40 minutes. It did no real damage because we had the window, but it is worth saying why it happened. We had done this same restore four days earlier during the rehearsal and it went fine, so on the night we were going from memory for that step instead of from a written list. That is exactly the kind of step you forget, and it is why it is in the checklist at the end of this post.

The database was around 2.3 GB, and the restore itself took 3 minutes and 24 seconds. That time is mainly affected by your database size and your connection, so take both into account when you plan your own cutover. If your database is much larger than this, do not take my number as a reference, measure it yourself during the rehearsal.

There was also one thing that could only be done on the night. The third-party integrations needed the new server to be registered with them, and that had to happen during the cutover itself, not before. So that was part of the night’s work, and after it was done we tested those flows again to confirm they were working from the new server. If you have integrations that are tied to a server or an IP, find that out early and keep it on your cutover list, because it is easy to forget until the moment something fails.

The rest of the night was testing. We started testing the main flows once again and spent the rest of the night doing that, our team and also support from the DevOps and infrastructure teams.

In total the database was not taking writes for about an hour and fifteen minutes. The product itself stayed closed overnight until we finished testing at 4 AM, so no customer was affected either way.

Just like that, the upgrade went live. We finished up and had everything running at 4 AM.

Verifying the data

One thing I would not skip, and would not do by eye: we had a script in place to verify the data after the restore.

It compared the old database and the new one and checked that everything matched. We ran it right after the second restore finished, before we moved on to anything else, and everything came back matching. That is what let us continue with the rest of the night without wondering in the background whether something had been lost.

Write that script before the cutover night, and run it during the rehearsal too. On the night you want a check that gives you a yes or a no, not a set of collections you have to go through manually while everyone is waiting on you.

The first day live

The next day was the first day going live with real customers on it.

We created a channel specifically for this, us and the operations team, and we stayed on alert to fix any issue that could happen as quickly as possible.

The reason for a separate channel is that the operations team will notice small things they are not sure about, and if there is no obvious place to report them they either say nothing, or it comes back later as a bigger problem. A channel just for this makes it easy for them to report anything, even small things, and small reports are exactly what you want on the first day. Anything that could affect the operation needs to be fixed as soon as possible, not held until the next meeting.

What happened after going live, and did we roll back?

After we went live we had six issues in total. I can’t go into the specifics of them here because they are internal to the product, but what I can say is this: all six were minor, we fixed every one of them in a short amount of time, and the operation did not get interrupted much.

I don’t think that was luck. Before the cutover we had done very intensive testing, and we included every person who is part of this product, not only the developers. That is how we caught the valuable issues early and were able to fix them before going live. The ones that would have hurt on day one were already gone by the time we dropped the database.

After that we stayed on alert for a couple of weeks. The issues became less and less, and then they stopped.

And the more interesting thing, we didn’t roll back.

We had spent real effort on the rollback plan, we had confirmed it, and we never used it. I still think it was worth it. Knowing we could go back was the reason we were comfortable going forward.

The checklist we used

This is the list we worked from, written out so you can take it and adapt it. It is not clever, and that is the point. On the night you don’t want clever, you want a list that stops you from forgetting the boring step.

Days before the cutover

On the cutover night

After going live

The one thing I would insist on: write the times down the moment they happen. I kept mine in a notebook during the night, and it is the only reason this post has real numbers in it.

Reflection

Looking back at the whole process, from Part 1 and Part 2 to this part, the process is not hard, but it is messy. I had to tackle each issue intentionally and carefully, and what helped was researching each one properly before touching anything. I knew what the options were and why I was choosing one over the other, and I documented the whole process I went through. That is what made it a controlled process instead of a messy one.

What I want you to take from the series, if you are doing something similar, is the shape of the process and how to approach it. Every code base and every technology has its own specifics.

Advice

If you are a tech lead, a CTO, or someone responsible for the tech department at your company, plan an upgrade of your product every time the technologies you use release a new version. Having seen this whole process first hand, I would not wait again. It does not feel urgent, but the alternative is the decay of an old version gap, and ending up in a place where your system can’t seamlessly add new features.

And if you are in the process, or about to start your upgrade, these are the things that made the process manageable for me:

1. Allocate enough time for this. It’s not a one week task. Allocate enough time so that you don’t overlook issues and end up rushing.
2. Read up on everything you do, thoroughly, and understand what you are doing very well. Apart from making informed decisions that you can take accountability for, this will also help you not panic when something fails, because you already know the pattern.
3. While doing the upgrade, document everything. When you are dealing with a number of issues, breakages and decisions, you won’t remember every single thing you did, and the document is what puts you in a position to answer any question that gets raised.
4. Do it in phases if possible. As I put it in Part 1 and Part 2, I intentionally decoupled the Node upgrade from the database upgrade, and it makes the whole process manageable. It helps you track down issues to the exact point that caused them, and it gives your mind a manageable scope to work with.
5. After the upgrade is done, communicate well with the team to undertake thorough testing. For us the testing phase took longer than the upgrade process itself, and that is the whole reason we didn’t have to roll back and the operation didn’t get affected. Considering the size of the version gap we covered, that is a big win. Do the same.

That is the end of the series. Node 6 to 20, MongoDB 3.4 to 6, on a live production system with 140,000+ lines of legacy JavaScript, and no rollback.

I wrote these three parts because when I started, I could not find them. Everything I found was either a small example project or a list of version changes. If you are in the middle of one of these, I would like to hear what broke for you.

And if you are planning an upgrade of your own, or you are in the middle of one and want my input, or you just want to collaborate on something similar, get in touch. You can reach me at:

Email — blenredwan@gmail.com LinkedIn — Find me on LinkedIn