Weeknotes s07e25

TL;DR: To the Moon and Back — Taking our Nervecentre EPR live

[week ending 07/10/2025]

This week we went live with our EPR. We finish the week in a good place, but it was a incredibly hard slog. This weeknote diverts from my normal format for obvious reasons.

As I sit here with a blank page in front of me, I’m struggling to know what to write. It’s been a terrible and tremendous week. I’m even more tired than when I return from leading a Cub camp and struggling to put a sentence together. However, I’m also compelled by the wish to write something; wishing to adhere to strongly held belief in seeking to work in the open ¹. I am inspired by Matthew Cain², who I recall lived this out by blogging openly about the cyber security attack at Hackney several years ago. This drive is of course balanced with the recognition that a) Many people have had a rubbish few days of disruption and b) NUH have been very much in the national spotlight this week. So what follows is me trying to carefully tread that line. I apologise if I don’t quite achieve it, but I hope you trust my intent.

In my last weeknote, we’d just taken Careflow and Nervecentre down in preparation for the go-live weekend.

Part 1: The Other Side of the Moon

I left the office on Friday evening, with great excitement. All looked on track for the go-live progressing according to the timeline below.

At around 9pm on Friday we had our Decision Point 4 meeting, which was the point when the configuration import commenced. We then entered into one of the periods which I coined our “Dark Side of the Moon” moments; times when things were kicked off and we had to wait for several hours for the task to complete, or some indicator to move to tell us things were happening. In my head, I played the section of the track by Public Service Broadcasting called The Other Side where (at 03:05) everything is quiet and ground control is waiting to hear the radio signals re-sync after Apollo 8 passes round the moon. The controller says “now we are in our period of the longest wait”. When the astronauts do finally speak (at 4:48), there is an audible delight in the controllers voice as they reconnect. There have been many of those moments this week.

Public Service Broadcasting album cover — The Race For Space

I was back in the EPR Command Centre before 7am on Saturday morning, ready for Decision Point to commence loading the data taken out of Careflow into Nervecentre. This was the longest of the Dark Side Of The Moon moments, with an estimated 24 hours to wait. I can’t really remember how the rest of the day went, but I seem to remember I was busy and failed to tackle my email backlog, which I envisaged I’d be able to make some progress on. I visited various wards with floor walkers checking business continuity processes and seeing that everyone was ok. In the various points where we saw progress on the migration, it looked like migration was running 1.3x compared to our Dress Rehearsal which would have given us a delay of c.7 hours. I left in the evening with the hope that that by the time I woke up, the migration would have caught up or would nearly be complete.

In the morning on Sunday I joined Decision Point Call DP6 to confirm that the import of data had completed, around 2 hours later than expected. At this point, Decision Points started moving faster as testing commenced. In the afternoon DP7 commenced integration — this was our point of no return; once signals started flowing into downstream systems we were firmly in a “fix-forward” situation. During the afternoon, our CEO Anthony came to see what was going on. In optimism, we recorded a video, explaining what was going to happen next about the data load.

I’m now wishing we’d got him a yellow t-shirt!

Across the hospital, floor walkers commenced going into areas and asking them to begin entering the data recorded on paper during our downtime. As one of the highest volume areas, several of us went into ED and helped re-key their data. This was the key to sequence of events — without admissions being recorded in ED, it would not be possible to transfer them electronically to the ward where they now resided.

Floorwalkers go!

We finished in ED at about 9pm and then I joined teams going round wards, helping them as needed and flagging with ED which of the admitted patients needed the clinical information adding. Our plan was to enter patient data for those who had been discharged later, as the priority was those in beds at the time. I think I finished it at about 02:00am on Monday morning and decided it was not worth driving home only to be back early in the morning. So being a good Scouter, I put up my camp bed in the office and got my head down, thinking that although it had been a long day, we had got the hospital ready to use Nervecentre in anger and in volume.

Be Prepared!

Part 2: The South Pole-Aitken Basin³

On Monday at 08:15 we started to receive messages that Nervecentre was running very slowly and performance data showed that the database CPUs were maxing out. Quite a few areas persevered but many felt it was too slow and moved into their Business Continuity Processes (BCP). Technical teams from Nervecentre and the Trust worked together and found a few potential causes, including a difference between database servers. A hot fail over to a second server took place. The performance was much improved, but in hindsight this was under reduced load, so it was hard to tell if the improvement was material. We helped ED do a data catch up for the second time and the hospital went into the night with an electronic ED and bed state.

I bounced into the office on Tuesday morning, thinking that the changes made would mean we could have a better day, but sadly by about 9am, slowness was being experienced again across the Trust. Once again, local decisions were taken to limp on or stay in BCP. At the same time, due to operational pressures, not helped (but not caused) by a slow EPR, the Trust called a Critical Incident. This resulted in national attention in the BBC and HSJ (£wall).

Source: https://www.bbc.co.uk/news/articles/c74jv0xxgpyo

Further work on the database performance took place, and we got generous support from neighbouring Trusts in the East Midlands and beyond (special thanks to UHL) to review our infrastructure. Several capacity increases were applied which made no appreciable difference. As the number of users dropped using the system in the afternoon, the system performance recovered and entering into the evening, we were able to get ED back using Nervecentre, and establish a full bed management position.​​ Nervecentre made a further series of fixes in the evening, which were deployed. Whilst these were shown to work under low volumes of users, it was acknowledged that it would only be when larger numbers were using the system that we would know if it has made any improvement.​​ At that point we made a plan not to allow universal access on Wednesday morning, but to carefully introduce new areas in a phased way. At that point we began to have wider access to a set of metrics across the EPR Programme team that we agreed reflected performance indicators.

Part 3: Re-Entry

On Wednesday, first thing we did nothing but have a Dark Side of the Moon moment, watching and waiting until mid morning; performance was stable and within tolerance, so we gradually enabled areas of Outpatients in 12 tranches, watching those metrics like hawks. During this period, Nervecentre applied some more database performance improvements. In the afternoon, the Trust stood down the Critical Incident. Thing were looking up enough for us to feel more encouraged that things were going in the right direction.

After night of good system performance, we followed the same path on Thursday; waiting until 10am, before starting to deploy Pre-Op and theatres, again in a phased manner.

On Friday, we enabled EPMA and a massive effort of transcribing c.1300 paper drug charts took place during a few hours, meaning we finished the day with all elements of the system up and running, with performance running within tolerance.

Looking back, this was an amazing week, of really hard moments, offset with some brilliant people all trying to get things back to a good state. People were simply fabulous and makes me so proud to work in this organisation.

Chart, showing slow-running (>1s) database requests, showing the impact of the improvements introduced on Monday and Tuesday.

Part 4: Post-Flight Debrief

The three parts above describe a pretty linear timeline, but I also wanted to reflect a few other things…

Good People: There were so many people who went above and beyond this week to try and help. I observed people of all stratas of the organisation rolling up their sleeves and doing the next job needing to be done. I’m so incredibly proud of the team and if I let myself as I write this, I could weep, simply thinking about the wonderful hearts of those fine folk. There are too many to mention, but we will find a way to thank and recognise them.

Agility: This was a pretty awful situation, and there was certainly a point on Tuesday when I began to wonder if we’d run out of potential solutions, and the means to test them out in advance without high user volumes. The key thing was that Nervecentre were able to be really agile in applying changes. I kept a tally at one point, but think I lost count after 12. There were other database changes that were able to be applied without any downtime, and then some other fixes to the front end, which could be applied within 10 minutes, simply requiring users to log in again. What made it hard to pin down in those first 48 hours was that the problems only materialised when there was heavy load on the system. We certainly made the mistake on Monday evening, thinking one of the first changes had made the difference. However, pinning down the right metrics to monitor was the route to the solution.

Before and after go live CPU on Comet servers, after a performance boost

Partnership: Often this word is used glibly and without meaning. I felt through this period, both NUH and Nervecentre were shoulder to shoulder, seeking a solution to the problem rather than pointing the finger. At times it was hard to know who was from what organisation, such was the focus on getting the Trust back up and running. I’ve not seen that from many , or perhaps any suppliers in my career. I will be fascinated to see what that means for how we work together in the next few years.

Multi-Disciplinary Leadership: Mark and I took on the Exec leadership roles at various times and at various hours. There was a time when I slipped into negativity and Mark rightly called me out. I listened to a podcast earlier today that talked about companies with joint CEOs and the benefit of complementary skills leading to better outcomes. It did make me think about how I enjoy working with him and am delighted that we have loads more ambition to build on this foundation with Nervecentre. The other thing that was really helpful is to have someone to talk things through all the time; especially as fatigue impacted on my clarity of thinking at times.

Widespread Support: This could have been a moment when people in the organisation and beyond could have pointed the finger at us and blamed us for what went wrong, and particularly when we went into the Critical Incident. Certainly, there were moments in the middle of the night when I felt so dreadful about this when I put myself in the shoes of others. However, I don’t recall anyone with NUH or in the wider NHS doing anything else but seeking to help. I had some lovely messages from colleagues old and new; many acknowledging that they couldn’t help, but offering their emotional support nonetheless. It was incredibly touching.

Doing One Thing vs 1000 Things: Whilst this week was stressful and incredibly tiring, there was something great about being free to focus on the One Big Thing, rather than trying to spin loads of plates. Several people commented that it had some equivalence to those early days of covid, where there was a focus to do whatever was necessary to meet the emerging and rapidly changing challenge. However, I do dread to think how long it will take me to recover my inbox.

Foundation for the Future: We now have a functioning EPR. It’s been harder work than I hoped to get there, but I’m so optimistic on what it means. There will be things that materialise in the coming weeks that need fixing, but I’ve seen first hand, how working together with Nervecentre we can turn these things round rapidly so I’m confident of our ability to face the next set of challenges.

Thank you so much to everyone who helped in anyway on this programme — you’re brilliant!

Footnotes

¹ I would have linked to the NHS Design Principles but disappointingly the hosting at https://service-manual.nhs.uk seems to have evaporated and replaced by something dodgy. I’d be gutted if it has gone forever.

² Dunno if I’m being useless in my fatigue, but the only link to Matthew Cain’s blog is in Emma’s — https://emma-mcgowan.medium.com/weeknotes-1-2-29th-may-4th-6b2f578b1715, but she does link to his YouTube video.

³ The South Pole-Aitken basin is the largest, deepest and oldest basin on the moon.