Towards Continuous Firmware Delivery

Or, Fixing the Engine While It’s On.

Stop me if you heard this one before: someone asks the embedded engineer why they don’t test their code and they say that “Testing is for software developers”.

Programming Like it's 1989

Embedded development moves slowly, and understandably so. We are a cautious bunch, shaped by the tales of hardfaults and stack overflows.

We want certainty, and the tried and true code written 20 years ago using Windows XP compatible IDEs is usually preferable to the shiny new tools and languages that have been created since then. Most of the time it's not even possible to use newer stuff because we simply lack the resources to do so. Either we don't have the compiler support or the new stuff doesn't fit into the available memory.

It's the 21st century, retrocomputing should be done only for fun.

However, when taken to extremes, this creates a culture of complacency. New techniques and tools are passed along simply as fads or even as something that does not apply to embedded. This is doubly true when we talk about development methodologies such as Test Driven Development and Continuous Delivery. I've heard many times that they can't be implemented for embedded development because embedded is different. But first let's tackle each of these topics.

Test Driven Development

TDD was created by or "rediscovered" by Kent Beck in the late 90’s (thanks Wikipedia) and consists of writing some test code before proceding to the implementation of a given feature. This way you have a working specification of your code that can help you check if you are on the right track.

Do this in small increments and for sufficient time, and you will end up with one (or several) comprehensive test suites that can be used verify your code. Seems great, right?

And here comes the first common argument against TDD, which is not embedded specific: it's hard. I have to agree with the naysayers here, it's quite hard to think about the code you still haven't written. You don't know the details of the implementation or even how the interfaces might look like! That is true. But bringing this to the front when implementing a test forces you to think about this stuff earlier, giving you hints on how to move forward.

Some of the older folks here might remember the advice of writing you header files first. This is the same idea, but with a bit more juice.

The second one I hear often is that "we are interacting with the real world here. TDD is for people who don't interact with hardware!". That one is half-true. Indeed we interact with real world stuff, with its own constraints. But that is a dependency like any other! Web developers need to query databases all the time. Databases have their own constraints. Endpoints and libraries are also dependencies, and people get around those.

Continuous Delivery

In their book on the subject, Dave Farley and Jez Humble point out that Continuous Delivery is not a tool, not a framework, but a discipline. The driving idea behind it is that we want to avoid the huge risks associated with a big delivery by splitting it up in small, more manageable ones. The term "delivery" also refers having your product in a deliverable state, not necessarily deployed to customers.

And guess what? This is extremely hard to achieve because it is built on top of a series of good software development practices. The end goal is to deliver value in small increments, but this requires us to shorten the feedback loop between development and validation. There is no way to deliver value continuously if the validation cycle takes four weeks while another 30 commits enter the release branch in the meantime.

If we are aiming for short feedback cycles, the best place to start is unit testing. And what helps us have a comprehensive test suite when we finish developing a feature? TDD.

Indeed, among the practices that helps us reach the state of Continuous Delivery, testing is one of the core ones. Test early and test frequently and fewer suprises will pop up as the code moves down the validation pipeline.

With these concepts out of the way, let's move on to more practical topics.

Who are we?

Stratio is a Portuguese company in the predictive automotive maintenance sector. We have tens of thousands devices installed in clients all over Europe, Asian and South America. Our main clients are in the Public Transportation sector and are really worried about service availability — after all, waiting for a late bus can ruin anyone’s day. That means our number crunching must be rock solid.

A few years ago we realized that our firmware development process was getting in the way of delivering valuable, timely updates to our customers. So we gathered our engineering team and set about changing this situation, as you will see.

The Past

Photo by Hadija on Unsplash

Think about the software development processes you know about and see if any of this sounds familiar:

• No standardized development environment. Each engineer does their own toolchain setup by following a series of loosely-related tutorials.
• Building a new binary is an IDE-dependent process. Someone needs to move the mouse and click on the "build" button.
• Production binaries come from the developer's machine. They click on the build button, zip the generated files and upload the release to the company's servers.
• Tests are manual. Someone keeps an eye on a terminal screen looking for a debug message. If it appears, the feature is approved.
• Validation cycles can take anywhere between one week and two months.

As you might have guessed, this is where we were just a few years ago, and it was unsustainable. Getting a release up to standard to be sent to client was an arduous process, with many tales of pain and terror.

Solving this became a matter of life and death and, with management buy-in, the fabled Refactoring Effort started. The biggest pain point was the communication unreliability. Our business model is based around crunching a bunch of datapoints every second, and not having datapoints is a Bad Thing™.

So, at the beginning we wanted to do the following:

• With as little rework as possible, solve the deadlocks that caused the comms outages.
• Increase reliability by writing multiple types of tests to validate desired behavior and detect issues early.
• Reduce our dependency of the target hardware in order to speed-up testing.
• Build and run tests in a pipeline on every commit

Testing the Untestable

The first effort was then building the software using CMake. This took some effort and required adjustments throughout the entire refactoring effort but once it was solved, we could finally put this into a pipeline.

The main hurdle towards this goal is that our code was never written with testability in mind. Poor abstractions and extreme coupling made it nigh impossible to test units separately.

Since our product is FreeRTOS based, we decided to try and run our code on top of a FreeRTOS x86 port, which solved this dependency. Running the entire application also involved writing emulators for our peripherals, since these did not come with the port. These peripherals included GNSS, an LTE Modem, which are connected via UART lines, a few I2C and SPI devices, ADCs…

So, how did we solve all of this? Simple, we implemented as little code as needed. For a while, emulated ADCs read only 0V for all cases, until we needed them to give a different value. Our FLASH emulator is simply a 128 MB array that lives in the host device RAM.

Once again, you are getting the good parts here. It took several months from going from zero to something usable, but the first time you get a test running in the pipeline, the feeling is great.

And it's important to raise a disclaimer here: testing the whole application as we did is incredibly difficult, even more when the application wasn't architected with testability in mind! If we had started development with testability in mind, things would be much different.

Hardware Tests

However, simulation can only take us so far. It is not uncommon to write something that passes with flying colors in the simulation and fails miserably in the target hardware. Identifying this pain point, we created a new target that ran a very small subset of tests directly in the hardware.

This comes with a different issue: there is no way around the laws of physics. Want to test the accelerometer? Better start throwing that device around. Do you want to format the device's memory? Hope you have five minutes to spare for each run.

Hardware is, well, hard. But it is the ultimate source of truth. An application that works only inside the simulation isn't worth very much. Having tests that validate the final execution environment, however small they may be, is an invaluable step.

Vehicles-In-The-Loop

Finally, once we started deploying the new firmware versions, we had another question: How is the version performing as a whole across our clients' fleets?

Fortunately we already had in place most of the infrastructure necessary for this, we "only" had to collect and generate statistics from it. We have thousands of vehicles spread across different clients on multiple continents. We had to find a way to group and query these clients to extracts metrics that made sense to the Firmware and Hardware teams.

So, one of our developers threw together a quick solution that did generate some of these metrics and we finally had an idea on how the new firmware was doing as a whole.

Soon we started seeing the fruits of our labor: as we deployed the new versions, communication outages reduced drastically, and we went from around 4 hours of delayed data from the vehicles to almost zero. We reduced GPS loss significantly, with some clients reporting aproximatelly total availability.

The Present

After around two years of efforts, we managed to rework the most critical modules in our systems, and we have a test suite that tells us when things are not going the way we expect.

We aren't rid of mysterious production bugs, but they now happen every six months instead of every two weeks. No one has really done any overtime in the last two years, spending those precious minutes with family and friends.

And that is the main point of all this effort: peace of mind, for everyone. Product Managers can focus on new features, customers have a high available service. Quality analysts now take around 1 week to validate a release. The firmware team specially can now focus on delivering new features instead of fixing bugs.

Software Architecture as a driving force

I believe the single greatest lesson we got from this is that yes, it is possible to test firmware, but is a great deal easier with architectural decisions that support testing, and the best way to do this is to make testing an integral part of the process from the very early stages.

One of the toughest parts of our process is when we need to add some emulation logic to the code, because it usually is a direct or very thin call to a FreeRTOS function. This means that the "seam" in the code that allows us to control the behavior of the system is closely tied to the operating system or the hardware, leading to rather complex emulators and test setup.

As we gained more experience, the way we tackled this was by writing proper abstraction layers that we then inject in the module. The application and the tests might even use different abstractions:

class IDependencyInterface {/*...*/};
class ActualDependency : public IDependencyInterface {/*...*/};
class FakeDependency: public IDependencyInterface {/*...*/};

// Application
int main()
{
  auto dep = ActualDependency()
  auto my_module = MyModule(dep);
}

// Test
TEST(MyModuleTest, ShouldDoStuff)
{
  auto dep = FakeDependency();
  auto my_module = MyModule(dep);

  my_module.write(1234);
  ASSERT_EQ(dep.last_value, 1234);
}

Well architected code shows the very same characteristics as code written following TDD. It allows us to write decoupled code with proper dependency management, allowing the delivery of small, constant batches of value that have been rigorously tested.

While it is possible to write well-architected code without TDD, extensive testing is the most obvious way to know if you are going in the right direction. Code that is hard to test is poorly abstracted, and the opposite is also true.

Why "Towards"?

Still, we haven't reached a state where we can say we are doing Continuous Delivery. There are still lots of manual tests to be automated, improvement points, flaky tests, ugly code.

We are moving towards that goal, but we can also get further away from it if we let things slide. If we stop testing, if we let ourselves get caught up in the heat of a particularly difficult bug, if we stop looking for improvement points, both in the code and within the team.

Still, it's goal worth pursuing. We are now free to chase the important things.

Conclusion

I'd like to be explicit here and draw your attention to a very important point: we did not use any shiny framework for this. We didn't pay a dime for the tools (besides GitLab), and stuck with tried and true libraries and good old elbow grease.

Also this was done with a teams of less than 10 developers and QA analysists, none of whom took costly courses on TDD or Continuous Delivery, nor relied on high-priced consultants. The theory upon we based our work is widely available for free in the internet. Our compiler is GCC, our build tool is CMake, the test framework is GoogleTest.

While the glue that binds all of this together was hand-written (no AI agents were injured in this refactoring. Don't look at us like that, this was done a few years ago…), a curious side-effect of this approach is that, once we started incorporating AI agents into our workflow, our tests allowed us to progress quickly and with more confidence that the AI wasn't breaking anything. Automated pipelines and comprehensive test coverage are the safety net that gives you confidence that the behavior of your product remains consistent across multiple iterations, regardless of your development method.

If you take away one thing from this post, let it be this: firmware isn’t exempt from modern software practices. We can — and should — build systems that are testable, maintainable, and safe. You just have to start.