Bringing Up a Secure Embedded Linux Platform on Arm Cortex Chip

From BL1 to OP-TEE, U-Boot, Linux, NAND, recovery, and production delivery

Some projects do not begin with a clean architecture diagram.

They begin with a board, a small 32-bit Arm Cortex-A class multicore SoC, a set of boot images, a NAND flash device, a display subsystem, and a very simple question:

Can we turn this into a secure embedded product that boots reliably, stores data safely, drives the display, survives recovery, and can be delivered as a production image?

That was the shape of this project.

It was not an application project, and it was not a single driver task. It was a platform bring-up effort that stretched across firmware, secure world, bootloader, Linux kernel, NAND, Buildroot, filesystem layout, display/runtime integration, memory-to-memory DMA utilities, rescue flow, and validation tooling.

The stack eventually looked simple when written down:

Boot ROM handoff
  -> BL1
  -> BL2
  -> Chain of Trust validation
  -> OP-TEE secure world runtime
  -> U-Boot normal-world bootloader
  -> Linux kernel + device tree
  -> read-only root filesystem
  -> writable runtime storage
  -> display/runtime packages
  -> validation and stress tools

But it did not feel simple while building it.

The work was really about turning that list into a set of agreements.

BL1 and BL2 had to prepare enough of the platform for the next stages to trust the machine state. Chain of Trust had to make each handoff meaningful rather than merely sequential. OP-TEE had to establish the secure-world runtime under Arm TrustZone. U-Boot had to continue in the normal world and pass Linux the right image and device tree. Linux had to agree with the bootloader about NAND. Buildroot had to produce both observable debug images and controlled release images. The display/runtime stack had to run on actual driver behavior, not on assumptions.

That is the difference between “Linux booted once” and “we have a product path.”

The First Decision: Treat Boot As A State Machine

In early bring-up, it is tempting to discuss boot in broad labels:

“the firmware” “the bootloader” “Linux”

Those words are convenient, but they hide too much.

When a board does not boot, “the bootloader failed” is not precise enough. Did BL1 fail to initialize required state? Did BL2 fail to load or authenticate the next stage? Did secure-world memory overlap with normal-world expectations? Did OP-TEE initialize correctly? Did U-Boot read NAND differently from Linux? Did the FIT image contain the wrong kernel or device tree? Did the root filesystem mount but lack the runtime state needed by user-space?

The first real design decision was to make the boot flow explicit.

Boot ROM / BL1 -> BL2 -> OP-TEE -> U-Boot -> Linux -> rootfs

Once the stages were named, each boundary could be reviewed:

• What does this stage load?
• What does it authenticate?
• What memory does it own?
• What state does it pass forward?
• What happens when it fails?
• Can the board still be recovered?

That last question became important much earlier than expected.

Chain of Trust Was Necessary, But Not Sufficient

Secure boot often gets summarized as “verify the next image.”

That is true, but incomplete.

Chain of Trust answers one important question:

Is the next image trusted enough to execute?

But an embedded product also needs to answer follow-up questions:

• What happens after the image is accepted?
• What remains protected after Linux starts?
• Where does persistent secure state live?
• What happens if the normal boot path is broken?
• How do we recover without creating a security bypass?

That is where Arm TrustZone and OP-TEE changed the design from a boot-only problem into a runtime boundary problem.

TrustZone gave the platform a hardware-backed separation between secure world and normal world. OP-TEE provided the secure-world runtime. U-Boot and Linux continued in the normal world, but not every operation and not every piece of persistent state belonged there.

This distinction mattered.

Secure boot protected the transition between stages. OP-TEE Secure Storage and encrypted filesystem design addressed a different layer of the problem: what should remain protected after the system is already alive.

I learned to avoid treating “secure boot” as a magic umbrella. Authenticating the next image does not automatically define the storage model. It does not automatically design recovery. It does not automatically tell Linux which state it should own.

Those had to be designed separately.

Cold Boot, Warm Boot, And The Value Of A Small Register

The next class of problems came from reset behavior.

At first glance, a reset is a reset. The machine restarts, early boot code runs, and the system tries again.

In practice, a product cannot treat every reset as identical.

A cold power-on, a warm restart, a watchdog reset, and a recovery entry may all need different behavior. Some state may need to be preserved long enough for early boot code to understand why the system came back. Some paths should continue normal boot. Some paths should enter recovery. Some paths should avoid doing work that only makes sense after a full power cycle.

That is where shadow-register boot reason tracking became useful.

The idea was simple: make the boot reason visible across early stages, so the platform can distinguish cold boot and warm boot behavior before Linux is in control.

This kind of mechanism does not look glamorous in an architecture diagram. It is not a new algorithm. It is not a big subsystem.

But in bring-up, these small pieces decide whether the system behaves like a controlled product or a board that only works when an engineer is watching the serial console.

Rescue Mode Had To Be Designed, Not Hacked In

Every bring-up project eventually needs a way back in.

Images break. NAND contents become unusable. A bad bootloader change can leave the normal path dead. Early security code can reject what the developer thought was a valid image. During development, recovery is not optional.

The dangerous shortcut is to create a rescue path that bypasses the security model.

We needed the opposite: a rescue mode that could recover the board while still preserving the secure boot concept.

The practical workflow used a serial terminal path, including TeraTerm during bring-up, to transfer a secure boot loader when the normal boot path could not be trusted. The important part was not the terminal tool itself. The important part was the rule:

Recovery must return to a controlled secure boot path.

That design changed how I thought about rescue features. A rescue path is not a debug convenience. In a secure embedded product, it is part of the platform architecture.

If it is too weak, engineers cannot recover boards efficiently. If it is too permissive, it becomes the easiest way around the system’s security design.

NAND Became A Cross-Layer Contract

NAND was one of the hardest and most educational parts of the project.

From a Linux point of view, NAND can look like a kernel driver and an MTD device. But in this platform, NAND appeared in multiple places:

• early boot and bootloader reads,
• U-Boot commands and environment behavior,
• Linux MTD and filesystem usage,
• secure-world storage assumptions,
• raw diagnostics,
• stress and production validation.

That meant storage correctness could not be judged in only one layer.

If U-Boot and Linux disagree about command timing, ECC behavior, OOB layout, bad-block markers, or raw page interpretation, the system may boot in the lab and fail later in ways that are hard to reproduce.

The important lesson was that NAND is not just a driver. It is a shared contract across bootloader, kernel, secure world, filesystem tools, and validation scripts.

I later wrote a focused article about modernizing a NAND controller driver across Linux, U-Boot, and OP-TEE. This larger Secure OS project was the system context behind that lesson.

Buildroot Became The Release Engineering Backbone

Once boot and storage started to work, the next question was repeatability.

How do we build a debug image that helps engineers see what is happening?

How do we build a release image that removes unnecessary debug surface but keeps the product functional?

How do we package the kernel, device tree, root filesystem, writable storage, OP-TEE components, U-Boot, and validation utilities in a way that can be rebuilt and reviewed?

Buildroot became the backbone for that.

The debug image was intentionally tool-rich. It included stress tools, storage benchmarks, MTD/NAND tests, OP-TEE tests, tracing tools, crypto tools, memory tests, and enough utilities to inspect the system while hardware and software were still changing.

The release image had a different goal. It needed controlled packages, lower debug exposure, reduced secure-world logging, and a clean image generation path.

This separation mattered.

A debug image that can diagnose everything is not automatically a release image. A release image that hides too much too early makes bring-up slow and frustrating. Both images need to exist, and both need to be intentional.

Display And Runtime Work Pulled The Platform Upward

After a system boots and stores data, it still has to become useful.

For this project, that meant display driver integration, runtime packages, Qt-related bring-up, and application startup behavior.

Display bring-up is often described as “make pixels appear.” That is only the first milestone.

The real questions came later:

• Which layer owns display memory?
• What does user-space assume about the framebuffer or display backend?
• How does the runtime start after the root filesystem and writable storage are ready?
• Can the same image survive both engineering tests and product-like startup?
• What happens after network state changes or wake events?

This part of the project reminded me that display is not only a driver problem and not only a UI problem. It is a platform boundary problem.

The display path also connected to memory-to-memory DMA-like utility work for bitwise and blending operations. On a constrained embedded platform, CPU-side image or framebuffer operations can become too expensive. But DMA is not just “faster memcpy.” It forces the design to be explicit about alignment, stride, cache coherency, buffer ownership, and hardware-visible layout.

Those details are where a demo becomes a usable platform.

Production Delivery Was Mostly About Discipline

By the end of the project, the most important work was not any single clever fix.

It was discipline.

The platform needed:

• a numbered boot flow,
• Chain of Trust validation,
• TrustZone and OP-TEE runtime separation,
• cold/warm boot reason handling,
• a controlled rescue path,
• NAND behavior aligned across stages,
• read-only and writable filesystem roles,
• encrypted filesystem and Secure Storage boundaries,
• display/runtime startup,
• DMA utility validation,
• debug and release image separation,
• stress and benchmark coverage,
• production image hardening.

None of these pieces alone tells the whole story.

Together, they describe the difference between a board that can boot and a product that can be delivered.

Why I Still Care About This Project

My recent public work has moved more toward Edge AI and AI-assisted engineering.

But this project remains one of the strongest pieces of evidence for my embedded systems background.

Edge AI products still need the layers underneath the model. They need secure boot, reliable storage, kernel drivers, display paths, runtime packaging, recovery design, and validation discipline. A model is only useful when the device beneath it can boot, protect state, survive stress, recover from failure, and ship.

That is why I still value this project.

It taught me again that a product is not one clever component.

The product is the agreement between all of them.