Why Embedded AI Design Is Really About Platform Choice

Why Embedded AI Design Is Really About Platform Choice I made the case in the earlier piece, “What Embedded AI Design Really Means,” that embedded artificial intelligence is essentially a systems design issue. Intelligence in these settings emerges from the interaction of hardware, software, data, and timing constraints.

That perspective naturally raises another question.

If embedded AI is a system, where exactly does that intelligence run? The answer is not trivial. One of the most important choices in many embedded AI projects is selecting the hardware platform that will house the intelligence before any model is trained.

The Platform Question Comes Before the Model In traditional AI workflows, model design usually takes centre stage. Engineers focus on architectures, training datasets, and benchmark accuracy.

Embedded environments reverse this order.

Hardware constraints determine the computational resources available for inference. Memory capacity determines how large a model can be. Power budgets determine how frequently inference can run. Real-time requirements determine how quickly a decision must be produced.

In other words, platform choice defines the design space in which embedded AI can exist.

A neural network that performs well in the cloud may require hundreds of megabytes of memory and billions of operations per second. Many embedded devices, by contrast, operate with only 256 KB–1 MB of RAM and processors running between 80–200 MHz.

The gap between these environments is not incremental. It is architectural.

Layered architecture diagram

Two Typical Embedded AI Platforms Microcontrollers and single-board computers are the two main hardware platforms used in most embedded AI implementations.

MCUs, or microcontrollers Deterministic control and incredibly low power consumption are features of microcontrollers. They rule settings like wearables, industrial monitoring systems, and Internet of Things sensors.

Standard configuration:

• Processing speed: 50–200 MHz
• 1MB to 128 KB of RAM
• 512 KB – 8 MB of flash storage
• Tens of milliwatts of power

These constraints need the aggressive optimisation of models. Commonly employed methods include quantisation, pruning, and TinyML frameworks like TensorFlow Lite Micro.

For example:

• INT8 quantisation can reduce model size by 4×
• Inference speed can improve 2–3× on ARM Cortex-M processors
• Memory usage drops significantly without a large accuracy loss

Despite these constraints, microcontrollers can still support tasks such as:

• keyword spotting
• vibration anomaly detection
• gesture recognition
• simple predictive maintenance

Single Board Computers (SBCs) Single-board computers provide substantially more computational capability and operate closer to general-purpose computers.

Typical requirements consist of:

• CPU speed(frequency): 1–2.5 GHz
• RAM: 1–8 GB
• Operating system: Linux
• optional GPU or neural accelerator

Heavy workloads can be supported by platforms like the Raspberry Pi or NVIDIA Jetson, including:

• computer vision
• multi-sensor fusion
• object detection

But there are trade-offs associated with these advantages:

• higher power consumption (5–15 watts)
• non-deterministic timing due to OS scheduling
• higher system cost

Comparing Embedded AI Platform

Five Hardware Evaluations Prior to Selecting an Embedded AI System

Engineers usually consider a number of practical limitations before committing to a hardware platform. These tests help determine whether the expected AI workload can be supported by an embedded device.

1. Memory Availability AI models consume memory in three ways:

• model parameters
• intermediate tensors
• input buffers

Even after compression, a basic CNN model could use hundreds of kilobytes. The design must change if the platform only offers 256 KB of RAM.

2. Compute Capability The processor must support the number of operations required per inference cycle.

For example:

• A TinyML model may require 5–20 million operations per inference
• A Cortex-M4 at 120 MHz may complete this within tens of milliseconds

If inference time exceeds system deadlines, the design fails regardless of model accuracy.

3. Power Budget Many embedded devices operate on batteries or energy-harvesting sources.

Key questions include:

• How often can inference run?
• Can the processor remain active continuously?
• What is the expected battery lifetime?

Even small increases in compute load can significantly reduce device lifetime.

4. Real-Time Requirements In control systems, decisions must occur within strict deadlines.

For example:

• Industrial control loops may require responses within 10–50 milliseconds
• It may be necessary to monitor machinery almost constantly in order to discover anomalies.

If the inference pipeline cannot meet these deadlines, the system may become unsafe or ineffective.

5. Connectivity and Data Flow Not all embedded systems can rely on stable internet connectivity.

Designers must consider:

• whether inference must occur fully on-device
• how often data is transmitted
• how models will be updated over time

These factors influence the choice between fully embedded intelligence and hybrid edge-cloud architectures.

Five Hardware Checks for Embedded AI Platforms

MATLAB-Based Embedded AI Prototyping Before implementing models on real hardware, engineers typically prototype and simulate embedded AI systems using development tools like MATLAB and Simulink.

MATLAB is useful particularly for embedded AI prototyping since it allows engineers to:

• simulate sensor signals
• design signal processing pipelines
• train and evaluate machine learning models
• generate embedded code for microcontrollers

For example, vibration data from an industrial pump can be analysed in MATLAB using signal processing tools to extract features such as:

• frequency spectrum
• RMS vibration energy
• statistical anomalies

These algorithms can be exported using MATLAB Coder or Embedded Coder after they have been verified, enabling them to operate on microcontrollers or edge processors.

This approach helps engineers evaluate:

• inference latency
• memory requirements
• algorithm performance under realistic constraints

before committing to hardware deployment.

A Useful Illustration: Predictive Maintenance

Consider an industrial pump monitoring system.

A vibration sensor continuously collects data about mechanical behaviour. The aim is to detect early abnormalities that might indicate mechanical imbalance or bearing wear.

Every few milliseconds, the embedded AI system may run a lightweight anomaly detection model if the monitoring node has a microcontroller. The model could operate on compressed vibration features and trigger alerts when unusual patterns appear.

If the same system uses a single board computer, more complex models could analyse full vibration spectrograms using convolutional neural networks.

Both systems perform predictive maintenance. The difference lies in how intelligence is implemented, and that difference is largely determined by the hardware platform.

Why Platform Choices Are More Important Than You Might Think

Why Platform Choices Are More Important Than Anticipated Many embedded AI initiatives struggle because the hardware platform cannot support the necessary processing, latency, or power profile, rather than because the model fails.

Common issues include:

• models exceeding available memory
• inference times missing real-time deadlines
• battery drain caused by heavy compute workloads

Selecting the right platform early reduces risk and allows engineers to design models that align with real hardware constraints.

Platform Choice as a Design Philosophy Embedded AI design ultimately reverses the traditional AI workflow.

Instead of starting with the most powerful model and attempting to deploy it later, engineers begin with a simpler question:

What hardware environment will this intelligence live in?

Models, data pipelines, and inference techniques can be created in accordance with the platform once it has been established.

Therefore, the goal of embedded AI shifts from optimising algorithmic complexity to creating intelligence that is appropriate for the physical environment in which it functions.

Layered architecture diagram: Embedded Devices → Edge Gateway → Cloud Platform

Annotations:

• Local inference
• Data aggregation
• Cloud training and model updates

Looking Ahead Once the platform is selected, another challenge emerges: how to design AI models that remain accurate while fitting within extremely limited hardware resources.

This challenge has given rise to the field often known as TinyML, where models are redesigned and compressed to operate on devices with only kilobytes of memory.

Understanding how these optimisations work is the next step in the journey of embedded AI design.