EDGE AI-ML: Fundamentals, Architectures, and Applications

The most common way to deploy ML models right now is by hosting them on the cloud. So, let’s say you are driving a self-driving car, and it’s connected to an AI model on the cloud. You are sitting and watching the AI do wonders on the road, and suddenly, due to something like traffic or network buffering, the inference from the AI experiences some lag, and your car suddenly decides to drive into oncoming traffic. This is the reason why we need Edge AI-ML.

Edge ML is a concept that brings the capability of running ML models locally on edge devices (without sending data to the cloud), so that there is minimal lag in getting inferences from the model. An edge device is any hardware (like a microcontroller, Raspberry Pi, or even a web browser) that is close to the user, in our case that is the hardware installed in your self-driving car. This enables real-time data processing, supports privacy, and reduces latency and bandwidth costs.

Training the models is still usually done in the cloud, where large datasets and powerful GPUs are available. Inference, the process of using a trained model to make predictions on the other hand happens at the edge.

Most edge devices have limited memory, processing power, and energy capacity compared to the cloud. Therefore Edge devices cannot handle the massive computational load of running large, uncompressed neural networks in their raw form. In the cloud, powerful GPUs and TPUs can run complex models with millions (or even billions) of parameters without breaking a sweat. But on an edge device, like the onboard computer in a self-driving car every megabyte of memory and every millisecond of processing time counts.

Running a large model directly on edge hardware would quickly drain power, overheat components, and cause unacceptable delays in inference. To use the performance of these large models while also accommodating the limitations of edge devices we use model compression techniques like quantization and pruning. These methods shrink the model’s size, reduce the number of operations needed, and make it more efficient, all while trying to preserve as much accuracy as possible.

MODEL COMPRESSION

Model compression reduces the size and computational complexity of neural networks without significantly compromising accuracy. Two of the main model compression approaches are:

1. Pruning

Model pruning refers to the act of removing unnecessary parameters like weights, neurons, or connections from a deep learning neural network. Neural networks usually have many redundant connections. Pruning identifies which parts of the model are less important and removes them. This creates a sparser model that still performs well.

There are mainly 2 types of pruning techniques based on when the pruning process occurs in relation to the training of the model, train-time pruning which integrates pruning during training phase and post-training pruning which applies after model training is complete. To keep performance stable, post-training pruned models are sometimes retrained or guided by the original model (a process called knowledge distillation).

Most widely adopted strategy is magnitude based post-training pruning approach where weights with the smallest absolute value are eliminated. The underlying assumption is that parameters with values close to zero contribute minimally to the network’s output and can therefore be discarded with limited impact on overall performance.

After pruning, the network typically undergoes a fine-tuning phase to allow the remaining parameters to adapt and recover potential accuracy loss. This makes magnitude-based post-training pruning a practical and effective method for reducing model size while preserving predictive capability, especially for deployment on edge devices. The following is an example of using TensorFlow Model Optimization Toolkit to implement post training magnitude based pruning along with fine tuning.

import tensorflow as tf
from tensorflow import keras
import tensorflow_model_optimization as tfmot

# 1) TensorFlow Model Optimization Toolkit (TFMOT), 
# provides built-in APIs for magnitude-based pruning with 
# automatic mask management and pruning schedules.

model = keras.models.load_model("my_trained_model.keras")

# 2) Apply magnitude-based pruning
#    Here we wrap the model with a pruning schedule.
#    polynomial_decay: gradually increases sparsity during fine-tuning

prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude

pruning_params = {
    "pruning_schedule": tfmot.sparsity.keras.PolynomialDecay(
        initial_sparsity=0.0,
        final_sparsity=0.8,    # prune to 80% sparsity
        begin_step=0,
        end_step=1000
    )
}

pruned_model = prune_low_magnitude(model, **pruning_params)

# 3) Compile and fine-tune
#    During fine-tuning, pruning masks are applied automatically.

pruned_model.compile(optimizer=keras.optimizers.Adam(1e-4),
                     loss="sparse_categorical_crossentropy",
                     metrics=["accuracy"])

callbacks = [
    tfmot.sparsity.keras.UpdatePruningStep(),   # updates pruning masks
    tfmot.sparsity.keras.PruningSummaries(log_dir="./logs")
]

pruned_model.fit(x_train, y_train,
                 batch_size=256, epochs=2,
                 validation_split=0.1,
                 callbacks=callbacks)

# 4) Strip pruning wrappers
#    Removes the pruning logic so the model is ready for deployment.

final_model = tfmot.sparsity.keras.strip_pruning(pruned_model)
final_model.save("my_model_pruned_finetuned.keras")

Pruning neural networks can also improve overall performance by removing parameters that introduce bias or noise into the model, making it more generalized.

2. Quantization Strategies

Quantization reduces the precision of model weights and activations from floating-point to lower precision formats (typically integers). This technique can reduce model size by 75% (float32 to int8) while maintaining acceptable accuracy.

TensorFlow Lite (now also known as LiteRT) is Google’s lightweight, open-source framework specifically designed for on-device machine learning inference. Unlike regular TensorFlow which handles both training and inference, TensorFlow Lite focuses exclusively on running pre-trained models efficiently on resource-constrained devices like smartphones, tablets, embedded systems, and microcontrollers.

TensorFlow Lite implements quantization through its converter, which transforms full-precision models into optimized formats during the conversion process.

TensorFlow Lite’s quantization approach:

1. Conversion-time optimization: Quantization happens when converting from TensorFlow to TFLite format
2. Multiple precision options: Supports int8, int16, float16 quantization
3. Automatic range estimation: Uses representative datasets to calibrate quantization parameters
4. Hardware-aware optimization: Tailors quantization for specific accelerators

Application example of dynamic range quantization (Easiest to implement):

import tensorflow as tf

# Convert model with basic quantization
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quantized_model = converter.convert()

# Save the quantized model
with open('quantized_model.tflite', 'wb') as f:
    f.write(tflite_quantized_model)

These optimization techniques make it possible for powerful models to run efficiently on limited hardware, but the future of Edge AI isn’t just about making models smaller, it’s also about making them smarter and more adaptive. This is where Interactive Edge AI comes in.

INTERACTIVE EDGE AI

Interactive Edge AI represents the next evolution of artificial intelligence. Unlike traditional static AI deployment, interactive edge systems don’t just run inference but actively learn, adapt, and improve their models locally while maintaining user privacy and enabling real-time personalization.

Interactive Edge AI is seeing rapid growth with two prominent emerging approaches: on-device learning and federated learning.

1. On-device Learning

On-device learning empowers smart devices such as smartphones, wearables, and IoT sensors, to adapt and update their machine learning models locally, based on user-specific data, rather than relying solely on periodic cloud updates. This approach maintains user privacy (since raw data never leaves the device) and provides real-time personalization, adapting to changes in user behavior, environment, or preferences.

A very popular application is smartphone keyboards, they use on-device training to adapt language models for each user. This ensures keyboard suggestions and auto-complete improve with local usage, respecting privacy since the input data remains on the device. Here’s the simplified conceptual workflow of on-device learning using TFLite.

import tensorflow as tf

# Load the pre-trained TFLite model (with training signature enabled)
interpreter = tf.lite.Interpreter(model_path="keyboard_model.tflite")
train_fn = interpreter.get_signature_runner("train")
infer_fn = interpreter.get_signature_runner("infer")

# Gather local typing data
for batch in local_text_batches:
    train_fn(input_text=batch['features'], target_words=batch['labels'])

# Model gradually adapts to new vocabulary and phrases unique to the user

# Optional: Save the fine-tuned weights back into a file
# (only works if your model has checkpointing enabled)
updated_model = interpreter.get_tensor(interpreter.get_output_details()[0]['index'])

2. Federated Learning

Federated learning is a distributed machine learning approach where multiple edge devices collaboratively train a shared global model, all while keeping individual training data local. Devices periodically download the latest global model, perform training using their private data, and send only the model updates (e.g., gradients or weights) back to a central server. This server aggregates the updates from hundreds or thousands of devices to improve the global model.

The benefits are twofold: first, user privacy is preserved because raw data (keystrokes, photos, health stats) never leaves the device, and second the global model becomes far more representative and robust, capturing the diversity of its user base. Federated learning is already a foundation for powerful AI features in consumer tech such as mobile voice assistants, healthcare systems, and smart home devices.

A common use-case is improving the “Hey Google” or “Hey Siri” wake-word detection collaboratively across millions of devices.

import tensorflow as tf
import tensorflow_federated as tff

# Define a model-building function (e.g., keyword detection CNN)
def model_fn():
    model = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(input_shape,)),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(num_classes, activation='softmax')
    ])
    return tff.learning.from_keras_model(
        model,
        input_spec=sample_batch,  # representative user data spec
        loss=tf.keras.losses.SparseCategoricalCrossentropy(),
        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()]
    )

# Federated learning process (simplified)
iterative_process = tff.learning.build_federated_averaging_process(model_fn)
state = iterative_process.initialize()

for round_num in range(num_rounds):
    # Each client (device) trains locally then uploads updates
    state, metrics = iterative_process.next(state, federated_train_data)
    print('Round', round_num, 'metrics:', metrics)

With this setup, devices with different acoustic environments (various accents, backgrounds) help collectively improve detection for everyone without anyone’s audio recordings being directly shared with the cloud.

Edge AI-ML is unlocking new possibilities for creating faster, more adaptive, and privacy-focused applications. Leveraging lightweight deployment frameworks such as TensorFlow Lite, TensorFlow Federated, and TensorFlow.js, developers can now run AI models directly on edge devices enabling powerful, real-time, and resilient solutions even in resource-constrained environments.

As these tools and techniques advance, the Edge AI industry is evolving toward broader adoption of decentralized, low-latency inference and seamless hybrid edge–cloud architectures. Developers are leveraging cost efficient, on-device processing to power real time applications from smartphones to industrial IoT with increasing support for interoperability standards and domain-specific optimizations.

A Step Further

This was just the overview of the possibilities of applications using Edge AI-ML, the actual rabbit hole goes way deeper. Here are some resources where you can start exploring:

• Edge Impulse — Introduction to Edge AI Course
• TensorFlow Lite Official Documentation
• DeepLearning.AI — Introduction to On-Device AI Course

Thank you for taking the time to read, have fun Edging your models!😊