edge ai, kflite, litert, ml

Learn about ‘Edge AI’ like a child

Artificial Intelligence has become part of our everyday lives, and many AI features send your data to the internet for processing. That’s where on device AI, also known as Edge AI, comes in. Instead of relying on cloud servers, the AI runs directly on your device, making phones, computers, watches, TVs, cameras, and even coffee makers and air conditioners smarter while keeping your data local.

In this article, we will learn what a machine learning (ML) model is, how it works, and how to use one in your application. You can think of an ML model as a processing unit with an input, a process, and an output. We’ll explore what the input looks like, how to feed data into a model, and how to understand its output. You’ll also see that both input and output are simply collections of numbers arranged in multidimensional arrays (called Tensors).

What’s this article not about?

• This article does not cover how to train or create ML models. Instead, it focuses on understanding how to use them.
• We focus on custom trained machine learning models. These are models that are created and trained for a specific task. This is different from using ready made AI engines such as Google ML Kit or MediaPipe.
• This is about Machine Learning models not Deep Learning. Learn the differences here:

[embed]Machine Learning vs Deep Learning Understand foundatiowww.databricks.com

What Is a Machine Learning Model?

All machine learning models are programs that have learned how to perform a specific task by studying a large amount of data. Unlike a normal program, where we write all the rules, an ML model learns those rules during training.

Once the training is finished, they become static; from that point on, it only performs predictions from new input data, also known as Inference or Interpreting.

The following hierarchy explains the relationship among AI, ML, and DL:

Artificial Intelligence (AI) Rules and Logic
    // Can not be trained during usage (static)
└── Machine Learning (ML) replaces rules with experience
    // Can be trained during usage (dynamic)
└── Deep Learning (DL) automatic learning

Every Model Has Three Parts

Every machine learning model follows the same pipeline.

Input
 ↓
Process
 ↓
Output

Input

The input is the data you provide to the model.

Depending on the model, the input could be:

• An image
• Audio
• Text
• Sensor values

However, a model cannot understand any of these directly. Before the data reaches the model, it must be converted into numbers called Tensor.

What Is a Tensor?

A tensor is simply a collection of numbers.

If you’ve worked with arrays in programming, you’ve already used something very similar. The only difference is that a tensor can have more than one dimension. It’s simply a matrix.

1D Tensor
[12, 34, 56]

2D Tensor
[
  [1, 2, 3],
  [4, 5, 6]
]

What Does Shape Mean?

The shape of a tensor describes how its numbers are organized.

It tells us the size of each dimension, but it does not change the actual values stored inside the tensor.

For example, a 1D tensor:

[12, 34, 56] -> shape is [3]

A 2D tensor:

[
  [1, 2, 3],
  [4, 5, 6]
]

This tensor has two dimensions:
2 rows
3 columns

Shape is [2,3]

What if the model receives an image?

Example for Face Detection Input

Suppose we have a face detection model that expects an RGB image with a size of 320 × 320 pixels.

The original image:

320 × 320 RGB Image

will be converted into a tensor with the shape of:

[1, 320, 320, 3]

where:

• 1 → one image (batch size)
• 320 → image height
• 320 → image width
• 3 → Red, Green, and Blue color channels

But internally, this is just one long array containing:

1 x 320 × 320 × 3 = 307,200 numbers

Check out this code example on converting a Bitmap to a Tensor.

But what if the model receives text?

Example for Text Model Input

Not every model works with images. Some models work with text instead.

For example, an English Grapheme-to-Phoneme (G2P) model converts written words into their pronunciation sounds.

A model like DeepPhonemizer receives text as input and predicts the phonemes (the basic sound units of a language).

The input might be:

"hello"

When we convert this to a Tensor related to the model input shape, it will become something like this: (each char will become a number)

[
  [
  21,
  8,
  15,
  15,
  18
  ]
]

The model receives this data as a tensor with shape of:

text [1, 96]

This means:

• 1 → one input sequence (batch size)
• 96 → the maximum number of tokens/characters in the sequence

If the text is shorter than 96 tokens, the remaining positions are usually filled with padding values.

Process

The process is the work performed inside the machine learning model.

Here, the model applies the knowledge it learned during training by performing thousands or even millions of mathematical operations on the input tensor.

Output

The output is the model’s prediction.

Depending on the model, the output might be:

• Detected faces
• Object locations
• A translated sentence

Just like the input, the output is also a tensor with a predefined shape.

Example: Face Detection Output

Suppose the model detects two faces in the image.

Instead of returning rectangles or drawing anything on the screen, it returns a tensor containing numbers such as:

[
  45, 62, 128, 180, 0.98,
  185, 74, 276, 192, 0.95
]

For a real face detection model, the output tensor might have the shape:

[1, 2500, 5]

This means:

• 1 → The model processed one input (image) (batch size).
• 2500 → The model returned 2,500 possible face predictions.
• 5 → Each prediction contains 5 values: x, y, width, height, and confidence.

You can visualize the output tensor like this:

[
  [                     ← Batch (1 image)
    [45, 62, 128, 180, 0.98],   ← Prediction 1
    [185, 74, 276, 192, 0.95],  ← Prediction 2
    [ ... ],                    ← Prediction 3
    ...
    [ ... ]                     ← Prediction 2500
  ]
]

in code, this will look like:

val output = Array(1) {
    Array(2500) {
        FloatArray(5)
    }
}

so the shape of our model output is [1,2500,5].

you can access the first prediction like this:

val firstPrediction = output[0][0]

val x = firstPrediction[0]
val y = firstPrediction[1]
val width = firstPrediction[2]
val height = firstPrediction[3]
val confidence = firstPrediction[4]

Most of these 2,500 predictions are not actual faces. Many will have very low confidence scores, such as 0.02 or 0.15. You must remove low confidence predictions and keep only the best ones. This is also known as Thresholding.

But what if the model produces text?

Example for Text Model Output

Consider the input we earlier gave in the input section. The model then processes that input tensor and produces an output tensor with a shape of:

[1, 96, 64]

This means:

• 1 → one output sequence
• 96 → one prediction for each position in the input sequence
• 64 → 64 possible phoneme classes

Here, the output is called logits. Logits are raw prediction scores before they are converted into probabilities.

For example, for the first character position, the model might output:

[
  0.2,
  -1.5,
  3.8,
  ...
]

After converting these predictions, the output might become:

hello
 ↓ G2P
HH AH L OW

What Is Normalization?

Normalization is the process of changing numerical values into a specific range or format that the machine learning model expects. Almost 90% of errors will hover around this section.

Normalization can happen in two places:

1. Input → Before sending data into the model. (converting images/text to tensor). Also known as preprocessing.
2. Output → After receiving data from the model, converting it back into a useful format. (convert tensor back to text or a bounding box image). Also known as postprocessing.

Using/Running ML Models in Your Software

Now that we understand how data moves through a machine learning model, the next step is running these models inside an application.

For edge AI, models are usually converted into formats optimized for mobile and embedded devices, such as tflite, onnx, or pt.

One of the most common libraries is TensorFlow Lite, now evolving under the name LiteRT.

LiteRT official website

Another popular option is PyTorch Runtime or ExecuTorch:

ExecuTorch official website

Thank you for reading.