Engineering the End-to-End Hybrid AI Stack on Android with LiteRT and Qualcomm QNN (Part 1)

The Advent of Edge

The proliferation of connected devices has fundamentally shifted the center of data creation from centralized servers to the perimeter of the network which is the edge. Phones, sensors, and IoT devices constantly generate massive amounts of data. This transition has fueled a global edge computing market projected to surge from $15.96 billion in 2023 to $216.76 billion by 2032. The drive to process this data locally is paramount, leveraging the computational power of modern smartphones to enable offline processing, significantly reduced latency for users, and improved privacy by handling sensitive information directly on the device. The singular goal of this architectural shift is to make every product, application, and device inherently more intelligent.

The Evolution of Android’s On-Device AI Stack

The demand for real-time, private, and power-efficient AI has driven a decade of innovation in the Android ecosystem. This imperative enables user experiences previously impossible due to cloud latency, such as always-on assistants and real-time AR effects, while ensuring privacy by processing data locally. This shift necessitated a complete rebuilding of the underlying Android software stack, moving from a single runtime to a sophisticated, multi-framework ecosystem.

Phase 1: The Foundation and the Quest for Speed

The on-device AI story began in 2017 with the introduction of TensorFlow Lite (TFLite). TFLite was conceived as a lightweight, mobile-first inference runtime optimized for “fast and small” execution. It introduced the optimized .tflite model format, which utilizes FlatBuffers for faster loading and a smaller memory footprint. Crucially, TFLite was integrated from day one with the new Android Neural Networks API (NNAPI), enabling access to hardware accelerators.

Over the following six years (2017–2023), TFLite matured significantly, transforming into a robust framework. Key enhancements included performance kernels like XNNPack and RUY for highly optimized CPU operations, alongside sophisticated model efficiency tools such as quantization (INT8, float16) and pruning to reduce model size.

However, as ML models grew more complex, reliance on the CPU became a critical bottleneck, making real-time applications demanding just a few milliseconds of inference time unattainable without significant battery drain. This spurred a shift to Heterogeneous Compute, leveraging not just the CPU, but also the GPU and dedicated Neural Processing Units (NPUs). This hardware revolution was transformative, with the integration of dedicated NPUs delivering unprecedented performance gains: up to 100x faster speed-ups over CPU and up to 10x faster inference compared to GPU for many models.

Unifying the Silicon: NNAPI and Vendor Acceleration

To manage the fragmented, multi-vendor hardware landscape, the architecture relies on abstraction layers. The NNAPI (Android 8.1+) provides the fundamental OS-level API, allowing applications to request hardware acceleration without needing vendor-specific code. Directly below the runtime, LiteRT Delegates are pluggable backends that execute specific portions of a model on dedicated hardware, such as the GPU Delegate or the CPU Delegate (XNNPack).

This system ensures maximum performance across the heterogeneous hardware ecosystem. Silicon vendors are integrated through these delegates: Qualcomm devices utilize the Hexagon NPU/DSP via the unified Qualcomm AI Engine Direct accelerator. Similarly, MediaTek uses the NeuroPilot APU, and Samsung uses the Exynos NPU, primarily accessed via NNAPI. The modern stack relies on the Qualcomm AI Hub which natively supports running smaller 1–7B models locally, simultaneously providing developers with tools to offload larger models to the cloud.

Phase 2: Convergence and Abstraction with LiteRT and MediaPipe

Running a model quickly is only part of the challenge; historically, complex tasks like hand tracking required building pre-processing and post-processing pipelines manually. This included handling camera frames, normalization, tensor conversion, interpreting raw output, and drawing results — a repetitive, error-prone process.

To abstract this complexity, MediaPipe, Google’s open-source, cross-platform framework for building perception pipelines — was developed. MediaPipe is widely used for processing and analyzing video, audio, and sensor data, supporting both dataset preparation and ML inference pipelines. MediaPipe evolved to offer the Tasks API (2022+), providing a library of ready-to-use, task-specific solutions. The Tasks API handles all the logic from input data to structured, usable results, reducing implementation from hundreds of lines of boilerplate code to a handful of declarative lines. Importantly, these pipelines are end-to-end accelerated on CPU, GPU, and TPU, fully abstracting hardware details from the developer.

This journey of performance and simplification converged in 2024 with a key rebranding: TFLite was renamed LiteRT (‘Lite Runtime’). This signifies its evolution into a high-performance, multi-framework inference engine with explicit native support for models from TensorFlow, PyTorch, and JAX. The new LiteRT ecosystem maintains full backward compatibility with the .tflite format while supporting the next wave of models. Developers now utilize the latest LiteRT and LiteRT-LM API for code integration.

The Generative AI Frontier

The LiteRT stack is built not just for classic ML models, but also for the latest generation of Generative AI, including LLMs and diffusion models. This capability is enabled by:

1. LiteRT-LM: A specialized runtime and .litertlm format designed for efficiently deploying large language models on mobile and desktop.

2. MediaPipe GenAI APIs (2024+): High-level APIs for generative tasks, including the MediaPipe LLM inference API. This API enables running LLMs such as Gemma2 directly on Android.

3. Model Conversion: To deploy a base LLM like Gemma, it must be converted into the format supported by the MediaPipe LLM inference API using the MediaPipe toolkit. For models outside of Google Edge AI’s default supported set, the converter.ConversionConfig method must be used to ensure compatibility.

The Qualcomm QNN Factor: Maximizing NPU Performance via the LiteRT Stack

The efficiency of running sophisticated models, particularly Large Language Models (LLMs), on mobile devices dictates the necessity of specialized hardware acceleration. While running LLMs exclusively on the CPU is slow, and GPUs (often utilizing OpenCL) offer strong performance but consume significant power, dedicated Neural Processing Units (NPUs) are specifically designed for high-throughput, power-efficient AI workloads. The integration of these dedicated NPUs into modern Android devices has delivered unprecedented performance gains, achieving up to 100x faster speed-ups over CPU and up to 10x faster inference compared to GPU for many models.

QNN and the Hexagon Architecture

Qualcomm’s solution for unlocking this performance on Android devices is the Qualcomm Neural Network (QNN) runtime, which enables the offloading of LLM computation to the Hexagon NPU/DSP. Qualcomm devices utilize this hardware via the unified Qualcomm AI Engine Direct accelerator.

Key components critical for maximizing throughput on Qualcomm NPUs include:

• HTP Backend: The Hexagon Tensor Processor (HTP) accelerates key matrix operations central to large neural networks.
• Quantization: NPUs operate most efficiently on low-precision integer formats. NPU-friendly models must undergo advanced quantization, typically into INT8 or INT4 formats. For instance, running optimized LLMs like Gemma 3n generally requires the INT4 format to fit the model into the limited mobile memory and leverage the NPU units effectively. The LiteRT-LM specialized runtime is designed for deploying these efficiently quantized LLMs on mobile.

Integrating QNN Through LiteRT Delegates

The flow from the application down to the Qualcomm silicon is managed by the LiteRT runtime (the successor to TFLite). LiteRT achieves deep vendor-specific optimization through Hardware Delegates.

When an application intends to run an LLM on the NPU, selecting the appropriate backend (e.g., Backend.NPU) activates the following technical flow:

1. Driver Check: LiteRT checks for the presence of necessary QNN drivers, specifically libQnnHtp.so, which facilitates communication with the Hexagon NPU/DSP.
2. Model Conversion and Graph Preparation: LiteRT converts the model graph (which is already in a format compatible with the MediaPipe LLM inference API for LLM deployment) into a QNN-compatible format.
3. Execution Push: The computation for the model is pushed directly to the Hexagon DSP for accelerated inference.
4. Fallback Mechanism: Due to the complexity of the hardware ecosystem, if any part of the model or the overall configuration is incompatible (e.g., missing operator coverage or specific firmware limitations), LiteRT must silently fall back. This fallback typically routes execution to the CPU (often utilizing highly optimized kernels like XNNPack) or the GPU.

Challenges of Consistency and the Hybrid Default

While NPU acceleration is crucial for enabling local AI, the practical deployment of vendor-specific solutions like QNN introduces several challenges related to Hardware Fragmentation and consistency. QNN’s effectiveness is highly sensitive to:

• Model Export Formats: The specific way a model is converted and quantized.
• Delegate Versions and Firmware Differences: Inconsistencies across various device generations and OS updates.
• Operator Coverage: Whether the NPU supports every specific mathematical operation required by the model graph.

Because of this sensitivity and the challenge of building a consistent, reliable experience across fragmented hardware, the sources suggest that for a general-release Android application targeting a wide range of devices, the GPU (OpenCL) is currently the most stable backend for complex models.

However, the core Hybrid AI architecture is built to handle this fragmentation. The LiteRT stack utilizes delegates to enable NPU switching for devices that have compatible binaries. Furthermore, the LiteRT-LM runtime is engineered with a built-in fallback mechanism. It automatically checks NPU constraints and runs smaller LLMs (like 1–3B parameter models) locally, but when the limits of consistency, scale, or endurance are hit, it transparently triggers a fallback to call cloud inference (e.g., via the Gemini API). This strategic approach of running locally what makes sense and offloading what requires scale ensures that the application delivers high performance and reliability across the entire Android ecosystem.