Must-Know Tips: How vLLM-Plugin-FL Unlocks Heterogeneous Computing Power via Unified Framework & Elastic Operators

As large model inference workloads grow rapidly, vLLM has become one of the most widely used inference frameworks in the industry. Meanwhile, the multi-chip ecosystem is also accelerating toward greater maturity, and a real-world problem has gradually surfaced: the same vLLM codebase is being split into more and more “chip-specific versions.” Accelerator cards from different vendors often need to maintain independent vLLM branches or plugins; every upstream model update forces adaptation work to be repeated across multiple branches; and when users attempt heterogeneous inference deployment, they are often forced to maintain multiple runtime environments simultaneously, driving up overall costs significantly.

The core issue is not vLLM itself, but that vLLM’s hardware-related capabilities have not been truly “decoupled.” The unified multi-chip vLLM Plugin (vLLM-plugin-FL) proposed by FlagOS attempts to solve this problem in a more engineering-oriented way: keep vLLM as a single version, and push hardware differences down into the operator layer.

Design Principle: A Virtual Backend with “Unified Framework Layer, Operator-Layer Dispatch”

vLLM-plugin-FL is a cross-chip extension plugin built by FlagOS for the vLLM inference framework. Built on four core FlagOS components — the FlagGems unified operator library, the FlagTree multi-hardware compiler, the FlagCX distributed communication library, and the unified scheduling abstraction layer — it creates a two-layer architecture with a unified framework layer and on-demand operator-layer dispatch, fundamentally addressing the fragmentation pain points of multi-chip adaptation.

The traditional approach to solving multi-chip adaptation is to add if-else branches inside the vLLM framework for each chip. This approach usually leads to bloated, hard-to-maintain code. Worse still, when the upstream vLLM version evolves, every downstream vendor branch has to face painful conflict resolution. vLLM-plugin-FL offers a completely different and more elegant approach: connect FlagOS to vLLM as a virtual backend in an Out-of-Tree manner.

From the overall architecture perspective, its core design logic can be summarized in ten Chinese characters: “unified framework layer, operator-layer dispatch.”

In this system, the upper-layer vLLM framework logic remains completely consistent and is unaware of any underlying hardware differences. All hardware-related capabilities are uniformly scheduled by FlagOS at the operator layer.

vLLM-plugin-FL uses an Out-of-Tree approach to connect FlagOS to vLLM as a virtual backend. Its core goals are: do not modify upstream vLLM; do not introduce hardware-related logic into the framework layer; and extend backend capabilities through the plugin mechanism.

In this way, a single vLLM version can run seamlessly across multiple types of chips. At the same time, the design naturally supports heterogeneous inference, allowing execution paths to be flexibly scheduled across different hardware in order to maximize the compute capability of each type of chip.

Looking at the evolution of vLLM, its multi-backend support has gone through three stages: code-level hardware abstraction, the EntryPoint plugin mechanism, and Out-of-Tree plugins. The FlagOS solution belongs to the third stage. Its defining characteristics are: using FlagOS as a unified virtual backend, confining hardware differences to the plugin layer, and preserving upstream vLLM integrity. In essence, this design replaces the multi-branch fork maintenance model with a unified plugin system.

Based on this workflow, multi-chip users gain three key benefits:

1.Single vLLM version, identical runtime experience: completely eliminate the maintenance nightmare of “one branch per chip” and significantly reduce the cost of syncing upstream updates.

2.Maximize heterogeneous compute power: through flexible operator dispatch, different chips can run their best operator implementations under the same framework, making full use of the strengths of each chip.

3.Simplified deployment and maintenance: users only need to install the corresponding plugin and FlagOS components to obtain a consistent inference experience on any supported chip.

At the operator layer, the FlagOS + Triton mechanism can also provide a basic runnable path, ensuring that different chips have a unified execution capability.

The diagram above shows the complete path from vLLM model code to multi-chip execution: the framework layer provides unified scheduling, while the operator layer automatically dispatches across three types of backends through OpRegistry + SelectionPolicy, ultimately executing on specific chips.

Core Architecture: Deep Dive into the “Multi-Chip Operator Dispatch Architecture”

If the “virtual backend” is the design philosophy of vLLM-plugin-FL, then the multi-chip operator dispatch architecture is the “central nervous system” that carries this philosophy. It consists of three core modules:

Unified Dispatch Mechanism: A Standardized “Bus” for Operator Integration

To allow operators from different sources to be managed and invoked uniformly, vLLM-plugin-FL designs a sophisticated dispatch mechanism. Its core components include OpManager (dispatch manager), OpRegistry (operator registry), SelectionPolicy (selection strategy), and BackendImplKind (backend type definition). The overall execution flow is as follows:

1.The operator call enters the dispatch layer.

2.OpRegistry reads the registered implementations.

3.SelectionPolicy selects the backend.

4.OpManager executes the final bound function.

Currently, vLLM-plugin-FL supports three standard backend implementations:

Click the image to view the full spreadsheet.

When an operator is called, the dispatch mechanism searches for available implementations by priority from high to low: it first tries FlagGems Triton operators; if unavailable, it falls back to vendor-native operators; finally, it falls back to PyTorch reference implementations. This layered fallback design both guarantees cross-chip generality and leaves room for vendors to perform deep optimization.

Operator-Level Auto-Tuning: Bridging the Performance Gap in the “Last Mile”

Merely being able to run is not enough; the goal of vLLM-plugin-FL is to run optimally. For this purpose, it builds an operator-level auto-tuning engine: the AutoTune Engine. Its workflow is:

• First, it automatically traverses all operators in the model and, based on the FlagGems, Vendor, and Reference backends, constructs a search space with multiple configurations for each operator.
• Then, it uses end-to-end model throughput as the evaluation metric and automatically tests the impact of different operator configurations on overall performance.
• Finally, through heuristic search or Bayesian optimization, it automatically finds the operator and configuration combination that maximizes end-to-end performance, and makes the result available as a lookup table for subsequent inference.

This mechanism optimizes not the peak performance of a single operator, but end-to-end throughput. Even if an operator is fast in a microbenchmark, it should not be selected if it slows down the critical path in the overall inference pipeline.

Practical Validation: Qwen3-Next Inference Performance Optimization

The effectiveness of this mechanism has been fully validated on the Qwen3-Next model, bringing real performance improvements:

Step 1: Identify time-consuming operators. Through timeline analysis of model operators, the FlagOS team found that the moe_align_block_size operator in the FlagGems general-purpose operator library accounted for as much as 21.7% of end-to-end runtime, while the CUDA version vllm::moe::moe_align_block_size_kernel accounted for only 2.5%. This operator clearly became a performance bottleneck.

Click the image to view the full spreadsheet.

Step 2: Auto-tune and select the optimal implementation. After searching, the AutoTune Engine selected the CUDA implementation for the moe_align_block_size operator. This single adjustment delivered a 3%–4.6% end-to-end performance improvement, demonstrating the clear benefits of auto-tuning for end-to-end performance.

Step 3: Joint framework and operator optimization. For FlagGems operators with high runtime shares, the team further performed targeted optimizations:

• Rewrote the GDN operator in the decoding stage and increased parallelism to accelerate computation.
• Removed unnecessary clone operations before operator calls at the framework layer.
• Optimized FlagGems operators with vectorized data reads and used TLE shared memory functionality to accelerate computation.

Step 4: FlagCX-CA communication operator optimization. In multi-card scenarios, communication operators are also on the critical path. FlagCX’s ncclAdator supports customAllReduce based on the NCCL device API and supports LSA/Multimem primitives:

• Small messages (≤512K): localAllReduce (MultimemSum + LSAStore) achieves performance on par with vLLM cross_device_1stage (IPC-based packed_reduce).
• Small-to-medium messages (512K, 8M]: interleavedAllReduce (MultimemSum + MultimemStore) delivers a modest performance improvement over vLLM cross_device_2stage (IPC-based reducescatter + allgather).

After this combination of optimizations, we achieved end-to-end inference for Qwen3 and Qwen3-Next on single-card and multi-card setups through the FlagOS unified backend, reaching 100% Triton operator replacement. On NVIDIA hardware, Qwen3-Next throughput can reach up to 99.6% of native performance. This means that unified cross-chip inference capability is gained without sacrificing performance.

Hands-On Integration: How Can Chip Companies Quickly Connect to vLLM-plugin-FL?

For chip vendors or developers, connecting their own hardware to the vLLM-plugin-FL ecosystem is a standardized and low-cost process. The overall workflow is mainly divided into two parts: adaptation and usage.

Vendor Operator Adaptation (Vendor Backend Integration)

The core of adaptation is to implement a Vendor Backend and register the vendor-optimized operators into the dispatch system. The whole workflow is clear and modular:

1. Create the Backend class

Create a vendor backend class in vllm_fl/dispatch/backends/vendor/<vendor_name>/<vendor_name>.py. Inherit from Backend, implement availability checks, and provide entry points for each operator:

Python

from ...base import Backend
class <VendorName>Backend(Backend):
    _available = None
    @property
    def name(self) -> str:
        return "<vendor_name>"
    @property
    def vendor(self) -> str:
        return "<vendor_name>"  # 必须与目录名一致
    def is_available(self) -> bool:
        """检查厂商库是否可用"""
        if <VendorName>Backend._available is None:
            try:
                import <vendor_library>  # 导入厂商库
                <VendorName>Backend._available = True
            except ImportError:
                <VendorName>Backend._available = False
        return <VendorName>Backend._available
    def silu_and_mul(self, x):
        from .impl.activation import silu_and_mul_<vendor>
        return silu_and_mul_<vendor>(x)
    def rms_norm(self, x, residual, weight, epsilon):
        from .impl.normalization import rms_norm_<vendor>
        return rms_norm_<vendor>(x, residual, weight, epsilon)
    def rotary_embedding(self, query, key, cos, sin, position_ids, is_neox_style):
        from .impl.rotary import rotary_embedding_<vendor>
        return rotary_embedding_<vendor>(query, key, cos, sin, position_ids, is_neox_style)
    def attention_backend(self):
        from .impl.attention import <VendorName>AttentionBackend
        return "<vendor_module>.impl.attention:<VendorName>AttentionBackend"

2. Implement the concrete operators

In the impl/ subdirectory, implement the concrete logic of each operator using vendor-native operators. For example, activation.py:

Python

import torch
def silu_and_mul_<vendor>(x: torch.Tensor) -> torch.Tensor:
    """
    实现 SiLU(x[:, :d]) * x[:, d:] 操作
    """
    d = x.shape[-1] // 2
    # 使用厂商原生算子实现
    # 例如华为：torch_npu.npu_silu_and_mul(x)
    return <vendor_native_op>(x)

normalization.py is similar:

Python

def rms_norm_<vendor>(x, residual, weight, epsilon):
    """RMS 归一化实现"""
    # 使用厂商原生算子
    # 例如华为：torch_npu.npu_rms_norm(x, weight, epsilon)
    return <vendor_native_rms_norm>(x, weight, epsilon)

3. Register the operators

In vllm_fl/dispatch/backends/vendor/<vendor_name>/register_ops.py, register the above operator implementations into OpRegistry and bind the availability check:

Python

import functools
from vllm_fl.dispatch.types import OpImpl, BackendImplKind, BackendPriority
def _bind_is_available(fn, is_available_fn):
    """绑定 is_available 检查函数"""
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        return fn(*args, **kwargs)
    wrapper._is_available = is_available_fn
    return wrapper
def register_builtins(registry) -> None:
    """注册所有厂商算子实现"""
    from .<vendor_name> import <VendorName>Backend
    backend = <VendorName>Backend()
    is_avail = backend.is_available
    impls = [
        OpImpl(
            op_name="silu_and_mul",
            impl_id="vendor.<vendor_name>",
            kind=BackendImplKind.VENDOR,
            fn=_bind_is_available(backend.silu_and_mul, is_avail),
            vendor="<vendor_name>",
            priority=BackendPriority.VENDOR,  # 100
        ),
        OpImpl(
            op_name="rms_norm",
            impl_id="vendor.<vendor_name>",
            kind=BackendImplKind.VENDOR,
            fn=_bind_is_available(backend.rms_norm, is_avail),
            vendor="<vendor_name>",
            priority=BackendPriority.VENDOR,
        ),
        OpImpl(
            op_name="rotary_embedding",
            impl_id="vendor.<vendor_name>",
            kind=BackendImplKind.VENDOR,
            fn=_bind_is_available(backend.rotary_embedding, is_avail),
            vendor="<vendor_name>",
            priority=BackendPriority.VENDOR,
        ),
        # 可选：attention_backend
        OpImpl(
            op_name="attention_backend",
            impl_id="vendor.<vendor_name>",
            kind=BackendImplKind.VENDOR,
            fn=_bind_is_available(backend.attention_backend, is_avail),
            vendor="<vendor_name>",
            priority=BackendPriority.VENDOR,
        ),
    ]
    registry.register_many(impls)

After these three steps are completed, the vendor operators will be automatically included in the Dispatch system. At runtime, SelectionPolicy will automatically decide whether to invoke the vendor implementation based on priority and availability. For reference, see the merged Huawei adaptation PR: https://github.com/flagos-ai/vllm-plugin-FL/pull/18.

Usage Workflow and Debugging

During integration, debugging is unavoidable. vLLM-plugin-FL provides four debugging methods:

1. Environment variable debugging

Markdown

# 启用调试日志
export VLLM_FL_LOG_LEVEL=DEBUG
# 启用分发调试模式（打印分发决策）
export VLLM_FL_DISPATCH_DEBUG=1
# 强制使用特定平台配置
export VLLM_FL_PLATFORM=<vendor_name>
# 强制使用特定后端
export VLLM_FL_PREFER=vendor
# 关闭FlagGems
export USE_FLAGGEMS=0
# 打开FlagGems
export USE_FLAGGEMS=1
# 使用FlagCX通信库
export FLAGCX_PATH="$FLAGCX_PWD"

2. Unit test validation

Python

# tests/test_<vendor>_backend.py
from vllm_fl.dispatch import get_default_manager
def test_<vendor>_backend_registration():
    manager = get_default_manager()
    manager.ensure_initialized()
    # 检查注册
    snap = manager.registry.snapshot()
    for op_name, impls in snap.impls_by_op.items():
        for impl in impls:
            if impl.vendor == "<vendor_name>":
                print(f"{op_name}: {impl.impl_id}, available={impl.is_available()}")
def test_<vendor>_silu_and_mul():
    import torch
    from vllm_fl.dispatch import call_op
    x = torch.randn(2, 128, device="<device>")
    result = call_op("silu_and_mul", x)
    assert result.shape == (2, 64)

3. Run the example script

Python

# 使用 examples/offline_inference.py 测试
cd /root/vllm-plugin-FL
python examples/offline_inference.py

4. Check operator dispatch

Python

from vllm_fl.dispatch import resolve_op
# 查看实际使用的实现
fn = resolve_op("silu_and_mul")
print(f"Using implementation: {fn}")

Configuration Priority and Common Environment Variables

When multiple configurations exist, vLLM-plugin-FL follows a clear priority order, from highest to lowest:

1.A user-defined configuration file specified by the VLLM_FL_CONFIG environment variable (complete override).

2.Environment variables (overriding specific items in the platform configuration).

3.Platform configuration files ascend.yaml / cuda.yaml (auto-detected).

4.Built-in defaults in the code.

Common environment variables:

Click the image to view the full spreadsheet.

If you need fine-grained control over FlagGems operators, there is also a whitelist/blacklist mechanism, which only takes effect when the flagos backend is used:

Python

# 白名单，默认空
export VLLM_FL_FLAGOS_WHITELIST="silu,rms_norm"
# 黑名单，默认空
export VLLM_FL_FLAGOS_BLACKLIST="mul"

The FlagGems operator statistics log path is /tmp/flaggems_enable_oplist.txt, which can be used to check which operators actually used the FlagGems implementation.

Conclusion

Looking back at the overall design of vLLM-plugin-FL, it confines hardware differences to the operator layer while keeping framework logic unified in the plugin layer. At the framework layer, it connects to vLLM through an Out-of-Tree plugin approach, ensuring that a single vLLM version can run across multiple chips. At the operator layer, it uses three types of backends plus a priority-based Dispatch mechanism, balancing generality with room for vendor optimization. Combined with the AutoTune Engine for end-to-end optimal selection and a standardized vendor integration workflow, the whole system reduces the adaptation challenge from “M models × N chips” to “M models + N chips.”

Judging from the measured results on Qwen3-Next — 100% Triton operator replacement and throughput reaching 99.6% of native performance — this technical route of “unification + dispatch + auto-tuning” is already capable of competing head-to-head with native paths in terms of performance. For chip vendors, the standardized workflow of three-step integration plus four debugging methods also lowers the barrier to cross-chip adaptation enough. This may also be why FlagOS can continue to achieve Day-0 multi-chip releases on models such as Qwen3.5, MiniMax M2.7, MiniCPM5, and Hunyuan MT2.

From design philosophy to hands-on integration, vLLM-plugin-FL demonstrates its core value as an open-source intelligent computing software system for diverse AI chips. We welcome more chip companies and developers to join the FlagOS community and work together to build a truly open, unified, and efficient AI open computing ecosystem.

Project Links and Resources

• vLLM-plugin-FL repository: https://github.com/flagos-ai/vllm-plugin-FL
• FlagOS official website: https://flagos.io