FlagOS Tech Workshop: As AI Begins to “Deploy Itself,” Large-Model Infrastructure Enters the Era of Autonomous Operations

What if AI could write its own kernels, deploy its own models, migrate workloads across heterogeneous hardware, and even troubleshoot the most tedious environment errors on its own?

At the 8th Beijing Academy of Artificial Intelligence (BAAI) Conference, this vision became reality at the FlagOS Tech Workshop.

Spanning a full day of technical sessions and hands-on labs, the workshop showcased the latest advances across the FlagOS ecosystem, including compiler optimization, edge AI enablement, intermediate representations, quantum-AI integration, LLM Agent-driven deployment, and AI-generated operator development. More importantly, developers were given access to real hardware environments for live experimentation. From manually implementing high-performance operators with Triton-TLE to generating optimized kernels using a single natural-language prompt through KernelGen, participants could learn, build, and deploy under the guidance of FlagOS researchers and engineers.

Building the Foundations of FlagOS: Compilers, Operator Libraries, and End-to-End Edge Optimization

In the opening keynote, Chunlei Men, Head of the AI Systems Research Team at BAAI, highlighted the growing demand for compiler and operator developers driven by the rapid rise of foundation models. While low-level system software is often overlooked, it directly determines token costs and hardware utilization efficiency. A unified and optimized AI system software stack can significantly reduce compute expenses while improving accelerator performance.

The workshop was designed to bridge the gap between hardware and models, providing developers with practical knowledge across the entire AI system software stack — from chips to applications.

Triton-TLE: High-Performance SparseMLA Without the Complexity

BAAI compiler researcher Hui Guo demonstrated how Triton-TLE enables developers to build high-performance SparseMLA operators with dramatically reduced complexity.

As context lengths expand from thousands to millions of tokens, the quadratic complexity of traditional attention mechanisms becomes increasingly prohibitive. DeepSeek’s sparse attention strategy reduces computation by selecting only a subset of TopK Tokens from the KV cache, but implementing such operators efficiently remains challenging due to irregular memory access patterns, wide computation dimensions, and pipeline scheduling complexity.

Guo introduced the three levels of Triton Language Extensions (TLE): Lite, Struct, and Raw, with a focus on the newly introduced Tle.pipe abstraction. By adopting a producer-consumer programming model, developers can simply declare pipes and invoke APIs such as acquire, commit, wait, and release, while the compiler automatically handles synchronization and barrier management.

Compared with manually implemented pipelines filled with barrier operations and synchronization logic, the TLE version requires only three pipes corresponding to different data transfer paths. On NVIDIA H800 GPUs, the resulting implementation achieves a 1.9×–2.1× speedup over native Triton baselines while reaching more than 90% of DeepSeek FlashMLA performance. Code volume is reduced by over 60%, and the same implementation can run across AMD GPUs and domestic accelerators without modification.

During the hands-on lab, participants implemented core SparseMLA pipeline logic using Tle.pipe while receiving guidance from FlagOS engineers on synchronization mechanisms and pipeline design patterns.

FlagOS on Arm64: Bringing AI to the Edge

BAAI performance engineer Shuai Zhao presented FlagOS enablement and optimization efforts for Arm64 platforms. Four development boards running FlagOS edge environments were made available on-site, allowing attendees to directly experiment with real hardware.

Zhao noted that edge NPUs are rapidly approaching 100 TOPS of compute capability, yet software ecosystems remain fragmented and largely proprietary. FlagOS aims to provide an open alternative for Arm v9 platforms with SVE2 support, RISC-V architectures, and Qualcomm robotics platforms.

Through a series of optimizations — including thread management improvements, kernel-level execution control, and operator fusion — the team achieved significant performance gains. INT8 GEMM reached 519 GOPS, RMSNorm improved by 1.5×, SwiGLU by 40%, GEMV by 30% over ARM ACL, and FlashAttention achieved a 9× improvement compared to ATen. Under INT8 quantization, MiniCPM and Qwen-1.7B achieved 70–80% of Llama.cpp performance on Arm-based systems.

FlagTree Compiler: Day-0 Multi-Chip Enablement

Compiler researcher Hang Xiao presented how FlagOS delivers Day-0 support for emerging foundation models through coordinated optimization across compilers, operator libraries, and model execution frameworks.

When DeepSeek V4 was released, FlagOS completed support across ten AI chip platforms on the same day. This capability relies on the FlagTree compiler and the FlagGems operator library, which now contains more than 600 cross-platform operators.

Recent compiler enhancements — including automatic layout elimination, loop unrolling, instruction scheduling, and warp-specialized pipelines — have enabled up to 2.2× acceleration on multiple quantized operators. Looking ahead, FlagGems is expected to expand to more than 2,000 operators while introducing advanced operator fusion and broader mixed-precision support.

FlagIR: A Unified Intermediate Representation for GPGPU and DSA Architectures

BAAI compiler researcher Yang Zheng introduced FlagIR, the unified intermediate representation underpinning the FlagTree compiler.

Rather than allowing separate Triton forks to emerge across different hardware vendors, FlagTree adopts a single-repository, multi-backend strategy. While GPGPU architectures such as NVIDIA, Moore Threads, and Hygon can largely reuse Triton’s existing GPU IR pipeline, NPUs and domain-specific accelerators require a different approach.

FlagIR bridges this gap by introducing a linear-algebra-based intermediate layer capable of supporting structured and unstructured memory operations as well as tensor computations. It serves as the key foundation enabling Triton-style programming across heterogeneous accelerator architectures.

Frontier Innovation and Engineering Automation

FlagQuantum: Accelerating AI Through Quantum-AI Integration

Quantum and AI systems researcher Wei Liu introduced FlagQuantum, FlagOS’s framework for quantum-AI integration.

Instead of attempting to replace classical computing, FlagQuantum treats quantum processors as accelerators embedded within neural networks. Its heterogeneous architecture combines traditional AI chips and quantum hardware through two key innovations:

• Dynamic two-dimensional partitioning for distributed quantum simulation, minimizing expensive inter-node communication.
• Reversible gradient computation, which leverages the reversible nature of quantum gates to reduce memory consumption during backpropagation.

FlagQuantum is currently the first framework capable of orchestrating domestic AI accelerators and domestic quantum processors within a unified PyTorch-based environment while supporting OpenQASM 2.0 and 3.0 export to real quantum hardware platforms.

LLM Agents for Automated AI Deployment and Migration

BAAI system software engineer Honghao Shi introduced FlagRelease, a deployment system designed to reduce integration complexity across AI inference stacks.

Inspired by layered abstractions in operating systems and networking, FlagRelease reduces deployment complexity from M×N combinations to M+N. The system implements a 13-stage automated workflow governed by explicit state machines rather than free-form agent decisions.

To address agent uncertainty, FlagRelease introduces external verification, layered sandboxing, and 32 operational constraints. As a result, model migration can be completed in approximately 92 minutes at a token cost of around RMB 100 — compared to the two to three days typically required by experienced engineers.

KernelGen: From Pixel-Level Optimization to One-Sentence Operator Generation

System intelligence researcher Dongxu Han presented KernelGen 2.0, which embeds more than a decade of expert operator development knowledge into an AI-assisted generation platform.

In benchmark comparisons, KernelGen generated a LayerNorm implementation in just 14 minutes using roughly 10,000 tokens, achieving 1.14× the performance of manually optimized expert code. By contrast, mainstream AI coding assistants required more than twice the tokens and produced significantly slower implementations.

KernelGen currently supports multiple accelerator platforms, including Ascend and Hygon, and enables developers to generate optimized operators through natural-language instructions while automatically validating correctness and performance.

At the workshop, participants logged into the KernelGen platform and generated custom operators for multiple hardware platforms using simple prompts, experiencing firsthand the productivity gains of AI-assisted systems programming.

FlagScale Agent: Autonomous AI Infrastructure Operations

BAAI framework researcher Zhou Cao introduced FlagScale Agent, a system designed to automate model training, inference migration, and deployment.

The project addresses three persistent industry challenges: infrastructure complexity, fragmented hardware ecosystems, and the continuous maintenance burden caused by rapidly evolving upstream frameworks.

FlagScale introduces two major innovations:

• Constraint-driven execution, where operational skills act as enforceable rules rather than recommendations.
• Progressive context compression, which dynamically manages long-running agent memory while preserving critical information.

In comparative testing, FlagScale Agent successfully completed a 100-step precision alignment workflow across NVIDIA and MetaX hardware platforms without human intervention, achieving convergence with only 0.12% difference. Comparable workflows using general-purpose coding agents required manual correction and produced significantly larger deviations.

Future versions of FlagScale Agent will support self-evolving Skills, allowing agents to generate and refine operational knowledge autonomously.

vLLM-Plugin-FL Skills: Turning Expert Experience into Executable Assets

BAAI inference optimization engineer Ruidi Zhu demonstrated how Skill-based automation transforms installation, version upgrades, and model migration into reusable software assets.

These Skills can automatically:

• Build complete inference environments from scratch.
• Resolve compatibility issues between different vLLM versions.
• Migrate models while validating token-level inference accuracy.

Unlike traditional scripts, Skills incorporate reasoning and exception handling capabilities, converting institutional knowledge into reusable and scalable engineering workflows.

Conclusion: From Correct Components to Autonomous Systems

Throughout the FlagOS Tech Workshop, a clear theme emerged:

FlagOS is evolving from a collection of optimized individual components into an AI-native system software stack where those components can collaborate autonomously in real-world environments.

Triton-TLE, FlagTree CPU extensions, and FlagIR demonstrated the deep systems engineering required to unlock hardware performance across heterogeneous architectures. At the same time, FlagQuantum, FlagRelease, KernelGen, FlagScale Agent, and the growing Skills ecosystem showcased how AI agents can increasingly take over migration, deployment, adaptation, and operator development tasks.

As compilers, quantum co-processors, agent orchestrators, and executable knowledge libraries converge into a continuously evolving software ecosystem, the era of autonomous AI infrastructure is no longer a future vision — it is already taking shape.