The Evolution of MLLMs

From Text to Unified Multimodal Intelligence (2018–2026)

This article traces the evolution of Multimodal Large Language Models (MLLMs) from 2018 to 2026, charting their transformation from text-centric transformers to unified agentic systems. We examine key milestones: dual-encoder foundations, CLIP’s cross-modal alignment, efficient bridging via Q-Former, the open-source surge of visual instruction tuning with LLaVA, and the emergence of real-time omni models. By 2025–2026, unified tokenization and next-token prediction enabled any-to-any understanding and generation. We discuss architectural shifts, specialized applications, persistent challenges, and future directions toward embodied world-modeling intelligence.

The Evolution of Multimodal Intelligence (2018–2026)

1. Foundational Era: Dual Encoders (2018–2020)

Before 2021, multimodal systems were defined by independent streams for vision and language. These models lacked generative reasoning and were largely task-specific.

1.1 The Text Paradigm

Models like BERT and early GPT variants established the transformer as the dominant architecture, demonstrating emergent reasoning through scaling, yet remained “blind” to non-textual data.

Text-to-Text Models: Transformers, GTPs, and BERT.

1.2 Early Fusion Models

Systems such as VisualLBERT (2019), LXMERT (2019), and UNITER (2020) used dual-stream or cross-attention encoders. They relied on object detectors (e.g., Faster R-CNN) for region features but lacked LLM-scale parameters and general instruction-following capabilities.

For example, VisualBERT (Li et al. 2019) feeds both text inputs and image regions into BERT such that it is able to discover the internal alignment between images and text with self-attention mechanism.

VisualBERT (2019)

1.3 Vision Transformer (ViT 2020) — The Architectural Bridge

While not multimodal itself, ViT fundamentally reshaped the vision side of multimodal AI. The original ViT decomposes an input image into fixed-size patches (e.g., 16×16 pixels), linearly projects each patch into a vector embedding, adds positional encodings to preserve spatial structure, and processes the sequence through a standard transformer encoder.

https://webisoft.com/articles/vision-transformer-model/

Why ViT mattered for multimodal systems:

• Architectural Uniformity: By using the same transformer building blocks as LLMs, ViT enabled seamless integration with language models — no more CNN-to-transformer adaptation headaches.
• Global Context: Self-attention across patches captures long-range dependencies that CNNs struggle with, improving semantic understanding for alignment tasks.
• Scalability: ViT scales predictably with data and compute, mirroring the scaling laws that drove LLM progress.
• Patch-Based Representation: The discrete patch tokens map naturally to LLM token embeddings, simplifying the design of projection modules.

2. The Bridging Era: Shared Latent Spaces (2021–2022)

The breakthrough in this period was the development of the “Rosetta Stone” for multimodal AI: the ability to align different data types into a shared mathematical space.

Key Developments

• CLIP (OpenAI, 2021): Contrastive Language-Image Pretraining (CLIP) aligned visual features and textual concepts using the InfoNCE loss function. It became the de facto visual backbone for nearly all subsequent MLLMs.

https://arxiv.org/pdf/2103.00020v1.pdf

• Flamingo (DeepMind, 2022): The first true MLLM-scale model (~80B parameters). It introduced gated cross-attention layers and the Perceiver resampler to interweave visual data into text prompts, enabling few-shot in-context learning.

Flamingo’s Architecture: •Inject vision vectors throughout the LLM and training using long multi-image documents

• BLIP-2 (2022): Introduced the Q-Former (Query Transformer), a lightweight module that extracted relevant visual features to feed into a frozen LLM, significantly reducing the computational cost of alignment.

BLIP-2: Bootstrapping Language-Image Pre-training with frozen image encoders and large language models. (source)

3. The LLaVA Paradigm: Visual Instruction Tuning (2023)

2023 marked the “explosion” of open-source MLLMs, driven by the realization that models needed to be taught how to communicate effectively about visual data.

The LLaVA Architecture

The Large Language and Vision Assistant (LLaVA) popularized a minimalist “projector” architecture:

1. Vision Encoder: (e.g., CLIP ViT) The “Eyes” that transform pixels into features.
2. Language Decoder: (e.g., Vicuna/LLaMA) The “Brain” that generates text.
3. Connection Module: A linear projection or Multi-Layer Perceptron (MLP) that acts as the “Translator.”

LLaVA: https://llava.hliu.cc/

Visual Instruction Tuning

LLaVA introduced a two-stage training pipeline:

• Stage 1 (Feature Alignment): Updates only the projector to align visual features with text embeddings.
• Stage 2 (Instruction Tuning): Unfreezes the LLM to learn complex reasoning using synthetic data generated by GPT-4. This data focused on conversation, detailed description, and complex reasoning (e.g., explaining why a man ironing clothes on a moving taxi is unusual).

LLaVA: Visual Instruction Tuning

4. Native Multimodality and Scaling (2024)

By 2024, models transitioned from “bolted-on” vision components to natively multimodal architectures where multiple modalities are integrated from the start.

• “Omni” Models: GPT-4o integrated text, vision, and audio into a single unified model to reduce latency and enable real-time voice interactions.

[embed]

• Massive Context: Google’s Gemini 1.5 and Anthropic’s Claude 3 family introduced native vision with context windows reaching 1M+ tokens.
• High-Resolution Handling: Innovations like LLaVA-NeXT used dynamic sub-image slicing to process high-resolution inputs without losing detail.

https://llava-vl.github.io/blog/2024-01-30-llava-next/

• Efficiency: Mixture of Experts (MoE) variants (e.g., MM1, MoE-LLaVA) were implemented to maintain performance while managing computational load.

5. Unified Intelligence: Any-to-Any Understanding and Generation (2025–2026)

By 2025–2026, the distinction between understanding and generation blurred. Models moved toward “any-to-any” capabilities where text, images, and video are generated within the same semantic space.

Technical Shifts

• Next-Token Prediction for Everything: Models like Emu3 and Chameleon proved that separate diffusion models (like Stable Diffusion) are not required. By tokenizing images and audio into discrete variables, MLLMs use pure autoregressive prediction to generate high-fidelity media.

https://www.nature.com/articles/s41586-025-10041-x

• Hybrid Architectures: Models like Janus and Transfusion fused causal masking for text with diffusion or flow-matching for images.

• Agentic Workflows: Systems like Qwen3-Omni and Gemini 3.1 Pro serve as real-time agents, autonomously navigating user interfaces and processing continuous video streams to act as collaborative partners.

Workflows vs. Autonomous AI Agents

6. Specialized Applications and Impact

The evolution of MLLMs has enabled high-precision applications in specialized fields:

• Medical Imaging (LLaVA-Med): Uses Curriculum Learning to move from general concept alignment to diagnostic instruction tuning, assisting in radiology and clinical dialogue.

• Robotics (PaLM-E): Embodied agents that use MLLMs as backbones for planning and environmental interaction.

https://research.google/blog/palm-e-an-embodied-multimodal-language-model/

• Agentic Tools (LLaVA-Plus): Incorporates a “Skill Repository,” allowing the model to trigger external tools like search engines or segmentation models (Grounding DINO) when internal knowledge is insufficient.

https://llava-vl.github.io/llava-plus/

7. Challenges and Future Directions

7.1 Persistent Limitations

Despite rapid progress, several persistent limitations remain:

• Hallucination: Visual grounding failures where the model “sees” things not present in the image.
• Modality Imbalance: Vision capabilities often remain weaker than textual reasoning.
• Inference Costs: High computational requirements for processing long visual sequences or high-resolution video.
• Safety: Vulnerabilities to jailbreaks and adversarial image inputs.

Persistent Friction Points in MLLM Architecture.

7.2 The Post-2026 Outlook

The post-2026 landscape is expected to focus on:

• Deeper Unification: Development of 3D and world models.
• Test-Time Scaling: Implementing multimodal Chain-of-Thought reasoning.
• Reliability: Utilizing Reinforcement Learning with Verifiable Rewards (RLVR) to ensure multimodal agents perform reliably in production environments.

The trajectory of MLLMs indicates that architectural efficiency and the democratization of AI are the primary engines of innovation, allowing smaller, open-source models to remain competitive with proprietary systems.