Vision Language Navigation (VLN) Insights (Part 1)

Image is generated by ChatGPT

Architecture advantages of VLN-specific models: In VLN tasks, both historical frames and the current frame are important, but the current frame carries more weight. Many VLN architectures (e.g., NaVid and UniNaVid) allocate more tokens to represent the current frame, thereby improving decision accuracy. Moreover, since VLN requires processing a large number of historical frames, the efficiency of retrieving and utilizing these frames directly impacts inference speed. For example, StreamVLN stores historical frame features using a KV-Cache mechanism, while MapNav explicitly maps historical semantic information (e.g., visited objects, trajectories) onto a BEV map, transforming multi-frame processing into single BEV map processing and significantly improving inference efficiency.

Data advantages of VLN: The RxR and R2R datasets contain abundant real-world trajectory information. Combined with Habitat simulation, this enables the creation of large-scale vision-language-action data for VLN, which can be used to fine-tune large models (SFT) and achieve strong performance on val unseen splits. Furthermore, incorporating ObjNav or other embodied VQA datasets for co-training can further enhance performance. At the same time, the DAgger mechanism is employed to collect data with navigation error-correction capability, thereby enhancing the model’s ability to recover when deviating from the intended trajectory. Since the input–output format is compatible with real-world settings, the model can generalize well to real scenarios.In addition, the output of such VLN data typically takes the form of “move forward N meters” and “turn left/right X degrees.” This instruction format facilitates cross-embodiment generalization. For instance, in NaVILA, feeding such VLN output commands into control models of different embodiments enables straightforward cross-embodiment transfer.

Limitations of base models: Compared to the architecture and data advantages of VLN-specific models, base models lack the ability to map visual-language inputs to navigational actions. Moreover, in the evaluation set, the videos are “skip-frame” sequences — two discrete frames from the current point to the next — which are relatively rare in base model pretraining. This scarcity makes it difficult for base models to capture the correspondence between discrete frames. Therefore, constructing matching skip-frame VLN video data in Habitat simulation can help large models adapt to and understand this special type of video.

Key points in VLN: To achieve generalizable and efficient VLN, we need to:

1. Build large-scale VLN datasets to learn the mapping from video–language inputs to actions;

2. Learn action transitions across skip/discrete frames;

3. Address agent “localization” within language instructions (which sub-instructions have been followed and which part to execute next);

4. Construct error-correction data via mechanisms such as DAgger to improve recovery under generalization;

5. Optimize storage and retrieval of historical frames (e.g., KV-cache, BEV semantic mapping) to accelerate inference.

VLN 专有模型的架构优势：在 VLN 任务中，历史帧与当前帧都很重要，但当前帧的重要性更高。许多 VLN 架构（如 NaVid 和 UniNaVid）会使用更多的 token 来表征当前帧，从而提升决策精度。此外，由于 VLN 需要处理大量历史帧，如何高效获取与利用这些历史信息也会影响推理速度。例如，StreamVLN 通过 KV-Cache 方式存储历史帧特征；MapNav 则将历史语义信息（如经过的物品、轨迹等）显式映射到 BEV 图中，将多帧处理转化为单张 BEV 图处理，从而显著提升推理效率。

VLN 数据优势：RxR 和 R2R 数据集包含大量真实轨迹信息，结合 Habitat 仿真，可以构建大规模 VLN 的视觉-语言-动作（Vision-Language-Action）数据，用于对大模型进行 SFT，在 val unseen 数据上取得良好效果。此外，加入 ObjNav 或其他具身相关 VQA 数据进行协同训练（co-training），能够进一步提升性能。同时，还采用 DAgger 机制收集具有导航纠错能力的数据，以增强模型在偏离轨迹时的纠错能力。由于输入输出格式与真实世界兼容，模型可直接泛化到真实场景。此外，这类 VLN 数据的输出通常采用“前进 N 米、左右转 X 度”的形式。该指令格式更易于实现跨本体的泛化。例如，在 NaVILA 中，将此类 VLN 输出指令输入至不同本体的控制模型，即可轻松实现跨本体迁移。

基础模型的局限性：相比 VLN 专有模型的架构和数据优势，基础模型缺乏从视觉语言到移动动作的映射能力。此外，评测集中的视频为跳帧形式，即由当前点和下一点的两帧构成，这类离散视频在基础模型的预训练中相对稀缺，导致其难以理解帧间关系。因此，在 Habitat 仿真中构建对应的离散视频 VLN 数据，可帮助大模型适应这种特殊视频模式的理解需求。

VLN核心点：要实现通用且高效的 VLN，需要：

1.构建大规模 VLN 数据，学习从视频-语言到动作的映射；

2.学习跳帧/离散帧之间的动作变换关系；

3.解决机器人在语言指令中的“定位”问题（已遵循哪些子指令，以及下一步应执行哪一部分）；

4.通过 DAgger 等机制构建纠错数据，提升在泛化场景下的错误恢复能力；

5.优化历史帧的存储与访问（如 KV-Cache、BEV 语义映射），以提高推理速度。