SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models — Paper Summary

Figure 1

Liu, B., Qiao, P., Ma, M., Zhang, X., Tang, Y., Xu, P., … & Yuan, T. (2025). Surveillancevqa-589k: A benchmark for comprehensive surveillance video-language understanding with large models. arXiv preprint arXiv:2505.12589.

Abstract: SurveillanceVQA-589K, the largest open-ended video question answering (VQA) benchmark tailored to the surveillance domain. The dataset comprises 589,380 QA pairs spanning 12 cognitively diverse question types, including temporal reasoning, causal inference, spatial understanding, and anomaly interpretation, across both normal and abnormal video scenarios. We design a hybrid annotation pipeline that combines temporally aligned human-written captions with Large Vision-Language Model (LVLM) assisted QA generation using prompt-based techniques. We also propose a multi-dimensional evaluation protocol to assess contextual, temporal, and causal comprehension. They evaluate eight LVLMs under this framework, revealing significant performance gaps, especially in causal and anomaly-related tasks, underscoring the limitations of current models in real-world surveillance contexts.

Introduction:

Main contributions are as follows:

• Constructed SurveillanceVQA-589K, the largest surveillance video QA dataset to date, containing 589,380 QA pairs across 12 task types and 18 abnormal event categories, enabling comprehensive semantic understanding under both normal and abnormal conditions.
• Introduced a scalable hybrid annotation pipeline, combining human-aligned captions with LVLM-generated content, and propose a multi-dimensional evaluation framework tailored to surveillance video reasoning tasks, including contextual, temporal, and causal metrics.
• The benchmark evaluates eight open-source LVLMs (0.5B–7B) (including variants like VideoLLaMa3 , LLaVA series, Qwen2.5-VL series, and InternVL) and reveals systematic weaknesses in causal inference and anomaly understanding, offering a practical testbed for real-world applications such as intelligent security and video anomaly response systems.

Surveillance Video Analysis Benchmark

The authors proposed the construction of a novel QA-driven benchmark for surveillance video understanding. This benchmark is designed to enable interactive evaluation and foster deeper semantic reasoning over real-world surveillance video content.

Figure 2: Our overall framework, including QA generation and evaluation.

SurveillanceVQA-589K

31,548 video clips + textual annotations, 27966 clips = normal and 3585 clips = anomalous. Total of 589,380 QA pairs.

Video Annotation Generation

1- Human labeled annotation

The process was supervised by AI researchers. This process involved generating event-level captions that included both precise timestamps and detailed event descriptions. Completed the manual annotation collection with a total of 31,548 sentence-level annotations accompanied by precise timestamps.

2- LVLM-labeled Annotation

They employed the video processing toolkit MoviePy to automatically segment the original videos and extract the corresponding short clips. Then utilized LLaVA-Video- 7B-Qwen2 to perform in-depth analysis on each segmented clip, generating detailed descriptions.

3- Integrated Human-LVLM Annotations

Qwen-Turbo was tasked with identifying and resolving redundant or inconsistent expressions, while enhancing semantic richness and logical coherence. This resulted in more fluent, structured, and contextually aligned event-level descriptions for each video clip.

Automatic QA Generation

They employed Qwen-Max to analyze the annotation data and classify each video segment as either normal or abnormal. For normal clips, the prompts were crafted to elicit a comprehensive understanding of video content, focusing on global scene descriptions, temporal sequencing, spatial detail extraction, and behavioral inference. In contrast, for abnormal clips, the prompts emphasized event detection, anomaly type classification, subject identification, detailed incident descriptions, and causal reasoning.

Data

Statistical information for the Survelliance VQA-589K

We split the dataset at the clip level rather than by entire videos, using an 8/2 ratio for the training and testing sets. Due to the difficulty of annotating every second of footage, the total duration of the training and testing sets is slightly shorter than that of the original raw video collection.

Distribution of QA pairs for normal and abnormal events across the training and testing sets.

Here, the normal category refers to non-anomalous clips, while the abnormal category corresponds to anomalous clips, which span 18 distinct abnormal event classes.

Experiments on SurveillanceVQA-589K

1- Evaluation Design

They follow VideoGPT+ to design the evaluation criteria. During the evaluation phase, they adopt LLM-based evaluation strategy, using an open API with GLM-4-Flash. With this approach the evaluation process considers the semantic consistency of the answers and everages the interpretive and reasoning capabilities of advanced language models.

This evaluation framework provides comprehensive assessment of model-generated answers across four key dimensions:

• Contextual Integration (CI): It measures whether the answer accurately reflects the factual content of the video, avoiding errors or misinterpretations.
• Detail Orientation (DO): It assesses the inclusion of specific and complete key elements.
• Contextual Understanding (CU): Evaluates the alignment of the answer with the overall narrative and emotional tone of the video.
• Temporal Understanding (TU): Focuses on the correctness of event sequences and time-related logic.

Each dimension is rated on a 0–5 integer scale, with 5 indicating full accuracy and relevance, and 0 indicating a completely incorrect response. To calculate Average Score (Avg), individual scores are normalized by multiplying each by 0.25 and summing the results.

2- Baselines and Settings

They evaluate 8 open-source video-language models, including the VideoLLaMA3, InternVL2.5, LLavA-OV-Qwen2), LLaVA-Video-Qwen2, and Qwen2.5-VL-Instruct series, with parameter sizes ranging from 0.5B to 7B.

Each model performs inference on one question at a time to prevent information leakage between questions. They select Qwen2.5-VL-Instruct- 3B for LoRA fine-tuning, which offers a good balance between parameter size and performance, and conduct the fine-tuning for one epoch on our training set. All experiments are carried out on an NVIDIA RTX 4090 GPU.

Overview of Evaluated Open-Source Video Models

3- Results on SurveillanceVQA-589K

Table 5: Model performance averaged on different QA tasks across five evaluation dimensions. †represents our finetuned LVLMs

Table 5 presents the evaluation results of various LVLMs across five key dimensions: CI, DO, CU, TU, along with Avg. LLaVA-Video-7B-Qwen2 achieved the highest overall performance, with an average score of 2.72. This strong performance can be attributed to its integration of the LLaVA and Qwen2 architectures and its use of the AnyRes technique, which enhances image-to-video reasoning.

In contrast, InternVL2.5–2B demonstrated the lowest overall average, with notably weak scores in CI and DO. While its surface-level integration of context and detail may lag, it retains potential in complex reasoning scenarios due to a Chain-of-Thought reasoning mechanism. Interestingly, the LLavA-OV-Qwen2–0.5B model, with just 0.5B parameters, achieved a competitive average score of 2.76. This demonstrates that smaller-scale models can still deliver strong video comprehension, thanks to efficient modality transfer learning between visual and textual inputs.

The Qwen2.5-VL-3B-Instruct model also showed robust performance. Designed for multimodal instruction following, this model emphasizes cross-modal alignment and optimized question-answer generation, equipping it to handle a variety of video, image, and text tasks with precision.

Finally, Video-LLaMA3–2B/7B incorporates advanced features like Any-resolution Vision Tokenization and the Differential Frame Pruner, which improve video representation quality and processing efficiency across varied resolutions and temporal segments.

Performance of different vision-language models across QA tasks. Blue: normal QA tasks on normal video clips, Green: normal QA tasks on abnormal video clips, Brown: abnormal QA tasks on abnormal video clips. †represents our finetuned LVLMs

A. Analysis of LVLMs performance across normal QA tasks

LLaVA-OV-Qwen2–7B demonstrates the highest overall performance, indicating robust capabilities in handling spatial and reasoning-based questions. In contrast, InternVL2.5–2B exhibits the lowest performance across most tasks. A notable trend is the higher performance on normal video clips compared to abnormal ones across all models and tasks.

B. Analysis of LVLMs performance across abnormal QA tasks

the LLaVA-OV-Qwen2 series models excel in Detection tasks, with the 0.5B variant achieving top scores on CI and CU metrics. For Classification tasks, both LLaVA-Video- 7B and LLaVA-OV-Qwen2–7B showcase superior performance across different evaluation criteria. The LLaVA-Video-7B model consistently demonstrates advantages in Subject and Description type questions.

Notably, all models perform substantially weaker on higher-order reasoning tasks such as Cause and Result inference, with scores generally falling below the midpoint threshold. LLaVA-NeXT-7B performs relatively better in causal reasoning while LLaVA-Video-7B shows slight advantages in result inference. These patterns indicate that current vision-language models still face significant challenges in understanding causal relationships and reasoning about complex video content, particularly in abnormal video scenarios.

Conclusion

The dataset contains 589,380 QA pairs spanning 12 cognitively diverse task types across both normal and abnormal surveillance video contexts. They propose a hybrid annotation pipeline that combines human-aligned captions with LVLM- assisted QA generation, enabling high-quality, scalable annotation. They benchmark eight state-of-the- art open-source LVLMs ranging from 0.5B to 7B parameters. Our experiments reveal that while these models demonstrate promising performance on general understanding tasks (e.g., Summary, Spatial, and Qeneric QA on normal videos), they struggle significantly with complex semantic reasoning, particularly in anomaly-specific tasks such as causal inference and result prediction indicating a clear performance bottleneck in high-level temporal and logical reasoning. They also examine the impact of fine-tuning through LoRA on a 3B model. While fine-tuning yields moderate gains on general tasks, it provides limited improvement in structured anomaly detection and classification, highlighting that current parameter-efficient tuning approaches are insufficient for enabling domain-specific, structured expression. Overall, this work provides a comprehensive testbed for evaluating multimodal models in realistic surveillance settings.

Limitations

Model evaluations are based on a limited dataset, and performance may vary when applied to different surveillance environments or low-quality video data. Although larger models perform well in terms of accuracy, they have high computational demands, which may make them challenging to deploy in resource-constrained environments. Future research should focus on optimizing the temporal reasoning capabilities of smaller models and further explore domain-specific fine-tuning techniques to improve performance in various monitoring scenarios.