When AI “Hears” With Its Eyes: How Multimodal Models Fake Audio Understanding

Your favorite AI video model claims it can hear the music in a clip. Spoiler: it can’t. It’s reading lips- and fooling everyone, including its creators.

Can AI models actually Hear or just pretend to hear? Image generated by Author

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!

This paper reveals something unsettling: the leading multimodal AI models- the ones that claim to understand video with audio- are faking it. They’re not actually listening to the sound. They’re looking at the visuals and guessing what the audio should be.

Let me explain why this matters, how they proved it, and what it means for anyone building with multimodal AI.

Huggingface page for the paper itself

The Clever Hans Problem

In 1904, a horse named Clever Hans became famous for apparently doing arithmetic. Ask him “What is 3 + 4?” and he’d tap his hoof seven times. Scientists were baffled. Psychologists tested him rigorously.

The truth? Hans couldn’t do math. He was reading his handler’s body language — subtle shifts in posture that changed when he’d reached the right number of taps.

The researchers behind “When Vision Speaks for Sound” found that today’s multimodal LLMs are Clever Hans. They look like they understand audio, but they’re reading visual cues instead.

# Conceptual demonstration of the Clever Hans effect
# 
# Think of it this way:

scenario_normal = {
    "video": "A person playing guitar on stage",
    "audio": "Guitar music playing",
    "model_question": "What instrument is being played?",
    "model_answer": "Guitar",  # Looks correct!
    "actual_reasoning": "I see a guitar in the visual frames"  # NOT "I heard guitar sounds"
}

scenario_tricky = {
    "video": "A person playing guitar on stage",  
    "audio": "Swapped: someone speaking French",  # Audio replaced!
    "model_question": "What instrument is being played?",
    "model_answer": "Guitar",  # Still says guitar!
    "actual_reasoning": "I still see a guitar in the visual frames"
}

# The model gives the same answer regardless of audio input.
# It was NEVER using the audio. It was reading the video.

This is exactly what the researchers found. And they proved it with a beautifully simple experimental framework.

The THUD Framework: Three Counterfactual Edits

The researchers created Thud- a framework that tests audio understanding through intervention. The idea: if the model truly uses audio, changing the audio should change its answers.

They designed three specific tests:

The Counterfactual Edits that can confuse the model

Test 1: SHIFT — Does the model notice when timing is off?

Take a video of someone clapping. The audio of the clap happens at the exact moment their hands meet. Now shift the audio so the clap sound happens 2 seconds early. A model that actually listens would notice the desynchronization. A model that only watches wouldn’t care.

# Conceptual: What SHIFT tests

Original video:  [hands move] ... [hands meet] ... [clap sound synced]
SHIFTed video:   [hands move] ... [hands meet] ... [clap sound 2s early]

Question: "Are the audio and video synchronized?"

# A model that HEARS: "No, the clap is out of sync"
# A model that only SEES: "Everything looks fine to me" (ignores audio entirely)

Test 2: MUTE — Does the model notice when sound disappears?

Take the same video of someone clapping. Now remove the audio entirely. Replace it with silence. Ask the model: “Is there sound in this video?”

A model that actually processes audio should say “No, it’s silent.” A model relying on visual cues would say “Yes, I hear clapping” — because it sees clapping and assumes audio must be there.

# Conceptual: What MUTE tests

Original video:  [person clapping] + [clapping sounds]
MUTED video:     [person clapping] + [silence]

Question: "Is there sound in this video?"

# A model that HEARS: "No, the audio is silent"
# A model that only SEES: "Yes, there's clapping" (hallucinated from visuals)

Test 3: SWAP — Does the model notice completely wrong audio?

Take a video of a dog barking. Replace the audio with someone playing piano. Ask: “What sound is in this video?”

# Conceptual: What SWAP tests

Original video:  [dog barking] + [barking sounds]
SWAPped video:   [dog barking] + [piano music]

Question: "What sound is in this video?"

# A model that HEARS: "Piano music"
# A model that only SEES: "Barking" (goes with the visual)

The Devastating Results

Here’s what happened when they tested state-of-the-art models — both open-source omni models and closed-source models from Google and OpenAI:

# The results (conceptual summary from the paper)
import matplotlib.pyplot as plt
import numpy as np

models = ['Model A\n(Open Source)', 'Model B\n(Open Source)', 'Model C\n(Closed Source)', 
          'Model D\n(Closed Source)', 'Model E\n(Closed Source)']

# Accuracy on the three THUD tests (conceptual — from paper's findings)
# Higher = model correctly uses audio; Lower = model relies on vision only
shift_acc = [35, 42, 55, 48, 60]   # Temporal sync detection
mute_acc = [28, 33, 45, 40, 52]    # Sound existence detection  
swap_acc = [32, 38, 50, 44, 58]    # Audio-visual consistency

# Random baseline would be ~50% for binary questions
# A model that truly understands audio should be >85%

x = np.arange(len(models))
width = 0.25

fig, ax = plt.subplots(figsize=(12, 6))
bars1 = ax.bar(x - width, shift_acc, width, label='SHIFT (timing)', color='#3498db', alpha=0.8)
bars2 = ax.bar(x, mute_acc, width, label='MUTE (existence)', color='#e74c3c', alpha=0.8)
bars3 = ax.bar(x + width, swap_acc, width, label='SWAP (consistency)', color='#27ae60', alpha=0.8)

ax.axhline(y=50, color='gray', linestyle='--', linewidth=1.5, label='Random Guess (50%)')
ax.axhline(y=85, color='green', linestyle=':', linewidth=1.5, label='Competent Audio Understanding (85%)')

ax.set_ylabel('Accuracy (%)', fontsize=12)
ax.set_title('How Well Do Multimodal Models Actually Use Audio?\n(Lower = More Vision-Dependent)',
             fontsize=14, fontweight='bold')
ax.set_xticks(x)
ax.set_xticklabels(models, fontsize=10)
ax.legend(fontsize=10, loc='upper left')
ax.set_ylim(0, 100)

for bars in [bars1, bars2, bars3]:
    for bar in bars:
        height = bar.get_height()
        ax.annotate(f'{height}%', xy=(bar.get_x() + bar.get_width() / 2, height),
                   xytext=(0, 3), textcoords="offset points", ha='center', fontsize=9)

plt.tight_layout()
plt.savefig('fig10_thud_results.png', dpi=150, bbox_inches='tight')
plt.show()

The punchline: Most models scored barely above random chance on detecting audio interventions. They were giving correct-sounding answers by reading visual cues, not processing audio. When you swap a dog’s bark with piano music and the model still says “barking” — it’s hallucinating audio from visual context.

The Fix: Teaching Models to Actually Listen

The paper doesn’t just diagnose the problem. They propose a fix — a two-stage alignment recipe:

Stage 1: Intervention-derived preference pairs Create pairs of examples: one with correct audio-visual alignment, one with a THUD-style intervention. Train the model to prefer the answer grounded in actual audio over the one driven by visual guessing.

# Conceptual training pipeline

# Stage 1: Create preference pairs from interventions
preference_pairs = [
    {
        "prompt": "What sound is in this video of a dog?",
        "chosen": "Barking sounds",      # From the version with real dog audio
        "rejected": "Barking sounds",     # Model's visual-only prediction on swapped audio
        # The rejected answer LOOKS right but was given for the WRONG reason
    },
    {
        "prompt": "Is the audio synchronized with the video?",
        "chosen": "No, the sound is shifted",   # Correct audio-aware answer
        "rejected": "Yes, everything is in sync", # Visual-only prediction on SHIFTed input
    },
    # ... 10,000 such pairs
]
# Stage 2: Regularize with general video preferences
# Prevent the model from over-correcting (becoming audio-only, ignoring visuals)
# Use standard video QA preference data to maintain balanced multimodal understanding

The results of their fix: With just 10,000 training samples, average performance across the three THUD dimensions improved by 28 percentage points. And crucially, performance on general video and audio-visual QA benchmarks didn’t drop — it actually slightly improved.

Why This Matters for You

If you’re building anything with multimodal AI — video analysis, content moderation, accessibility tools, video search — this paper is a wake-up call.

The problem in plain English: When you ask a multimodal model “What’s making that sound in this video?” and it answers correctly, you assume it heard the sound. But it might have just seen the source and guessed. This works fine when audio and video are naturally aligned (as they usually are in training data). But it fails catastrophically when:

• Audio and video are out of sync (common in real-world footage)
• Audio is missing or corrupted (common in surveillance, old recordings)
• Audio contradicts the visual (adversarial scenarios, dubbed content)

# Real-world failure scenarios
failure_cases = {
    "lip_sync_detection": {
        "task": "Is this video dubbed?",
        "visual_model": "No, looks fine to me",  # WRONG - can't tell lips from audio are mismatched
        "audio_aware_model": "Yes, the audio doesn't match lip movements"
    },
    "content_moderation": {
        "task": "Is there violent audio in this video?",
        "visual_model": "No",  # Sees a calm scene, ignores aggressive audio overlay
        "audio_aware_model": "Yes, there are threatening vocalizations"
    },
    "accessibility": {
        "task": "Describe the sounds for a hearing-impaired viewer",
        "visual_model": "I see a concert",  # Ignores what the music actually sounds like
        "audio_aware_model": "A rock band is playing, drums are prominent, crowd is cheering"
    },
    "video_search": {
        "task": "Find videos where someone plays piano",
        "visual_model": "Returns videos with pianos visible (even if silent/unplayed)",
        "audio_aware_model": "Returns videos where piano music is actually audible"
    }
}

The Toy Demo: See the Clever Hans Effect Yourself

Here’s a minimal experiment you can run to see this effect (conceptual — requires a multimodal model API):

"""
Toy experiment to demonstrate the Clever Hans effect in multimodal models.
Requires: access to a multimodal LLM API (e.g., GPT-4V, Gemini, etc.)
The idea: Take a video, ask about audio. Then swap the audio and ask again.
If the model's answer doesn't change, it's reading visuals, not listening.
"""
# Pseudocode - adapt to your multimodal API
def test_clever_hans(model_api, video_path):
    """Test whether a multimodal model actually uses audio."""

    # Original video with its natural audio
    original_answer = model_api.query(
        video=video_path,
        prompt="What sound or music is playing in this video? Be specific."
    )

    # Create a version with swapped audio (e.g., replace with white noise or music)
    swapped_video = swap_audio(video_path, new_audio="classical_piano.mp3")
    swapped_answer = model_api.query(
        video=swapped_video,
        prompt="What sound or music is playing in this video? Be specific."
    )

    # Create a muted version
    muted_video = mute_audio(video_path)
    muted_answer = model_api.query(
        video=muted_video,
        prompt="Is there any sound playing in this video? What do you hear?"
    )

    # Analyze
    print(f"Original: {original_answer}")
    print(f"Audio swapped to piano: {swapped_answer}")
    print(f"Audio muted: {muted_answer}")

    # If answers are similar across all three → Clever Hans effect detected
    if similar(original_answer, swapped_answer):
        print("⚠️  CLEVER HANS DETECTED: Model answer unchanged after audio swap!")
    if "yes" in muted_answer.lower() and "hear" in muted_answer.lower():
        print("⚠️  AUDIO HALLUCINATION: Model claims to hear sound in muted video!")
# Helper functions (you'd implement these with ffmpeg)
# pip install moviepy
def swap_audio(video_path, new_audio):
    """Replace video's audio track with new_audio using moviepy."""
    from moviepy.editor import VideoFileClip, AudioFileClip
    video = VideoFileClip(video_path)
    new_audio_clip = AudioFileClip(new_audio)
    video = video.set_audio(new_audio_clip)
    output_path = video_path.replace('.mp4', '_swapped.mp4')
    video.write_videofile(output_path, logger=None)
    return output_path
def mute_audio(video_path):
    """Remove audio from video using moviepy."""
    from moviepy.editor import VideoFileClip
    video = VideoFileClip(video_path)
    video = video.without_audio()
    output_path = video_path.replace('.mp4', '_muted.mp4')
    video.write_videofile(output_path, logger=None)
    return output_path

The Bigger Picture

This paper fits into a growing body of work on model faithfulness — the question of whether models are solving problems the way we think they are. Just as:

• Shortcut learning showed that image classifiers use background textures instead of object shapes
• Annotation artifacts showed that NLP models exploit dataset-specific patterns
• The Clever Hans effect in this paper shows that multimodal models exploit visual-audio correlations without truly processing audio

The pattern is always the same: models find the easiest path to the right answer on training data, and that path isn’t always the one we intended.

# The pattern across AI history
shortcuts = {
    "Image Classification": {
        "apparent_skill": "Recognizes objects",
        "actual_strategy": "Matches background textures",
        "paper": "Shortcut Learning in DNNs (Geirhos et al., 2020)"
    },
    "NLP (NLI)": {
        "apparent_skill": "Understands language logic", 
        "actual_strategy": "Exploits hypothesis-only patterns",
        "paper": "Annotation Artifacts (Gururangan et al., 2018)"
    },
    "Multimodal (Video+Audio)": {
        "apparent_skill": "Understands audio in video",
        "actual_strategy": "Reads visual cues, ignores audio",
        "paper": "When Vision Speaks for Sound (Wen et al., 2026)"
    }
}

# The fix is always the same: intervention-based testing
# Don't just test if the model gets the right answer.
# Test if it gets the right answer FOR THE RIGHT REASON.

What to Take Away

1. Don’t trust multimodal model outputs blindly. If your application depends on actual audio understanding (not just visual inference), test it with the THUD approach — swap, mute, or shift the audio and see if the model’s answers change.
2. The 28-point fix is practical. The paper’s alignment recipe works with just 10K samples. If you’re fine-tuning a multimodal model for audio-sensitive applications, this is a proven approach.
3. This is a general principle. Whenever a model seems to understand something, ask: “Is it using the signal I think it’s using, or is it shortcutting?” The THUD framework (intervention-based probing) is a template you can apply to any modality.
4. The field is catching up. The fact that this paper got 158 upvotes and was published by UC Davis researchers means the community is taking model faithfulness seriously. We’re moving past “it gets the right answer” to “it gets the right answer for the right reason.”

Paper: “When Vision Speaks for Sound” — Xiaofei Wen et al., UC Davis, May 2026

arXiv: https://arxiv.org/abs/2605.16403

Code: https://github.com/rakanWen/wvs-code

Project page: https://rakanwen.github.io/when-vision-speaks-for-sound/

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!