Audio Noise Removal using Deep Learning

As video communication has become a part of everyday life, clear audio has become more important than ever. Whether in professional broadcasting, asynchronous video presentations, or real-time teleconferencing, background noise remains a persistent challenge. Historically, mitigating this noise relied on classical signal processing techniques. However, the paradigm has shifted dramatically toward deep learning.

This article explores why traditional methods fall short and examines the neural network architectures employed nowadays for this purpose.

The Limitations of Traditional Signal Processing

For decades, audio noise reduction relied on statistical and mathematical models, primarily utilizing techniques like Spectral Subtraction and Wiener filtering. These methods operate on a fundamental assumption: that noise is “stationary,” meaning its statistical properties (like frequency distribution and amplitude) remain relatively constant over time. Examples of stationary noise include the hum of an air conditioner or the static of a microphone preamplifier.

Classical methods estimate the characteristics of the background noise and then attenuate frequencies believed to contain mostly noise. While effective for simple, continuous hums, they fail significantly in real-world environments due to two primary reasons:

1. Phase Distortion and “Musical Noise”: Because human speech and background noise often share overlapping frequencies, aggressive spectral subtraction often removes parts of the human voice. This results in a hollow, robotic sound. Furthermore, errors in the subtraction process leave behind isolated narrow-band artifacts in the audio spectrum, creating a bubbling, synthetic artifact known as “musical noise”, which resembles random chirps or underwater bubbling sounds.
2. Non-Stationary Noise: Real-world interruptions, eg. a dog barking, a siren passing, or keyboard typing, are non-stationary. Traditional filters cannot predict these sudden changes, allowing the noise to pass through unaltered.

The Paradigm Shift: Deep Learning in Modern Communication

To overcome the inability of handcrafted signal processing methods to distinguish between voice and complex noise, the industry turned to Deep Learning. Modern communication platforms such as Zoom, Microsoft Teams, and Google Meet are understood to employ AI-based speech enhancement, though the exact production architectures are not publicly disclosed. Instead of relying on statistical assumptions, these deep learning models are trained to recognize the fundamental acoustic features of human phonetics.

• Masking and Spectrograms: Most of these algorithms operate in the frequency domain. They convert the raw audio waveform into a visual representation of frequencies over time (a spectrogram, explained below). A neural network analyzes this spectrogram and predicts a mask or set of gains for each time-frequency bin indicating how much of the signal should be retained.
• Lightweight Architectures: Because video streaming platforms require real-time processing with near-zero latency, they cannot rely on massive, computationally expensive models. Published systems use a variety of compact architectures including temporal convolution models, DeepFilterNet, and other lightweight speech enhancement networks. These models are heavily optimized to run on standard local CPUs rather than requiring dedicated cloud GPUs. Notably, not all of these systems abandon classical DSP outright. RNNoise, one of the earliest widely-used real-time models, is a hybrid: a small recurrent network estimates gains for Bark-scale critical bands (a classical psychoacoustic frequency division), while a traditional pitch filter handles noise between harmonics. It’s an early example of the DSP-plus-deep-learning symbiosis this article returns to in the conclusion.

The Mathematical Bridge: The Short-Time Fourier Transform (STFT)

Before a deep neural network can analyze the raw audio, it must undergo a fundamental mathematical conversion to create a Spectrogram. Raw digital audio exists as a one-dimensional time-series signal, which is simply a continuous stream of amplitude measurements over time. While this format is necessary for playback, it obscures the distinct patterns that separate human speech from background noise. To reveal these patterns and generate a two-dimensional spectrogram, engineers employ the Short-Time Fourier Transform (STFT).

Spectrogram. Image source: https://en.wikipedia.org/wiki/Spectrogram#/media/File:Spectrogram-19thC.png

The standard Fourier Transform is capable of identifying all frequencies within a signal, but it loses the element of time; it can tell you what frequencies exist, but not when they occurred. The STFT solves this by localizing the analysis. The process operates through three distinct steps:

1. Windowing: The continuous audio waveform is sliced into discrete, overlapping segments called “frames.” In real-time communications, these frames are incredibly short, typically ranging from 10 to 30 milliseconds. Overlapping the windows prevents data loss and minimizes boundary artifacts at the edges of each slice.
2. Transformation: A Fast Fourier Transform (FFT) is applied individually to each frame. The FFT mathematically decomposes the waveform of that specific millisecond window into its individual frequency components, revealing the energy present across discrete frequency bins and their respective amplitudes present at that exact moment.
3. Matrix Assembly: The frequency data from each transformed frame is stacked chronologically.

The output of the STFT is a complex matrix. The horizontal axis represents time (the progression of frames), the vertical axis represents frequency “bins,” and the data points within the matrix capture both the magnitude (volume) and the phase (wave alignment) of those frequencies.

When the magnitude data from this matrix is mapped to a color scale, it creates the visual image known as a Spectrogram. It is this structured, grid-like representation of frequency over time that allows Deep Neural Networks to “see” the audio and effectively draw masking boundaries between speech and non-stationary noise.

The Physics of Timing: Why Speech Enhancement Operates in Milliseconds

It is intuitive to assume that an audio processing window in the STFT algorithm described above should capture a complete word or phrase to be effective. After all, human listeners recognize words as distinct units of meaning. However, this assumption confuses linguistic interpretation with the underlying acoustic physics. At the digital signal processing (DSP) level, the Short-Time Fourier Transform (STFT) does not “understand” language; it only measures the physical vibrations of sound waves. The requirement for 10–30 millisecond windows is governed by three fundamental principles:

1. The Principle of Stationarity: The Fourier Transform relies on a mathematical assumption of “stationarity,” which posits that the signal’s statistical properties remain constant over the duration of the analysis window. If you were to process a one-second chunk of audio containing several words, the resulting spectral data would essentially “average out” all the sounds, creating a smeared, blurry frequency profile. To effectively distinguish between a voice and a distinct noise, like a sudden keystroke, the algorithm must isolate the specific moment that sound occurs. Larger windows destroy this temporal precision.

2. The Dynamics of Human Phonetics: Human speech is incredibly fast and transient. Consonants, specifically “plosives” like p, b, t, and k, are essentially micro-explosions of air that last for only 5 to 15 milliseconds. If an analysis window is too large (e.g., 200+ ms), these crucial building blocks of speech are completely swallowed by the surrounding acoustic data, making it impossible for a neural network to reconstruct them accurately. By keeping windows within the 10–30 ms range, the algorithm remains “fast enough” to capture these transient peaks.

3. The Time-Frequency Trade-off: There is a fundamental trade-off in wave physics often compared to Heisenberg’s Uncertainty Principle: you cannot achieve perfect precision in both time and frequency simultaneously.

• Narrow windows (e.g., 2 ms): Provide excellent time resolution, identifying exactly when a sound occurs, but they result in poor frequency resolution, making pitch data inaccurate.
• Wide windows (e.g., 100+ ms): Provide excellent frequency resolution, revealing precise pitches, but destroy the time data, making the output feel sluggish or smeared.

The “Goldilocks” Zone: Engineers have found that 10–30 milliseconds provides the optimal balance. This window is short enough to freeze the rapid physical changes of the human vocal tract (satisfying the stationarity requirement) but long enough to capture several complete cycles of the fundamental frequency of the human voice (which typically ranges from 85 Hz to 255 Hz). By operating at this millisecond scale, the algorithm treats audio as a series of physical snapshots, allowing for the precise, surgical removal of noise without damaging the structural integrity of the speech itself.

Implementing DeepFilterNet: A Practical Pipeline

The following Python implementation demonstrates how these advanced neural network architectures can be practically applied to video processing. The script below utilizes moviepy to handle the multimedia streams and leverages DeepFilterNet, which is a highly efficient, open-source deep learning framework for the acoustic processing.

[embed]OpenNLP/Video Intelligence/video_noise_reduction.ipynb at main · atmabodha/OpenNLP This repository is for storing various open source projects on Natural Language Processing [NLP]. - OpenNLP/Video…github.com

The code above acts as a wrapper, decoupling the audio layer and passing it to the deepFilter subprocess. But what is occurring within the DeepFilterNet algorithm during Step 2?

DeepFilterNet stands out from earlier Deep Learning models by focusing on computational efficiency and perceptual audio quality. It achieves this through a two-stage processing architecture:

• ERB-Scaled Coarse Filtering: Human hearing does not perceive all frequencies equally; we are far more sensitive to changes in lower frequencies than in higher ones. DeepFilterNet mimics this biological reality by grouping frequencies into Equivalent Rectangular Bandwidth (ERB) bands. In its first stage, the neural network calculates coarse gains (volume adjustments) for these ERB bands. By processing grouped bands rather than every single frequency bin, the model vastly reduces its computational load, allowing it to run swiftly on standard hardware.
• Complex Deep Filtering: While coarse filtering removes the bulk of the noise, it is not precise enough to preserve the micro-details of the voice. The second stage utilizes “Deep Filtering” (DF). Unlike older models that only adjust the magnitude (volume) of frequencies, DeepFilterNet also operates on the phase (the alignment of the sound waves), and unlike a simple mask, it does so using a short filter applied across several neighboring time frames per frequency bin, not just the current instant. This temporal context is what lets it reconstruct the fine periodic structure of a voice (the harmonics that give speech its natural, non-robotic character) rather than just gating volume frame by frame. To keep this efficient, DF is applied only to the lower frequency bins where speech periodicity carries the most energy; the coarse ERB gains from stage one handle the rest of the spectrum. Together, the two stages eliminate the robotic artifacts characteristic of older AI models and traditional Wiener filters.

DeepFilterNet model (image from the paper)

What was DeepFilterNet trained on?

Unlike massive Large Language Models (LLMs) that require trillions of tokens, DeepFilterNet’s training data is focused on acoustic diversity. It is trained in a supervised manner on large datasets consisting of thousands of hours of “noisy-clean” audio pairs. Key components of its training data include:

• Clean Speech Databases: Massive corpora of read and spontaneous speech (e.g., LibriSpeech or similar public speech datasets) serve as the “ground truth.”
• Diverse Noise Profiles: The model is trained on a wide variety of non-stationary noises, keyboard typing, office chatter, traffic, wind, and fan hums, to ensure it doesn’t just learn to remove one type of “static” sound.
• Acoustic Simulations: Because real-world recording environments are rarely perfect, the training data is augmented with simulated room acoustics (reverberation) and varying signal-to-noise ratios (SNR). This teaches the model to focus on the speaker’s voice even in echoey, small rooms or when the noise level is nearly as loud as the speaker.

How does it learn? DeepFilterNet uses a supervised learning approach where it calculates the difference between its output and the original “clean” audio. By minimizing the loss between these two, it learns to predict complex filter coefficients that reconstruct the speech signal, rather than just “guessing” what the clean signal should look like.

[embed]DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep… Complex-valued processing has brought deep learning-based speech enhancement and signal extraction to a new level…arxiv.org

Reconstructing the Waveform: The Inverse STFT

Once the neural network has analyzed the spectrogram, the data still exists entirely in the frequency domain. You cannot play a matrix of frequencies through a speaker; it must be converted back into a continuous, one-dimensional audio waveform. This final reconstruction step is achieved using the Inverse Short-Time Fourier Transform (iSTFT).

Because the original STFT sliced the audio into overlapping windows to capture temporal detail, the iSTFT must carefully stitch these cleaned frames back together. Using a technique called “overlap-add,” the overlapping boundaries of each frame are cross-faded. This prevents boundary artifacts, ensuring the resulting audio is a smooth, continuous stream rather than a series of disconnected clicks and pops.

Only after this inverse transformation is successfully executed does the neural network’s output become a listenable, studio-quality .wav file, ready to be muxed back into the final video container.

The Latency Barrier and the Whisper Paradox

Given the recent leaps in artificial intelligence, readers may naturally wonder:

Why not utilize massive, state-of-the-art models like OpenAI’s Whisper for real-time video noise removal?

The answer lies in a combination of architectural purpose and rigid computational constraints. Whisper is fundamentally an Automatic Speech Recognition (ASR) system designed to output text, not an audio-to-audio reconstruction tool. Interestingly, Whisper is robust to background noise for the purpose of recognizing speech, but not for the reason one might assume. Research probing Whisper’s internal representations has found that its encoder does not actually learn to strip noise out of its latent embeddings since the embeddings still encode rich information about the background sound itself. Whisper appears to recognize speech conditioned on the noise type rather than by filtering the noise out mathematically. Regardless of mechanism, this doesn’t help for enhancement: because Whisper’s decoder is trained strictly to output text tokens, there is no pathway that reconstructs any version of the speech, clean or otherwise, back into a playable audio waveform.

[embed]Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour…arxiv.org

Furthermore, even if researchers were to successfully adapt a Whisper-sized transformer architecture to output clean audio, it would immediately fail the demands of live communication. Practical speech enhancement models often process audio in frames of roughly 10–30 ms so that overall system latency remains low enough for natural conversations. These rigid temporal and hardware limitations explain why real-time systems must eschew massive, billion-parameter transformers like Whisper. Instead, they typically utilize highly optimized and lightweight architectures specifically designed for sub-millisecond execution.

Conclusion

Deep learning has transformed speech enhancement, but it did not replace classical DSP. The narrative that AI replaces traditional Digital Signal Processing (DSP) is a misconception; in reality, modern high-performance systems rely on a symbiotic relationship between the two.

Modern AI models, such as those powering top-tier teleconferencing platforms and frameworks like DeepFilterNet, do not operate in a vacuum. They depend entirely on the foundational mathematical principles of classical DSP, such as the Short-Time Fourier Transform (STFT), to convert raw, time-domain audio into a structured, frequency-domain format that neural networks can interpret.

DSP provides the mathematical framework and deep learning provides intelligent decision-making.

While traditional models were historically constrained by rigid statistical assumptions regarding noise consistency, modern systems use data-driven acoustic modeling to overcome those limitations. By leveraging classical techniques like STFT for input representation and merging them with the adaptive pattern-recognition capabilities of neural networks, these algorithms achieve what neither could accomplish alone. They interpret audio through spectrograms, mimic human perceptual scales like ERB, and manipulate complex phase data to surgically isolate speech from chaotic acoustic environments. This integration — where DSP provides the mathematical framework and deep learning provides the intelligent decision-making — is the engine behind the unprecedented clarity we experience in digital communications today.