Lip Reading - Building a Visual Speech Recognition App

Team Members: Manikanta Allanki, Tarun Kumanduri, Shashank Boppana, Ibrahim Ahmed, Paul Boadu Asamoah Project Supervisor: Prof. Maskim Tsetovat

Speech is the primary source of communication. Over the years, we have advanced our technology so much however we have been dependent on audio for lots of communication. There are tons of situations where you just can’t rely on sound. Maybe you’re in a noisy room, a video call glitches out, or someone simply can’t hear well. In all these cases, lip reading becomes a superpower. But what if we could teach machines to have that same skill?

What did we do?

Have you ever imagined a system that could understand what you’re saying, without hearing a single word? That’s exactly what we want to build. In this project, we explored the world of visual speech recognition, where the model learns to read lips from video frames, not audio. Our approach is inspired by LipNet, a pioneering model that translates silent videos of lip movements into text using deep learning. We combined 3D CNNs to capture frame-level motion, BiLSTMs to understand temporal patterns, and CTC loss to align the video with the text, so no manual labeling was needed. From dataset processing to model deployment, we’ve built a full pipeline that brings silent communication to life.

Why This Matters

Lip reading systems have diverse applications:

• Accessibility: Helping the hearing-impaired communicate better
• Military & Aerospace: Silent speech interfaces for extreme environments
• Surveillance: Decoding conversations from muted footage
• Noise-Robust Interfaces: For factories, hospitals, and public transport

Related work

Our project draws strong inspiration from LipNet, one of the first deep learning models to perform end to end sentence level lip reading. LipNet introduced the idea of using 3D Convolutional Neural Networks (3D CNNs) to capture spatiotemporal visual features, followed by Bidirectional GRUs for sequence modeling. Crucially, it used Connectionist Temporal Classification (CTC) loss to align predicted text with video frames without requiring manual annotation of each frame, removing a major bottleneck in training.

While LipNet laid the groundwork, we took a slightly different approach in our implementation:

• Instead of GRUs, we used a Bidirectional LSTM layer for sequence learning. LSTMs are often better at capturing longer-term dependencies and can enhance model performance when trained carefully.
• We built our model pipeline using PyTorch and made it modular and easier to deploy.
• For deployment and demonstration, we integrated our model with Streamlit, allowing users to upload a silent video and view the predicted text in real-time through a clean web interface.

By reimagining the LipNet architecture with these improvements, our goal was to create a system that’s not only accurate but also practical for real-world applications and experimentation.

Where did we get the Data from?

To train and evaluate our lip-reading model, we turned to the GRID Corpus Dataset, one of the most trusted and widely used benchmarks in the field of visual speech recognition. It’s not just a random collection of videos, it was specifically designed to support research like ours.

Each sample in this dataset is a short 3-second clip, containing exactly 75 frames of a person speaking a sentence. What makes it unique is the structured sentence format phrases like “bin blue at f two now” that are drawn from a fixed grammar of command words, colors, letters, digits, and adverbs. This consistent structure helps in building models that can focus purely on visual speech cues.

There is data from 34 different speakers, which brought in natural variations in speaking styles, lip shapes, and movement that are all critical for training a generalizable model. For each speaker there are 1,000 video samples, ensuring a rich and diverse training set that mimics real world variability while staying within the constraints of controlled, labeled data.

By training on this dataset, our model learns to read lips not just from a single person, but from a wide range of faces helping it move one step closer to real-world application.

Why impact it can create?

Lip reading systems go far beyond just being an exciting application of deep learning, they have the potential to solve real problems in meaningful ways.

In accessibility, such systems can dramatically improve communication for people who are deaf or hard of hearing. Rather than relying solely on sign language or interpreters, individuals could have real-time translation of lip movements into text, enabling more natural conversations in everyday life.

In high-stakes environments like military operations or aerospace missions, where speaking aloud isn’t always possible or safe, silent communication systems powered by visual speech recognition can offer a discreet and efficient alternative.

In the context of security and surveillance, lip reading models could provide valuable insights from video footage that lacks audio something that’s particularly relevant for forensic investigations or when reviewing archived recordings.

Finally, in noisy environments such as factories, hospitals, or public transit systems, traditional voice-based interfaces struggle to function effectively. A system that can “understand” speech visually can open new doors for robust human-machine interaction, regardless of background noise.

The broader implication is clear: by teaching machines to interpret speech without sound, we move one step closer to creating technology that adapts to us not the other way around.

Work Flow

Fig. 1. Workflow

1.Data Processing

The first and most crucial step in building a lip reading system is making sure the model sees clean and consistent visual input. Raw videos, no matter how well recorded and often contain a lot of noise, variation in lighting, and extra facial information that the model doesn’t need. Our goal in this stage is to narrow things down to just the mouth movements and give the model a clean canvas to learn from.

Step 1: Extracting Frames from Video We start with short video clips of people speaking. These videos are sliced frame by frame by essentially creating a flipbook of lip movements. This frame extraction allows us to analyze each moment of speech individually. Tools like OpenCV help automate this process efficiently.

Step 2: Detecting and Cropping the Mouth Region Once we have the frames, we use a face detection model that is MTCNN in our case it is to locate key facial landmarks such as the eyes, nose, and most importantly, the mouth. We then crop out just the mouth region from every frame. By focusing only on this part, we remove distractions from other parts of the face and background, allowing the model to pay full attention to lip shapes and movements.

Step 3: Resizing and Denoising Different videos come with different resolutions, lighting conditions, and angles. To standardize the input, we resize all cropped mouth frames to a fixed size, usually something like 150x50 pixels. We also apply normalization and denoising techniques to reduce lighting variation and remove background clutter, ensuring that every frame the model sees is as clean and consistent as possible.

This preprocessing pipeline ensures that our model learns from high-quality, mouth-only data by setting a solid foundation for accurate visual speech recognition.

2. Feature Extraction and Sequence Modelling

Once we’ve preprocessed and isolated the mouth movements from each video frame, the next step is teaching the model to actually understand what’s being said. This is where feature extraction and sequence modeling come into play.

We begin with a series of 3D Convolutional Neural Networks (3D CNNs). Unlike traditional 2D CNNs, which only look at spatial features in a single image, 3D CNNs analyze both space and time simultaneously. This means they don’t just see how the lips are shaped in one frame but also they also track how those shapes change across multiple frames. It’s like giving the model short video clips of lip motion, rather than single stills.

In our setup, we use three 3D convolutional blocks, each followed by batch normalization, ReLU activation, and max pooling to reduce spatial dimensions while keeping the temporal resolution intact. By the end of this stage, the model has a compact, information-rich representation of how the lips move across time.

Next, we pass this temporal sequence into a Bidirectional LSTM (Long Short-Term Memory) network. This is a type of recurrent neural network that excels at processing sequences. What makes the bidirectional version especially powerful is that it looks at the data both forward and backward in time. That way, the model doesn’t just guess what’s being said based on past frames, also it also considers what comes after. This is crucial for natural speech, where the context of a letter or word often depends on the surrounding sounds.

To connect the visual features to actual characters, we apply a dense (fully connected) layer followed by a softmax function. This gives us a probability distribution over a vocabulary of characters for every frame.

3. Inference: Translating Lip Movements to Text

After the model processes the video through 3D CNN and BiLSTM layers, it generates a sequence of probability distributions like one per frame over a set of possible characters. But instead of needing perfectly aligned labels for every frame (which would be incredibly difficult for lip reading), we use CTC decoding to interpret this sequence.

CTC, or Connectionist Temporal Classification, is designed to handle sequences where the input and output lengths don’t line up neatly. It decodes the most likely sequence of characters while ignoring blank predictions and repeated letters — for example, a raw prediction like — hhheeellllooo — is cleaned up to give a meaningful result: “HELLO”.

In essence, inference is the final step where the model takes all of its learned understanding of lip motion and translates it into text without ever hearing a sound. It’s the moment where silent lip movements become readable speech.

4. Serving the Model: Real-Time Lip Reading Through a Web Interface

Once the model is trained and evaluated, the final step is to make it accessible and interactive. For this project, we deployed the system using a user-friendly web interface built with Streamlit, allowing anyone to experience lip reading in action.

Users can upload a short video clip directly through the app. Behind the scenes, the video is broken down into frames, the mouth region is extracted, and the same preprocessing pipeline is applied. The model then predicts the spoken content, and the result is displayed on the screen.

The interface also shows a short animation of the extracted lip region, helping users visualize what the model is analyzing. This seamless interaction between machine learning and front-end design allows for a transparent, educational, and interactive experience.

Architecture and Model Design: From Silent Frames to Spoken Words

Our goal was to develop a model capable of predicting a complete sequence of characters essentially, words that are directly from silent video clips of a speaker’s lip movements, without needing frame-level annotations. This is particularly challenging because it requires the model to not only understand spatial features but also learn how these features evolve over time.

The input to our model consists of short video clips, each lasting about three seconds and containing 75 frames. Each frame captures the cropped region of the speaker’s mouth at a resolution of 50 by 150 pixels with three color channels (RGB). The shape of the input is therefore (75, 50, 150, 3), where the first dimension captures the temporal aspect.

1. Spatio-Temporal Feature Extraction with 3D Convolutional Layers

To begin, we use 3D Convolutional Neural Networks (3D CNNs). Unlike traditional 2D CNNs that only capture spatial features, 3D CNNs allow us to learn both spatial and temporal features simultaneously. This is crucial for lip reading, where the way lips move across time carries vital information.

Our architecture includes three Conv3D blocks with increasing filter depth (128 → 256 → 256), each followed by batch normalization, ReLU activation, and spatial max pooling. We also apply a SpatialDropout3D layer to prevent overfitting. These layers compress the spatial dimensions while preserving the time dimension, resulting in a compact yet informative representation of lip movements across frames.

2. TimeDistributed Flattening for Sequential Input

Once the spatio-temporal features are extracted, we apply a TimeDistributed Flatten layer. This operation flattens the spatial feature map of each frame into a single vector, while maintaining the time structure. In effect, we convert our feature maps into a sequence of 75 vectors, one for each frame, ready for sequential modeling.

3. Temporal Modeling with Bidirectional LSTMs

Understanding a spoken word from visual cues requires more than interpreting individual frames. It demands context: how lip positions evolve based on what came before and what comes next. To capture this, we use two stacked Bidirectional LSTM layers, each with 512 hidden units. These layers process the sequence in both forward and backward directions, helping the model understand dependencies in the full temporal span of the clip.

4. Character Prediction via Dense Layer and Softmax

The LSTM outputs are then passed through a fully connected Dense layer, where each time step outputs a probability distribution over a fixed vocabulary of characters (plus one special token called the CTC “blank” label). The softmax activation ensures that the model outputs meaningful probabilities at each time step.

5. Sequence Alignment Using CTC Loss

One of the key innovations in our model is the use of Connectionist Temporal Classification (CTC) loss. Unlike conventional approaches that require precise alignment between video frames and the characters being spoken, CTC allows the model to learn this alignment on its own.

CTC helps in two important ways. First, it allows the model to ignore repeated predictions, so even if a frame sequence predicts multiple ‘l’ characters, the final output can still be “hello.” Second, it supports variable length inputs and outputs, which is essential when different speakers pronounce words at slightly different speeds.

Training Progress & Observations

To evaluate the learning capability of our lip reading model, we tracked both the predicted outputs and the loss values across epochs.

In the early stages of training, the model struggled to decode even simple phrases. For instance, at Epoch 1, the original sentence was:

• “place red with a eight please” Prediction: “l e e e o”

This highlights the model’s initial difficulty in mapping lip movements to meaningful character sequences. However, as training progressed, both the training loss and validation loss steadily decreased, indicating effective learning and generalization.

By Epoch 29, the model had improved significantly:

• “place red with n four now” Prediction: “place red with four now”

The model accurately captured the structure of the sentence and most of the words, demonstrating its ability to generalize on seen speakers.

Later in the training, we observed predictions that were almost indistinguishable from the ground truth. In Epoch 87, for example:

• “place blue by m four please” Prediction: “place blue by m four please”

This nearly perfect alignment showcases the effectiveness of our 3D CNN + BiLSTM + CTC pipeline in decoding silent video into coherent text.

However, it’s important to note occasional misclassifications, such as:

• “lay red in u zero now” Prediction: “lay red at u zero now”

Such errors, although minor, reflect the nuances of lip movement and how easily certain phonemes can be confused without audio. Still, the overall trend shows the model’s increasing ability to interpret subtle lip motions accurately.

These examples provide qualitative evidence that the model not only learns to align video to characters via CTC loss, but also captures the temporal context necessary for accurate lip reading.

Model Performance and Evaluation

To evaluate the effectiveness of our lip-reading model, we used Word Error Rate (WER) as the primary metric. WER is a widely accepted standard in speech recognition tasks and reflects the number of substitutions, insertions, and deletions needed to match the predicted output to the actual transcript. The formula is:

WER = (Substitutions + Insertions + Deletions) / Total Words

In simpler terms, a lower WER means better model performance, as fewer errors are made in transcription.

Validation Set — WER ≈ 19.6%

This set contains clips from the same speakers used during training, but includes different video samples. The goal here was to tune the model’s hyperparameters such as learning rate and dropout, without overfitting to the training data. A WER around 19.6% shows the model is learning to generalize reasonably well within familiar speaker contexts.

Test Set (Seen Speakers) — WER ≈ 18.9%

This portion of the test data also comes from the original six speakers used during training, but includes entirely new videos. The slightly better WER of 18.9% demonstrates that the model performs consistently well when encountering fresh samples from speakers it has already learned from.

Test Set (Unseen Speakers) — WER ≈ 34.7%

This is the most challenging benchmark. It consists of videos from speakers the model has never seen before during training. With a WER of 34.7%, we observe a noticeable drop in performance, which is expected. It reflects the difficulty of adapting to new lip shapes, speaking speeds, and articulation styles, all of which vary greatly across individuals.

Despite this, the model shows promising generalization ability, especially considering that no speaker-specific fine-tuning was applied. These results indicate that with additional data diversity and augmentation, speaker-independent performance could improve significantly.

Interpretation

Our model shows promising performance when it comes to speakers it has already seen during training. With a Word Error Rate (WER) under 20%, it’s able to transcribe silent video clips with a good degree of accuracy for familiar faces.

However, when tested on entirely new speakers — those it hasn’t encountered before — the WER rises to around 35%. This drop in performance isn’t surprising. Every person has a unique way of speaking: lip shapes, motion patterns, articulation speed, and even subtle facial features vary significantly from person to person.

What this tells us is that the model has learned to pick up on speaker-specific cues quite well, but it still struggles to generalize to unfamiliar faces. Building better speaker-independent models remains a key challenge in visual speech recognition, and this opens the door for future work in enhancing generalization through data augmentation, domain adaptation, or more diverse training sets.

Conclusion

This project demonstrates the potential of deep learning to bridge the gap between vision and speech. We successfully developed an end-to-end lip reading system that predicts spoken words from silent videos that is a task once considered extremely difficult due to the subtlety of lip movements and variations between speakers.

Our approach combined multiple powerful components: MTCNN for accurate mouth localization, 3D CNNs to capture how lips move across time, BiLSTM layers to understand the sequence of those movements, and CTC loss to align those sequences with meaningful text without requiring frame-level labels.

To bring this work closer to real-world use, we wrapped our model into an interactive web app using Streamlit, making it easy for anyone to upload a video and see the system in action.

Future Work

While our model shows strong performance on familiar speakers, there’s room to grow, especially when it comes to handling diverse, unseen faces and real-world variability. Some promising directions for future work include:

• Training on more diverse and challenging datasets like LRS2 and LRS3 to improve generalization and vocabulary coverage.
• Incorporating attention mechanisms to help the model focus on the most informative frames in a video.
• Extending the system to full sentence-level predictions, enabling smoother and more natural transcriptions.
• Building an interview monitoring tool that detects lip-sync mismatches or speech inconsistencies during recorded conversations.
• Improving speaker-independence through data augmentation, domain adaptation, or speaker normalization techniques.

In essence, this project takes a meaningful step toward machines that can “hear” even in silence, systems that could make communication more inclusive and resilient across many real-world settings.