Building a Nepali Lip Reading System Using Facial Mesh Data with ML5.js: A Research Project

Foreword

This article is part of an ongoing research project, and I plan to keep it updated as I make progress. As I continue to refine the methods, collect more data, and train models, I’ll be adding new findings, challenges, and breakthroughs. If you’re interested in following this journey and the evolution of the Nepali lip-reading system using facial mesh data, check back for updates. I’ll be documenting the steps in real time, and your insights or feedback are always welcome as we move forward.

Introduction

Last month, I was brainstorming ways to generate lyrics dynamically or, at the very least, sync the SRT file dynamically for an application I’m working on. During this process, I stumbled upon an intriguing idea: what if I could build a system that reads speech directly from facial data, specifically the movement of the lips and the positioning of the tongue? This led me down the path of lip-reading, but with a unique twist — using facial mesh data from ML5.js to decode spoken language.

The goal of this project is to develop a method to predict Nepali speech through facial mesh data and movement alone. By analyzing how the shape of the face — especially the lips, jaw, and other facial expressions — corresponds to different sounds in Nepali, I aim to build a system that can generate or predict words spoken, purely based on real-time facial expressions and movements.

Phase 1: Preliminary Tests and Hypotheses

To begin, I needed to create a solid foundation. Using ML5’s facial mesh library, I captured real-time data of me speaking various syllables in Nepali. This data included points on the face such as the eyebrows, lips, cheeks, and jawline. My initial focus was on the movement of the upper and lower lips.

For the first tests, I concentrated on a set of basic phonetic syllables in Nepali, like “ka”, “kha”, “ga”, “gha”, and “cha”. These sounds provided a starting point for understanding how facial movements correspond to specific sounds in the Nepali language.

However, I soon realized that relying only on lip mesh points wasn’t sufficient. While these points provided important data, they couldn’t give a complete picture. I needed to account for additional factors like the rise and fall of the eyebrows, the position of the jaw, and subtle tension in the tongue. Consequently, I expanded the data collection to include a broader range of facial markers, focusing not just on the lips, but on the entire face.

Points Used in Data Collecting

Points Used in Data Collection

It’s important to note that, at this stage, I am not tracking tongue movements, as there aren’t readily available models that track tongue movements yet. The data collected currently includes facial mesh points related to the eyes, eyebrows, lips, cheeks, and jaw. These points are what I’m using to analyze facial movements and correlate them with speech patterns.

Phase 2: Data Collection and Dynamic Tracking

Once I recognized the limitations of static data collection, I transitioned to dynamic tracking. Instead of just capturing facial data at single moments in time, I began to focus on tracking the changes in facial expressions as the speaker moved through different syllables and words. This meant continuously recording how the shape of the lips, eyebrow movements, and jaw position shifted as words were spoken in real-time.

I created a front-facing data collection system that captures video input from subjects speaking in Nepali. Each participant would speak selected syllables, and their facial movements would be recorded, with data points tracked continuously throughout the speech. I’ve been specifically focusing on lip position, cheek position, and jaw position. By collecting data over time, I’ve been able to identify how different movements contribute to the pronunciation of specific words.

The data collection page includes these sections:

1. Choose the syllables you want to record currently

You can select syllables from a predefined set of Nepali syllables.

Select Nepali Syllables

2. Record the syllables

Each syllable is recorded for 600 milliseconds, which is enough to capture a single phonetic sound.

3. Data Validation

After recording, the data is validated by you, the volunteer, to ensure both accuracy and consistency. You can hover over the recordings in the live app to play them, allowing you to also practice lip reading. It’s a fun way to contribute while improving your skills at the same time!

Validate the recorded data

The data collection system is available here.

Next Steps and Challenges Ahead

The most significant hurdle right now is the amount of data needed for training the model. To ensure robust predictions, I’ll need at least 1,000 recordings of each syllable. Since the Nepali language contains 444 syllables (including consonants, vowels, and independent vowels), this means I will need a dataset consisting of around 440,000 data points.

Once the data collection phase is complete, the next challenge will be optimizing the model for accuracy in various conditions, including noisy environments and different speakers with varying facial types and accents. Human facial expressions and speech patterns are highly variable, which means my next step will be to implement a more advanced neural network that can handle these nuances more effectively.

Conclusion:

This project has taken me deeper into the world of facial recognition, machine learning, and linguistics than I ever anticipated. From the initial idea of generating lyrics dynamically, I’ve now moved into the realm of predicting spoken language purely from facial mesh data. There is still much work to be done, but every new step brings the possibility of a system that can “read” speech without ever hearing a word. A future where the face, the ultimate canvas of human expression, speaks in ways we are just beginning to understand.