Ilya’s 30u30, Part 19: DeepSpeech2

Hello, all you Sutskevites!

We are back with Part 19 of my “Ilya’s 30u30” series, which means I only have one more article to go before I hit my goal for the year.

This is huge, because it proves that if you set a realistic goal, work consistently, and publicly embarrass yourself, you too can almost accomplish something by August.

Today’s paper is DeepSpeech2, from 2015.

And I have to say, this one is either pretty simple, or I am missing something obvious and one of you is going to politely destroy me in the comments.

The basic idea is this:

Back then, speech recognition still used a lot of hand-made stuff. People would take audio, study it, and write programs that looked for little clues in the sound. Like, “This pattern probably means this person is making a B sound,” or “This curve probably means an O sound.”

That is very smart, but it is also a lot of work. You are basically trying to teach the computer how to listen by writing a giant rulebook.

DeepSpeech2 said: what if we do not write the giant rulebook?

What if we just let the AI learn from the sound directly?

Sound Can Become a Picture

One thing I have always liked about speech recognition is that sound can be turned into an image.

That image is called a spectrogram.

A spectrogram is basically a picture of sound. It shows which sound frequencies are happening at each moment. So if someone says “dog,” you can turn that sound into a weird stripey image.

Sadly, it does not turn into an actual picture of a dog.

You do not say “dog” into a microphone and get a golden retriever wearing sunglasses.

Instead, you get something that looks like a submarine operator is about to say, “Captain, I am detecting jazz.”

Evidence of undwerater Jazz

Years ago, I had an idea.

I thought: what if I take a bunch of music clips, turn them into spectrograms, train a GAN on those images, and then turn the fake spectrograms back into audio?

I thought I had invented AI music.

So I did it. I gathered the clips, converted them, trained the GAN, waited forever, adjusted things, trained it again, and eventually got audio out.

It sounded terrible.

Not “experimental jazz” terrible. More like “random beeps and boops” terrible.

Around that time, OpenAI released a much better music generator, and I was emotionally destroyed for like a week. That is one of the underrated emotional experiences in AI research: you spend a ton of time building something, it crawls out of the lab covered in slime, and then OpenAI releases a beautiful demo that makes your project look like it was built by raccoons in a parking lot.

But the idea itself was not crazy.

Turning sound into pictures has been around for a long time. The question is what you do after that.

The Old Way Was a Giant Speech Machine

Before systems like DeepSpeech2, speech recognition often had a bunch of steps.

First, you would process the audio.

Then you might try to figure out the small sound pieces, called phonemes.

Then you might map those sounds to possible words.

Then you might use another system to guess which sentence makes the most sense.

So instead of one AI model going from sound to text, you had this big machine made of many smaller machines.

And again, this was not dumb. It was actually very impressive. If you do not have giant neural networks and huge piles of data, you need humans to help the computer. You need smart people to say, “Look here. These are the parts of the sound that matter.”

DeepSpeech2’s big move was to make the whole thing more direct.

Put in audio.

Get out text.

Not “first build a huge speech factory with seventeen little stations.”

Just: here is the sound, here is the correct sentence, now you learn, little AI.

The Model Learned the Middle Part by Itself

People mumble. People stretch words. People cough. People say “doooooog” because they are either calling their pet or having a spiritual moment at a barbecue.

So instead of forcing humans to label every tiny slice of audio, DeepSpeech2 lets the model figure out a lot of that middle stuff by itself.

That is the deep learning magic trick. You do not tell the model exactly what to look for. You give it examples and say:

“Here is the sound. Here is what it should say. Please figure out the mysterious goo in the middle.”

And the model says:

“Fine, but I will need a concerning amount of data and electricity.”

The GPU Part: Eight Copies Learning Together

The paper also talks a lot about training speed.

That matters because the model was big, the dataset was big, and GPUs in 2015 were not as powerful as they are now. Training one version of the model could take a long time.

And long training runs are painful because AI research is mostly:

Have an idea.

Train the model.

Watch it fail.

Change something.

Train it again.

Watch it fail in a slightly more interesting way.

Repeat until either the model works or you quit and become a fish salesman.

So the team used multiple GPUs to speed things up.

The idea is called data parallelism, which sounds fancy, but it is pretty simple.

Imagine you have eight GPUs.

Each GPU gets its own copy of the same model. Then you take a big batch of audio clips and split it into eight smaller groups.

Each GPU listens to its own group of clips. Each one calculates how wrong the model was. Then each one says, “Based on what I saw, I think we should change the model like this.”

So now you have eight GPUs with eight opinions.

GPU 1 says, “Move this weight up.”

GPU 2 says, “Move it down.”

GPU 3 says, “Meh. Up would be nice.”

Then the system averages all those opinions together.

That average becomes the update.

Then every GPU gets the same updated model again, and they all move on to the next batch of audio clips.

That is data parallelism.

Same model, different data, shared lesson.

It is basically a committee meeting, except everyone is a graphics card.

CTC: A Horrible Name for a Cool Idea

DeepSpeech2 also used something called Connectionist Temporal Classification, or CTC.

This name is ridiculous.

Connectionist Temporal Classification sounds like something a lawyer says in a sci-fi courtroom.

“Your Honor, under the Connectionist Temporal Classification Act, my client is clearly hearing minds from the past.”

But the idea is actually really cool.

CTC helps with this problem:

We know the final word is “dog,” but we do not know exactly which tiny slice of audio should be labeled D, which should be O, and which should be G.

So CTC says, “That is okay. Let the model guess at every time step, and then we will clean up the guesses afterward.”

For example, maybe the model hears someone say “dog” and gives us this:

D, D, blank, O, G

The blank means “nothing new here.”

Now CTC cleans it up.

First, it smashes repeated letters together:

D, blank, O, G

Then it removes the blank:

D, O, G

And now we have:

DOG

That is the whole idea.

The model does not need to know the exact perfect timing of each letter. It just needs to produce something that can be cleaned up into the right answer.

That is really flexible.

It is basically saying:

“You can be messy in the middle, as long as the final thing comes out right.”

Which, honestly, is also my entire cooking process.

Why the Blank Token Matters

The blank token is important because the model should not be forced to invent a letter at every moment.

Sometimes there is silence.

Sometimes the person is between sounds.

Sometimes the audio is just weird.

Without a blank option, the model has to say something every time, even when nothing useful is happening. That is how you end up with random extra letters.

The blank token lets the model say:

“Nothing to report.”

And I respect that deeply. I wish I could follow it.

It also helps with repeated letters. For example, if someone says a word with two of the same letter, the model needs a way to tell the difference between “this is the same sound continuing” and “this is the same letter happening again.”

That sounds like a small detail, but these small details are the difference between speech recognition working and your phone turning “book” into “bok,” like it is naming IKEA furniture.

The Loss Function Is More Like a Little Program

What I found interesting about CTC is that it is not just a simple loss function.

When you first learn about neural networks, loss functions usually sound pretty basic.

For regression, it is something like:

Subtract the prediction from the answer, square it, average it, cry.

For classification, you use cross-entropy, pretend you fully understand logits, and move on.

But CTC feels more like a little program.

It looks at all the guesses over time. It allows blanks. It handles repeated letters. It checks whether the messy stream of guesses can become the correct final word.

That is cool to me.

It made me realize a loss function does not have to be just “how wrong was the answer?” It can include a whole process.

That makes me want to try even more ridiculous loss functions.

Like what if the model gives an output, then we run that output through a Rube Goldberg machine, then a tiny plastic boot kicks a marble down a ramp, then a cage falls on a mouse, and that is how we calculate the loss?

If CTC is legal, then the boot should be legal.

Obviously, in real life, you cannot put the board game Mouse Trap inside PyTorch and expect it to scale.

But conceptually, I love it.

The model does not always have to be judged directly. Sometimes the model can be judged by what happens after its answer goes through a process.

That feels powerful.

The Big Takeaway

To me, the big lesson of DeepSpeech2 is that you can replace a huge complicated speech pipeline with something much more direct.

Instead of humans carefully building every little step, the model learns a lot of the hard stuff by itself.

But this only works if you have the right ingredients:

A lot of data.

A big enough model.

A smart loss function.

Enough GPU power.

And a training setup that does not take forever every time you want to test an idea.

That last part matters more than people think. AI research is not just one genius having one beautiful idea and then everything works. A lot of it is being able to try enough ideas fast enough that you do not die of old age before the good one appears.

DeepSpeech2 is cool because it is not just a model paper. It is also an engineering paper.

It says: here is how we train this thing, here is how we make it faster, here is how we handle the weird timing problem in speech, and here is how we make the system learn more directly from the data.

Thank you for reading.

The next article will be Part 20, which means I will finally hit my goal for the year, unless I get distracted by another emotionally expensive machine learning side quest involving GANs, spectrograms, music generation, or something equally shiny.