How to use custom features in ESPnet for ASR

ESPnet is a great toolkit to build machine learning pipelines for speech related task such as Automatic Speech Recognition (ASR) or Text-To-Speech (TTS). The problem is that for a non-fluent Bash speaker (like myself), it can be tricky to tweak its general behavior… Like what if you want to drop the traditional Fbanks/MFCC features and use your own ?

I will explain in this article what files to modify/replace in the ESPnet pipeline to use you custom features. A little disclaimer, I’m definitely not a pro and frankly not sure that my way is the perfect way to do this but hey, it ain’t stupid if it works right (or is it…?). So please, feel free to give me any feedback so we can all learn something !

So now let’s do this, shall we ?

Photo by Bill Jelen on Unsplash

Before touching any files, we have to understand the basic pipeline of ESPnet for ASR. I used the yesno recipe as a base for my explanation, as it’s a very simple one, but all the recipe are very similar so you should be okay (still some change might be necessary).

Pipeline overview

There are usually 6 to 7stages in the classic pipeline :

-1 — (Optional) Data Download 0 — Data Preparation 1 — Feature Generation 2 — Dictionary and Json Data Preparation 3 — (Optional) Language Model Preparation 4 — Model Training 5 — Decoding

Here, we need to focus on stages 0/1/2 as those three are the ones in charge of creating the features and all the important files we need. You will see later that we’ll only need to modify the Stage 1 to use our own features.

You can specify which stage you want to run in the run.sh by giving the arguments--stage x(stage to start from) and --stop-stage x (stage to stop at).

Stage 0 prepare the data related files in the data directory of the recipe. The directory structure is the following:

data
├── local
├── train_yesno
│   ├── wav.scp     # wav id to wav path
│   ├── text        # wav id to transcription
│   ├── utt2spk     # wav id to speaker id 
│   └── spk2utt     # speaker id to wavs id
└── test_yesno
    └── ...

Stage 1 generates the features using path informations from the data directory . It will generate the fbank and save them into the fbank directory (we do not care about the structure of this directory as we want to get rid of the fbanks), then it will dump the features into the dumpdirectory.

In the case of the yesno recipe, it will also create two directories from data/train_yesno : data/train_dev and data/train_nodev to create a train and a validation split.

The dump directory structure is as followed :

data
fbank
dump
├── train_nodev        # data used for training
│   ├── deltafalse
│   ├── feats.1.ark    # file that contains the feature
│   ├── feats.1.scp    # wav id to features location in .ark file
│   ├── feats.2.ark    
│   ├── feats.2.scp    
│   ├── feats.scp      # concatenation of all the feats.#.scp
│   ├── utt2num_frames # wav id to number of frames
│   └── filetype
├── train_dev          # data used for validation
│   └── ...
└── test_yesno         # data used for test
    └── ...

So for every set, we have the features stored in the feats.#.ark files. The number of .ark files to create to store our features is a parameter in run.sh (for yesno it’s 2). The feats.scp is a simple concatenation of our feats.#.scp files.

Stage 2 creates a vocabulary from the data/train_nodev/text file and transcribe all the informations of the dumped features into json files located at dump/$SET/delatafalse/data.json (here $SET is either train_nodev , train_dev or test_yesno). They do not contain the features for every wav but link to the feats.#.ark file containing them.

It’s those data.json files that will be used during all the other stages.

We will now pass to the heart of the subject : using our customs features.

Photo by SpaceX on Unsplash

Script and modifications

If you understood the pipeline, to use our custom features we need create our custom feats.#.ark/scp and feats.scp files in the dump directory, then generate the data.json files.

To do so, I created this simple python script :

[embed]

I took inspiration from this article on how to create .ark and .scp files.

We give 3 arguments to the script :

• nj the number of feats.#.ark/scp files we have to create
• wav the path to the wav.scp file where we are going to get our wav ids and wav file paths
• dir the path to the directory where we want to store our file

⚠️ WARNING ⚠️ You may have to change how you read the id and wav path from the wav.scp file in the line id2path = [x.split() for x in id2path] as the file structure may vary . Here it works fine for yesno but would not work for Librispeech, which would require id2path = [(x.split()[0], x.split()[-2]) for x in id2path].

Now that we have our python script ready, we have to incorporate it in the run.sh.

As I previously said, we are only going to modify the script in the stage 1. This is the original stage 1 script :

[embed]

The Feature extraction and the compute global CMVNparts are useless as they are related to the old features. But we still need to keep the make a dev set part and modif the dump features part.

This lead to the new stage 1 script :

[embed]

Fortunately, the make a dev set part still works fine even if you delete the old Feature extraction part. This will create the dump directory with everything needed to compute the data.json files in the next stage.

As all recipe differ slightly due to the original dataset being different in their structure, you will have to adapt how you modify the stage 1 to use the python script. I’ve done it without trouble for Librispeech so you should be okay 👌

With this you should be able to use your custom features !

Photo by NASA on Unsplash