Build a GAN (part 3): generate music with WaveGlow model

Is it possible to modify Glow (see previous article) to generate music instead of images? Well, if we keep the model as it is, it won’t. Unless we modify the architecture, that’s what the WaveGlow model does and we will explain it. WaveGlow is designed to generate audio from a spectrogram (of someone reading a text), so we will modify it a bit to see if we can generate music.”

Image vs audio

First of all, you know that images are represented using a 3D matrix (H, W, C). But how can 3 minutes of music be represented on a computer?

It is a 1D array that contains a LOT of numbers. For example, two minutes of music can contain something like millions of numbers. These values are between -1 and 1 (in general) and represent the amplitude.

Quick note on log likelihood:

We have some data, and the goal is to find its probability density function p*. Remember: the pdf shows how dense a value is. It is not probabilities!

to find p*, we will approximate it with p_θ , for that we need to maximze this :

where xi is a sample from p, and using the properties of log. After maximizing this, we will have p_θ ~ p.

it is equivalent of maximizing:

where p_θ is the probability density function if x is continuous, and just the probability if x is discrete.

Mels vs Hertz:

Both mels and hertz are used to measure frequency :

Hertz measures the physical reality of sound, while mels measure the human perception of sound.

So when we say a mel-spectrogram, it means a spectrogram where the frequency axis is transformed to the Mel scale, which compresses higher frequencies and gives more resolution to lower frequencies to better match human hearing

0 db means the loudest sound

A spectrogram provides the frequencies and amplitudes at a given time, but it is not sufficient because it does not indicate the phase of each frequency. As a result, it is impossible to reconstruct the original sound wave using only the spectrogram. This is where WaveGlow comes in: its goal is to generate a sound waveform given a spectrogram and a text as input. So first we have a text, then we use another model to generate a spectrogram, then we use WaveGlow to generate the audio.

if zi follows a standard normal distribution, then the log density of zi is :

WaveGlow

waveglow architecture

To construct WaveGlow, starting from Glow, they first removed the ActNorm layer.

A batch of B audio samples will initially have a shape of (B, T). Then, instead of processing the audio as (1, T), they separate it into 8 chunks and stack them on top of each other, so the audio dimension becomes (8, T/8). This is the squeeze operation in WaveGlow. A batch of audio now has shape (B, 8, T/8). This makes training faster and makes it easier to learn patterns between neighboring audio samples.

Also instead of using L iteration and in each iteration do K step_flows and split , they define a total number of step_flows at the begining and every n step they split. IT is equivalent. So in Waveglow, every 4 flows we split some channels to generate the latent.

And we don’t split the tensor in half along the channel dimension as in GLOW. Instead, we define each zi to have a channel dimension of 2. So when splitting, we only take the first 2 channels of x, and thats our zi. We also assume zi follows a standard normal dsitribution, so there is no need to standardize it or estimate its statistics, unlike in GLOW

The model forward pass (audio => latent) should look like this: :

#initialization of step_of_flows with in_channels: 
flows= [flow(8),flow(8),flow(8),flow(8), flow(6),flow(6),flow(6),        
        flow(6),flow(4),flow(4),flow(4),flow(4)]

def forward(x):
    x = unfold(x) #so x (B,1,T) => (B,8,T//8)
    objective = [0,0,0,0....0,0] #size : B
    latents = []
    logdet = [0,0,0,0....0,0] #size : B

    for i, flow in enumerate(flows):
        x, logdet = flow(x, logdet=logdet, reverse=False) #cumulative logdet
        # split condition
        if (i + 1) % 4 == 0 and i not last index:
            x, zi= split(x)
            logp_zi = log_density_of(zi)  
            objective += logp_zi           
            latents.append(zi)

    objective = objective + logdet
    # calculate p(final latent) for the loss
    logp = -(log(2π) + x²)/2
    logp = sum(logp, dim=(1, 2))
    objective = objective + logp

    latents.append(x)

    nll = -objective.mean()
    nll_per_dim = nll / num_dims
    return nll_per_dim, latents

each flow() is : Affine coupling layer -> 1x convolution layer

quick note on Affine coupling layer:

what brilliant about this layer, is that it is invertible even though the NN operation is not. Because when inverting the network, we can compute s and t from the output x2 by simply recomputing NN(x2).

We can put whatever we want in NN. In WaveGlow they put a neural network heavily inspired by WaveNet, they call it WN (explained below).

The original WaveGlow was designed to generate audio from text. In the official implementation, the network also takes as input a spectrogram representing how the text should sound. In our case, we only want to generate music audio from latent variables, so we remove this conditioning

Here is my implementation of that modified waveglow for music generation, i used fma_small dataset and only wanted to generate 1 second of music to begin with. and like in the paper: a batch size of 24, 12 step of flows, Adam optimizer, a learning rate between 10^-4 and 10^-5…

WaveGlow is only made of convolution layers, and convolutions only care about number of channels, so actually we can train the model on short audio samples (e.g., 1 second) and still generate longer audio sequences. To generate a 10 sec audio, one option is to generate 10 random latents, which will produce separate 1-second audio segments that are not temporally coherent. Alternatively, we can sample a single larger latent corresponding to 10 sec. Because the latents produced by waveglow and glow, after concatenation, have the exact same shape as the inital x. So in our case, instead of generating a latent of shape (1,8,2048) (1 sec), we will generate a latent of shape (1,8,20480) that we will separate in three: (1,2,20480) , (1,2,20480), (1,4,20480) (since the model produces 3 latents).

The audio generated by our custom WaveGlow model does resemble white noise, but this is not a concern, just as with image generation, we will explore more complex models in the next articles in order to reach our goal.