Dancing is just math, so can AI do it ?

How I trained a model to dance on music and make it dance on my songs. (and it worked)

Click to replay the gif

I’ve always be terrible at dancing

Although it does not seems that hard. I mean it’s just position in space in sync with music. That’s what made me realize:

“if dancing is just 3D coordinates and encoded audio chunks, well, it’s basically numbers predicting numbers”

And AI is really good at that.

So why not train a model that generates body positions from music?

First step: Let the model watch TikTok dance videos (poor him..)

No, I’m kidding, I’ve planned to train of Tiktok videos in the future but the data are quite messy for now, so we’re going to use AIST++ for this one: a well-known dataset of dance videos.

First step: get rid of the videos entirely. I used MediaPipe BlazePose to extract body poses (33 keypoints per frame at 25 fps): shoulders, hips, knees, wrists, etc. (see below)

Click to replay the gif

After that we don’t have videos anymore, just coordinates. One important thing: I normalize everything so the hip is always at the center and the torso height is always 1. That way the model doesn’t care if you’re tall or short, close to the camera or far: just pure movement.

And to give the model a better feel for how the body is moving (not just where), I added velocity (and later acceleration..) to each coordinate. So each frame goes from 3 numbers per landmark to 9.

• 3 for position
• 3 for velocity
• 3 for acceleration

That gives us a tensor of shape (T, 33, 9).

On the audio side, raw audio is 320,000 numbers (air pressure measured 32,000 times per second for 10 seconds).

Now, 320,000 numbers is way too much to feed a model (and useless), so every 1,280 samples (= 40ms = exactly one pose frame at 25fps) I take a frequency snapshot using a Fourier transform.

That gives me 25 snapshots per second (same as the coords!) I then compress the thousands of raw frequency bins into 80 mel bins, logarithmically spaced to match how ears actually work (more resolution in the low frequencies, less in the highs.)

Then instead of keeping thousands of raw frequency bins, I compress them into 80 buckets (bins) spaced the way human ears actually work more detail in the low frequencies, That’s the “80 bins” part.

Let’s geek out about the architecture for a bit

So the whole thing is basically a translation model (Transformer) where the two “languages” happen to be audio and motion.

The audio gets linearly projected into a 256-dimensional space, layer-normalized, and given positional encodings, that becomes the encoder memory.

The pose sequence goes through the exact same treatment and becomes the decoder target. Then it’s a six-layer Transformer decoder, pre-norm style (more stable to train than post-norm).

For the architecture geeks:

Audio (B, T, 82)  →  Linear + LayerNorm  →  + positional encoding  →  (B, T, 256)
Pose  (B, T, 297) →  Linear + LayerNorm  →  + positional encoding  →  (B, T, 256)

TransformerDecoder × 6 layers
  (masked self-attention on poses, cross-attention on audio)
        ↓
  Linear head  →  predicted poses (B, T, 297)

Each decoder layer does three things: masked self-attention on the pose sequence (frame i and frame i-1 are suppose to guess i+1 ), cross-attention on audio and finally, a feed-forward block.

The model is always learning “given the real previous pose, the current pose and the current audio, predict the next pose ”.

Ok so we got the model, we got the dataset, now let’s train.

Attempt one: the swaying skeleton

Life Reminder: the training never works the 1st time. 🥲

BUT NOT FOR ME ! 😎

Honestly I was pretty happy with the first results after 120 epochs. My skeleton was moving. Just it was SO SLOW.

But look at those badass moves after only 120 epochs:

Click to replay the gif

Ok, It’s not amazing like that, seems that our skeleton is tripping on something what interests me is that the movement seems consistent, the val loss is converging for now, and sure, some legs are still ‘flying’, but for a beginning it’s pretty good. (from an expert)

The ‘Rho la la, of course!’ Phase

You see, a dancer never (or rarely) dances at the same speed and acceleration all the time, and that’s what makes the dance exciting. And my training was not taking that (enough) into account.

So I modified a tiny bit the training goal:

W_TEMP_VEL   = 1.0   # was 0.1 — now a real training signal
W_TEMP_ACCEL = 0.5   # penalizes robotic, drifty movement
W_BEAT       = 0.2   # body energy should peak when the music does

So now the model has to guess the right position, and the right speed and the right acceleration at the right time.

Yeah, that’s seems a lot but this is necessary and don’t worry, if an LLM can generates the right sentence with the right tone and without anything scandalous, it’s because it was trained under a LOT of constraints (will make an article about that folks).

Anyway, so let’s restart from 120 epochs with the additionnal goals.

My god, it got the moves

Click to replay the gif

I mean, this is better than I can do.

Now that the skeleton can dances, let’s make it dance on my song M-A, (available on all platforms lol: Spotify, Deezer and Apple Music.)

Well. It’s not great. I mean it’s not as great as before. Why ? Because I only trained on similar music and poses, and the model might no be generalist enough to make it dance on all types of music.

Training info

The model has been trained for 175 epochs with a G4 on google colab: yeah it was long but I’m trying to not use A100 all the time for personal projects to not destroy the planet with my stupid projects.

IMPORTANT: I will train on more epochs in the meantime as it is not enough so make sure to retry the model later if you’re not satisfied.

All the code, model are available on my **HF 🤗 and [Git](https://github.com/Emma5099) 🐈‍⬛!**

Scheduled sampling. Early training is 100% teacher forcing: the model always sees the real ground-truth previous pose as input. But at inference time it has to use its own predictions, and errors compound fast. So over training, I gradually ramp down the ratio of real-to-predicted input: roughly 100% teacher forcing in epochs 1–15, down to around 50% by epoch 100. The model learns to handle its own imperfect outputs, not just pristine data.

Windowing. Most of training used full 250-frame sequences, but about a quarter of every batch used a window of 60-frame crop instead. Why? Because if you only train on long sequences, the model silently learns to lean on that long-range history, then completely falls apart at inference when that history isn’t there.

So, does it dance better than me?

Setting the bar low here since I mentioned I’m terrible, but yes, it does.

So could it replace dancers ?

I’ll keep saying it: training on data is just generalising data.

My goal is never to see if it AI can replace dancers (or artists in general) because that’s not even a question here, but my goal is to show what generalisation can do: by training on all those videos of dancers, we’ve been mixing all the dancers together as one and only person.

And I think that’s the pretty beautiful. All at once. On my song that I produce, created, wrote and recorded from scratch. We can almost see it in a way of creating a choregraphy for my own song.

So can it dances ?

Yes, the skeleton actually grooves. It catches beats, weight-shifts before moves, gets more energetic when the music picks up. But why ? Because we just generalise all the amazing dancers of the video at once and made them dance as one skeleton.

IMPORTANT: I will train on more epochs in the meantime as it is not enough so make sure to retry the model later if you’re not satisfied.

So Emma, what is the next step ?

• Next step could be to stop making dance a poor skeleton but instead a 3D Model ? (but I kind of like the skeleton, and I think we might dig deep in the uncanny valley by making a 3D model…) and actually the point is not to replace dancers so what would be the purpose behind that ?
• Next step could be to generate of a large set of videos with different musics, dancing moves etc… but actually the point is not to replace dancers so what would be the purpose behind that ?
• Next step could be to train on TikTok videos to see a similar pattern behind that. That could be interesting to understand in terms, again, of generalization.
• Next step could be to reverse the model: instead of a Music2Dance generator, why not a Dance2Music and ask my dancing friend to generate some music based on her dancing ?

Go check that **portfolio **to not miss any of my other projects/articles/music/paintings/code/etc….

Beautifully written, amazingly coded, and brilliantly imagined by Emma Genthon (or M-A, my artist name). For the story: I’m an (french) AI Engineer, I have worked in ethics and compression among LLMs and now work for clients using and implementing AI the best way. On the side, I’m an artist and I like to use AI in a smart way with art. I insist on smart. I do not generate brain rot.

Portfolio ○ Linkedin ○ Github ○ HuggingFace