Photo by Glenn Carstens-Peters on Unsplash

Am I The Only One Who Likes DLSS 5?

Nvidia has, out of nowhere, announced DLSS 5. DLSS originally stood for Deep Learning Super Sampling. That is, it would increase the resolution of the game using deep learning (AKA AI). But now it does so much more. Recently they announced that it would do frame generation meaning it would insert fake frames to increase the frame rate.

Now Nvidia is taking the next step and using their deep learning tech to alter the appearance of the frames too, that is what DLSS 5 is.

[embed]

People appear to be up in arms about this and I’m not exactly sure why.

I think part of it is that anything new is automatically bad. Like when a company rolls out a new logo everyone always complains about it. But wait a few months or years and people are like, “This is fine.”

Same thing here. People have played the games in the demo so they know what things are supposed to look like, then when Nvidia changes it they’re like, “This isn’t what it’s supposed to look like.”

In fact, DLSS technically already did something similar. Recently Nvidia introduced DLSS 4.5 at CES. And it did appear to make things pop more. Like in this LTT video around 4:47.

[embed]

It looks really good. It seems to me that with DLSS 5 Nvidia wants to make it official. And I think it has the potential to make games look way better.

Now, of course, there are problems that I foresee with the tech. Whenever you try to apply an AI filter, which is what this is, you have to worry about persistence. This means you want characters to look consistent between scenes, angles, and lighting. So far AI has not had a great job of doing that.

Although looking at the press release, it looks like Nvidia thought of this.

DLSS 5 takes a game’s color and motion vectors for each frame as input, and uses an AI model to infuse the scene with photoreal lighting and materials that are anchored to source 3D content and consistent from frame to frame. DLSS 5 runs in real time at up to 4K resolution for smooth, interactive gameplay.

Another potential problem is if the character models do not look great in the first place. This means the AI has to do more work to make the image look better and could lead to an uncanny feeling.

Some commenters were actually discussing this. In the trailer above it looks really good for photorealistic games like Resident Evil Requiem. But for less realistic games like Starfield it still looks good, but there’s only so much that AI can do. Or how much Nvidia is letting the AI do.

It appears as though Nvidia will allow developers to tweak how much DLSS 5 will affect their games so I wouldn’t worry too much about it.

Although the main thing people keep on worrying about is the artists’ intent. Would DLSS 5 negatively affect what the artists intended you to see? No it would not. Because what does the artist decide? They decide things like the architecture and lighting of the scene and what the characters look like.

But they can only go so far. Because video games have to render things 60 times a second. That means 60 times a second the GPU has to compute the colour of every single pixel. That means looking up the texture, calculating the lighting (for each light), factoring in reflections, and applying transparency. That’s a lot of work. So game engines are just a bunch of hacks to make it go faster.

Many of these hacks make things look worse, especially characters, resulting in the work falling short of the artists’ intent. In order to render characters properly you’d have to simulate the subsurface light scattering of the human skin. This requires a ton of math. Movies can do this because they can afford to wait hours for a single frame to render (and even then it doesn’t look 100% correct), games cannot.

That is why video game characters just look a bit more ‘plasticy’ than movies. Because they are real-time. Now characters have gotten better and better over time. Sometimes I wonder if you could render a movie in real time. Would anyone notice a difference?

But they’re still not perfectly realistic. Enter DLSS 5. DLSS 5 sidesteps this problem by simply using AI to improve the frames. This makes photorealistic games look a lot better and will actually help the game capture the artists’ intent. Especially if an artist goes in and tweaks the settings.

The only problem is stylized games. Those are games with cartoony characters or cell shading. How will DLSS 5 affect these games? Unfortunately I suspect the AI filter will need to be disabled for these games as training a model for each stylized game would take a lot of time. But the majority of games are not stylized, they are photorealistic, and DLSS will work perfectly for those.

Finally I should note that this is only generation 1 of the AI filter technology. Not even generation 1, this is an early preview of it. And it already looks amazing. It will only get better from here.

Remember generation 1 of ray tracing? It made things shinier and cuts your frame rate in half. We’ve come a long way since then.

I think it’s just all the AI hate now. It’s cool to hate on AI. And, to be fair, a lot of the products using AI are terrible. DLSS was not that.

DLSS was always a way of squeezing out more performance without brute force rendering. I was skeptical of it early on, especially when they announced frame insertion. Because we saw that on TVs and know how that went. But surprisingly it turned out fine.

This is because what Nvidia is doing is not just some dumb filter. It has motion vector information from the game and knows where objects should be between frames. And now it looks like Nvidia is applying this same technology to improve visual fidelity in a big way.