Autonomous Driver a.k.a. Toddler Behind the Wheel

Hey there again! This time we will dive even deeper to our developments in AI racing. Just to notify, we are Czech-Slovak AI racing team and we explore possibilities of implementing advanced AI/ML frameworks to make computer control racing car in open-source racing game TORCS. To catch up, see our other blogs!

Since the last time we presented our ideas, we proceeded even deeper into race engineering and AI. The main topic we have research deeply is so called Reinforcement Learning. The foundations of this method builds up on very simple idea introduced even last time.

Core is to define so-called reward function which corresponds with the concept — higher values means better performance of the car on the track. What is better performance? Longer distance, longer distance while being fast, you name it…

In deep reinforcement learning, a common paradox emerges when training an agent for autonomous driving: if you only reward the agent for its forward velocity, it will happily floor the accelerator and crash straight into the first wall at 200 MPH. Conversely, if you penalize it too heavily for going off-track, it will discover that the safest strategy is to sit parked at the starting line forever.

This is where reward engineering comes into the game. The most straightforward first-time strategy is to simultaneously penalize slow driving and going-off the track. Sounds good!

If you run such a framework, the most precise description could be toddler behind the wheel. Car tends to go further but it does so with terrifying instability.

While a basic baseline reward gets the vehicle moving down the track, the agent quickly discovers a bizarre loophole: it can maximize its survival rewards by rapidly “sawing” the steering wheel back and forth down straightaways, sliding wildly into corners, and violently hugging the track boundaries. The car survives, but it drives like a manic arcade gamer rather than a professional race car driver.

Straightforwardly, we designed element of the reward function to make the car to drive in the straight line as long as possible. This way, car will spent the time in high speeds as long as possible and thus make the track faster!

Even though, AI driver could not make it past the chicane on the track for the time being, training is still under way! Check out!

[embed]