Taking Flight: Mastering Skydio X2 Altitude Control with Deep Reinforcement Learning

Achieving stable flight with a quadcopter is no small feat. While traditional control systems can handle the physics, they often lack the adaptability required for complex, dynamic environments. The motivation behind this project was to move beyond rigid programming and leverage Deep Reinforcement Learning (DRL) to teach a drone how to fly itself. By applying Proximal Policy Optimization (PPO) to a simulated Skydio X2 drone, the goal was to automate attitude and altitude control, allowing the AI agent to independently learn the delicate balance of thrust and torque required for a stable hover.

Here is a breakdown of how the model was structured, how the neural networks were trained, and the results of the simulated flights.

The Drone Setup: Simulating the Skydio X2:

Before training an AI, it needs an environment. This project uses a custom MJCF model of the Skydio X2.

• The drone’s body is built with six degrees of freedom, meaning it can move and orient itself freely in any position or direction.
• The model relies on an accelerometer to measure linear acceleration and a gyroscope to measure angular velocity.
• Instead of complex quaternions, Euler angles (Roll, Pitch, Yaw) are utilized to make the orientations easier to read.
• The control input is handled by four rotors, where one control unit provides 1 N of upward thrust alongside the necessary torque to yaw around the z-axis.

Skydio X2 in MuJoCo

The Brains of the Operation: Actor-Critic Architecture

To teach the drone how to fly, the project utilizes an Actor-Critic policy update system.

• The observation space feeds the networks 12 variables, tracking position, velocity, and angles.
• The Actor Network takes these 12 inputs, passes them through two inner layers of 64 neurons each, and outputs a Gaussian probability distribution for the action space (thrust and torques).
• The Critic Network mirrors this input structure but outputs a single value representing the value function Vϕ​(st​).

Training relies heavily on the PPO-Clip algorithm, which calculates the loss and updates weights using backpropagation. This method is highly advantageous because it prevents the policy from updating too drastically over a single state-action pair. The objective function evaluated by the actor is calculated as follows.

where,

Depending on whether the advantage of the state-action pair is positive or negative, the optimizer limits the policy ratio update using either 1+ϵ or 1−ϵ.

A Two-Phased Training Approach

Training the agent to fly required a structured, two-phase curriculum.

• Phase 1 (Hovering): The environment was modified to spawn the quadcopter exactly at a 1-metre height. The agent focused entirely on learning how to maintain this altitude stably.
• Phase 2 (Takeoff): Once hovering was mastered, the spawn height was moved back down near the ground. The agent loaded the optimized weights from Phase 1 and was tasked with successfully taking off and reaching the 1-metre hover point.

Conclusion: Successes and the Drawbacks

By implementing the on-policy Proximal Policy Optimization algorithm, the project successfully created a DRL agent capable of applying the appropriate thrusts to stably take off and hover the Skydio X2 at a height of 1 metre.

However, working with PPO did reveal some inherent limitations. Over the course of the training, the policy progressively becomes less random. Because the update rule heavily encourages the agent to exploit the rewards it has already found, it loses the incentive to explore new actions. Ultimately, this dynamic can cause the policy to get trapped in local optima, preventing it from discovering potentially better flight strategies.