An AI Anaesthesiologist: Using PPO for Safe, Personalised Drug Dosing

Source: Generated with Nano Banana Pro

Reinforcement learning (RL) is well suited for problems requiring sequential decision making, delayed feedback and continous control. While RL has well known success within robotics and gaming, recently its application in **healthcare** has grown.

In this tutorial, we will explore one such application: personalised durg delivery for general anaesthesia. Specifically, I will demonstrate how Proximal Policy Optimisation (PPO), can be applied to control propofol infusion for a simulated patient. Using the Schnider pharmacokinetic-pharmacodynamic (PK/PD) model, an agent will be trained to regulate a patient’s depth of anaesthesia in real time, adapting its dosing strategy based on demographic information such as age, sex, height and weight.

Note: This work is a proof-of-concept conducted entirely in simulation. The PK/PD parameters and infusion ranges are simplified and are not intended for direct clinical use. The goal is to demonstrate the behaviour and interpretability of PPO in a safety-critical control setting.

1. What is Proximal Policy Optimisation (PPO)?

PPO is an on-policy, policy gradient algorithm introduced by OpenAI in 2017. It is widely used in continuous control problems due to its balance between performance, stability and ease of implementation.

PPO builds on mathematical ideas from TRPO (Trust Region Policy Optimisation) but removes its computational complexity by introducing a clipped surrogate objective. This allows policies to improve while preventing overly aggressive updates, making PPO particularly suitable for safety-critical domains such as robotics and healthcare.

To understand how PPO achieves this, we need to look at its three core components: the actor-critic architecture, the advantage function, and its signature clipped surrogate objective.

Actor-Critic Architecture

PPO belongs to the class of actor-critic algorithms, which combine policy-based and value-based learning.

• Actor (Policy): Represents the policy π(a∣s), which maps the probability of taking an action a in state s. The objective is to maximise the expected cumulative reward.
• Critic (Value Function): Estimates how good a state is by learning a value function V(s) and provides feedback on the actor’s decisions.

The Advantage Function

How does the Actor know if its action was actually good? It relies on the Advantage Function (Aₜ). This measures the ‘quality’ of the action compared to the Critic’s expectation:

• Positive Advantage: The action yielded a better result than the Critic expected. PPO increases the probability of doing this again.
• Negative Advantage: The result was worse than expected. PPO decreases the probability.

Note: In practice, we use Generalised Advantage Estimation (GAE). This blends multi-step errors to balance bias and variance, smoothing out the learning signal to prevent abrupt updates.

Clipped Surrogate Objective

In standard policy gradients, if an action leads to a huge advantage, the model might drastically shift its weights to exploit it. In complex control tasks, this overconfidence can cause the policy to collapse into failure — a disaster in clinical settings.

PPO solves this by introducing a Safety Constraint (Clipping). It looks at the Probability Ratio between the new policy and the old policy:

Then it optimises the following Clipped Objective Function:

Here is how this equation works. A hyperparameter ϵ defines the allowable update range:

• If the ratio remains within [1−ϵ,1+ϵ]: the update proceeds normally
• If it exceeds this range: the update is clipped, preventing overly large policy changes

The graph below visualises this mechanism:

Figure from the original PPO Paper (Schulman et al.)

Putting It Together: The Total Objective

The final loss function PPO optimises is a sum of these parts:

It maximises the clipped reward, while simultaneously minimising the Critic’s prediction error and encouraging exploration via an entropy bonus.

Pseudocode

Here is the pseudocode for PPO:

By combining advantage-based learning with conservative policy updates, PPO enables agents to learn complex behaviours reliably without destabilising their own learning process.

2. The Problem: Why Automate Anaesthesia?

General anaesthesia is a delicate balancing act. Drugs like Propofol are administered to ensure a patient remains unconscious and pain-free. However, the margin for error is slim — too little drug risks patient awareness, while too much increases the risk of prolonged unconsciousness or neurological harm.

Currently, anaesthesiologists manually adjust infusion rates based on vital signs. Automating this process is challenging due to inter-patient variability: a dose suitable for a young, healthy adult may be unsafe for an elderly patient. Traditional control systems, such as PID systems, struggle to adapt to these non-linear biological differences.

There is where PPO is well suited to this problem:

1. Continuous Control: Unlike simple games with “Left/Right” buttons, drug delivery requires precise, continuous values, which PPO naturally handles.
2. Safety Stability: In surgery, abrupt changes in dosage are dangerous. PPO’s clipping mechanism prevents the agent from making drastic, unsafe updates to its policy, ensuring smooth drug delivery.

3. Coding Time with Stable Baselines 3

To train our agent, real patients are not used but instead simulated using the **Schnider model. This model describes how propofol distributes across physiological compartments and how drug concentration affects depth of anaesthesia, measured using the Bispectral Index (BIS). We will focus on the induction phase (first ~5 mins), which is the crucial, high-risk transition **where the patient drifts into unconsciousness.

Note: this set-up is inspired by Schamberg et al.’s work, who applied actor-critic methods to a similar simulator. I built upon their approach by using PPO as the clipped objective provided more stable and reliable updates.

This diagram illustrates the RL feedback loop designed for the AI Anaesthesiologist:

System architecture: The PPO Agent (Left) operates in a closed loop with the Virtual Patient (Right). The agent observes the patient’s state, including real-time vitals and static demographics, and outputs a continuous propofol infusion rate. The environment simulates the drug’s effect using the Schnider Pharmacokinetic (PK) and Hill Pharmacodynamic (PD) equations, returning a reward signal based on patient safety.

Step 1. The Environment Setup

First we define our cutom Gym environment. This sets up the:

• Action Space: What the agent can do
• Observation Space: Information the agent is provided

import gymnasium as gym
import numpy as np
from gymnasium import spaces

class AnaesthesiaEnv(gym.Env):
    def __init__(self):
        super(AnaesthesiaEnv, self).__init__()

        # Action: Propofol infusion rate (Normalised for PPO: -1 to 1)
        self.action_space = spaces.Box(low=-1.0, high=1.0, shape=(1,), dtype=np.float32)

        # Observation: [Current BIS, Target BIS, Effect Concentration, Age, Weight, Height, Gender]
        self.observation_space = spaces.Box(low=0, high=200, shape=(7,), dtype=np.float32)

        self.target_bis = 50.0
        self.dt = 1.0  # Time step: 1 second

        # PD parameters (Hill Equation - Sigmoid Curve)
        self.E0 = 100.0
        self.Emax = 100.0
        self.EC50 = 4.0
        self.gamma = 2.0

As shown in the code, the RL environment provides the agent with:

1. Current BIS: Depth of anaesthesia (0 = Coma, 100 = Awake).
2. Target BIS: Set to 50 (the ideal surgical state)
3. Effect Concentration (Ce): The estimated amount of drug currently in the brain
4. Demographics: Age, Weight, Height, and Gender.

The agent outputs a continuous infusion rate, updated once per second.

Step 2. The Schnider Model

This helper function will calculate the unique pharmacokinetic (PK) constants for every patient based on their age, gender, height and weight. This is the code that ensures that the agent is trained on a diverse population, not just a single average patient.

    def _get_schnider_params(self, age, weight, height, gender):
        """
        Calculates PK constants based on the Schnider Model.
        This enables the agent to treat random patients (Domain Randomisation).
        Values taken from: Schnider, T. W., et al. (1998). "The influence of age on propofol pharmacodynamics."
        """
        # Lean Body Mass (lbm) calculated from James Equation
        if gender == 0: # Male
            lbm = 1.1 * weight - 128 * (weight / height) ** 2
        else: # Female
            lbm = 1.07 * weight - 148 * (weight / height) ** 2

        # Schnider Model Equations for Compartment Volumes (V) and Transfer Rates (k)
        v1 = 4.27
        v2 = 18.9 - 0.391 * (age - 53)
        v3 = 238

        k10 = 0.443 + 0.0107 * (weight - 77) - 0.0159 * (lbm - 59) + 0.0062 * (height - 177)
        k12 = 0.302 - 0.0056 * (age - 53)
        k13 = 0.196
        k21 = (1.29 - 0.024 * (age - 53)) / v2
        k31 = 0.0035
        ke0 = 0.456

        return k10, k12, k13, k21, k31, ke0

Step 3. The Step Function

This is the most important part of the simulation. Every second, the step function will take the agent’s action, value of the infusion rate, and updates the differential equations. This calculates how the drug flows from the blood to the brain and updates the BIS score.

def step(self, action):
    # 1. Convert Action: PPO outputs [-1, 1], we map to [0, 10] mg/sec
    infusion_rate = float(np.clip((action[0] + 1) * 5, 0, 10))

    # 2. Physics Step (Pharmacokinetics - Euler Integration) 
    x1, x2, x3, xe = self.state
    dx1 = infusion_rate + self.k21 * x2 + self.k31 * x3 - (self.k10 + self.k12 + self.k13) * x1
    dx2 = self.k12 * x1 - self.k21 * x2
    dx3 = self.k13 * x1 - self.k31 * x3
    dxe = self.ke0 * (x1 - xe)
    self.state += np.array([dx1, dx2, dx3, dxe]) * self.dt

    # 3. Calculate BIS (Pharmacodynamics - Hill Equation)
    Ce = self.state[3]
    effect = (self.Emax * (Ce**self.gamma)) / (Ce**self.gamma + self.EC50**self.gamma)
    self.current_bis = self.E0 - effect

Step 4. The Reward Function

The reward function, adapted from Schamberg et al.’s work, priorities safety and accuracy:

• Survival Bonus (+0.5): A small constant reward encourages episode continuation
• Accuracy penalty (-|BIS(current) — BIS(target)|): A penalty based on how far the patient’s BIS score is from the target. The closer to 50, the higher the reward.
• Safety Violation (-5.0): A large penalty if the BIS drops below 40 (Risk of overdose)

This structure forces the agent to avoid overdosing before optimising performance.

# 4. REWARD FUNCTION (Implementation of Schamberg et al., Eq. 4)
        # r = 0.5 - |error| - (rho1 * dose) - (rho2 * safety_violation)

        # Calculate normalized error (Target - Current)
        error = (self.target_bis - self.current_bis) / 50.0 

        # Base Reward (Reward for keeping patient alive)
        reward = 0.5

        # A. Performance Penalty (minimise error)
        reward -= abs(error)

        # B. Sparsity Penalty (Rho 1) - "Use less drug"
        reward -= 0.1 * infusion_rate 

        # C. Safety Penalty (Rho 2) - "Avoid over-sedation"
        # If BIS drops below 40, harsh penalty
        if self.current_bis < 40:
            reward -= 5.0 

        self.steps += 1
        truncated = self.steps >= 300 # Episode ends after 5 minutes (300 secs)

        return self._get_obs(), reward, False, truncated, {"bis": self.current_bis, "infusion": infusion_rate}

Step 5. Training our agent

Training is performed using Stable Baselines 3, allowing the focus to remain on designing the environment and reward system while abstracting away the complex implementation details of PPO. This can be done easily with a few lines of code!

import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3 import PPO
from patient_env import AnaesthesiaEnv

env = AnaesthesiaEnv()
print("Starting PPO Training... ")

model = PPO("MlpPolicy", env, verbose=1, learning_rate=0.0003)
model.learn(total_timesteps=150000) 

print("Training Complete!")
model.save("ppo_anaesthesia_agent")

4. Results and Visualisation

To evaluate the performance of PPO in this application, we analysed its behaviour at both the individual-patient level and across a diverse simulated population.

Individual Patient Behaviour

Figure 1 compares PPO performance for a young (25-year-old) and elderly (80-year-old) patient with identical height and weight. In both cases, the agent successfully drives BIS score toward the clinical target of 50 and maintains it within a narrow range.

Notably, the elderly patient receives a lower steady-state infusion rate. This behaviour is not hard-coded but it emerges from training and reflects the agent’s ability to recognise age-dependent pharmacokinetic differences.

Figure 1 — Top: BIS score trajectories for a young (25 y/o, blue) and elderly (80 y/o, orange) patient over a 300s simulation, relative to the target BIS of 50 (green dotted line). Bottom: The corresponding propofol infusion rates (mg/sec) administered by the agent over the same time period.

Demographic Sensitivity

Figure 2 compares male and female patients with identical age, height, and weight. While BIS trajectories remain similar, the learned dosing profiles differ slightly. In this simulated setting, the policy administers a lower infusion rate for male patients, consistent with sex-dependent differences in lean body mass within the Schnider model.

These results suggest that the policy has internalised structured physiological relationships rather than memorising a single dosing strategy.

Figure 2 — Top: BIS score trajectories for male (blue) and female (pink) patients with identical weight and height over a 300s simulation. Bottom: The corresponding propofol infusion rates (mg/sec), showing the Female patient receiving a higher continuous dose than the Male patient to maintain the same anaesthetic depth.

Population level robustness

To rigorously assess robustness, the trained agent was evaluated on an initial cohort of 100 patients, followed by a larger validation set of 500 randomly generated patients (See Figure 3).

Performance was quantified using Time-in-Target-Range (TTR), the percentage of time BIS remains within the clinically acceptable interval of 40–60. The agent demonstrated remarkable consistency, achieving a mean accuracy of 98.3% (N=100) and maintaining 97.9% (N=500) across the expanded population.

These consistently high TTR values confirm that the agent successfully maintains stability, avoiding significant deviations (such as dangerous overdoses) for the vast majority of the population.

Figure 3: Distribution of Time-in-Target-Range (TTR) scores for 100 (left) and 500 (right) randomly generated patients. The x-axis represents the percentage of the simulation time the patient’s BIS score remained within the target interval (40–60). The vertical dashed red line indicates the population mean accuracy.

Policy Structure and Stability

A policy heatmap was generated to visualise the agent’s action as a function of BIS and effect-site concentration (Figure 4).

As expected, higher infusion rates (yellow/green) are selected when BIS is high and effect concentration is low. Dosing decreases smoothly as the patient approaches the target depth of anaesthesia.

This smoothness is a direct result of PPO’s clipped objective function, which discourages abrupt policy changes. In continuous medical control tasks, this stability is essential, as aggressive or discontinuous actions could lead to unsafe physiological responses.

Figure 4: Heatmap visualisation of the agent’s learned policy surface. The x-axis represents the patient’s consciousness level (Current BIS), and the y-axis represents the estimated drug concentration in the brain (Effect Site Concentration). The color scale indicates the agent’s output action (Propofol Infusion Rate), transitioning smoothly from minimum dosage (dark purple) to maximum dosage (yellow).

Learning Dynamics

The learning curve (Figure 5) illustrates a steady improvement in average reward without large oscillations or performance collapse. While PPO is generally less sample-efficient than off-policy methods (like SAC or TD3), its stable convergence makes it attractive for safety-critical applications where reliability is critical.

Figure 5: Learning curve showing the agent’s training progress over 250,000 steps. The x-axis represents the number of training steps, and the y-axis shows the smoothed average reward per episode. The curve demonstrates a rapid initial increase in performance followed by a stable plateau as the agent converges on an optimal policy.

Conclusion

This project demonstrates that PPO can successfully simulate general anaesthesia delivery. By combining PPO with a physiologically informed PK/PD model, the agent learns smooth, personalised dosing strategies that adapts to patient demographics.

The clipped objective in PPO promotes conservative policy updates, which is essential in medical control tasks. While this work is limited to simulation, it highlights the potential of reinforcement learning as a tool for personalised, safety-aware decision support in bioengineering applications.

This GitHub repository contains all the code used for this tutorial.

I acknowledge the use of ChatGPT-5 and Gemini for assistance with conceptual clarification and code understanding. I confirm that no content generated by AI has been presented as my own work.