Reinforcement Learning: Teaching an Agent to Operate a Power Plant

Imagine a toddler learning to walk. They take a step and might stumble, but with each attempt, they adjust their movements based on the success or failure of the previous step. Over time, they learn to walk steadily by continuously adapting their actions to achieve the goal of moving forward without falling. This trial-and-error learning process is the essence of reinforcement learning (RL). All animals learn from their experiences of interacting with the environment, adjusting their behaviour to achieve desired outcomes.

Inspired by this concept, RL in artificial intelligence enables machines to interact with their environment and learn optimal behaviour from experience. It’s one of the most exciting branches of machine learning (also one of the most challenging). In 2016, Google’s DeepMind developed AlphaGo, an AI agent that won against the world champion in the game of Go, an abstract strategy board game known for its complexity. AlphaGo learned to play at a superhuman level not by mimicking expert moves but by playing millions of games against itself. Many consider this a landmark achievement that showcased the power of RL in solving problems once thought to be uniquely human.

Despite the excitement surrounding RL, resources on the topic can be limited and very difficult to follow, making it inaccessible to many. In this article, I aim to provide a humble introduction to RL, covering the basics of its mathematical foundations and illustrating a set up through an example; teaching an agent to optimally operate a simplified power plant. I am currently learning and working with RL so a lot of the work here will be my condensed notes.

Introduction to Reinforcement Learning

The core of reinforcement learning is about training an agent to make a decision or a sequence of decisions within an environment to maximise its cumulative reward. Unlike supervised learning, where the correct labelled output is provided for each input, RL relies on the agent learning from the consequences of its actions i.e. essentially learning from experience. Before diving into the mathematics, let’s outline the key components/concepts within a RL framework:

• Agent: The decision-maker.
• Environment: The system with which the agent interacts to determine the consequences of its actions
• State (s): A representation of a situation within the environment.
• Action (a): A set of all possible actions the agent can perform
• Reward (r): Immediate return received after transitioning from one state; i.e. the return from taking an action
• Policy (π): The strategy that the agent will use to determine the next action based on the current state.

This framework is also schematically demonstrated in this image:

Image obtained from TechVidvan

The Markov Decision Process (MDP)

To formulate reinforcement learning problems, Markov Decision Process (MDP) is used. An MDP provides a mathematical model for decision-making, where outcomes are partly random and partly under the control of the decision-maker (agent).

An Markov Decision Processes is defined by (S,A,P,R,γ), where:

• S: A set of states.
• A: A set of actions.
• P(s′∣s,a): The probability of transitioning from state s to state s′ after action a.
• R(s,a): The expected reward received after taking action a in state s.
• γ ∈[0,1] : The discount factor; balances immediate and future rewards. A value close to 0 makes the agent short-sighted (prioritising immediate rewards), while a value close to 1 encourages the agent to consider long-term rewards.

The Markov Property is a key assumption in MDPs which states that the future state depends only on the current state and action taken, not on the history of states and actions that preceded it. This “memoryless” property simplifies the design of RL algorithms.

Objective: Maximising Cumulative Reward

The goal/purpose of the agent is to find an optimal policy π* that maximises the expected cumulative reward over time. The policy maps states to actions, π: S→A, determining the action the agent will perform in each given state.

The cumulative reward/return, is defined as the sum of discounted future rewards:

· Gt​: return at time step t.

· r_(t+k+1): Reward received at time step t+k+1.

· γ: Discount factor (0≤γ≤1).

Value Functions

To make decisions, the agent estimates the value of states and actions. Value functions quantify how good it is for the agent to be in a particular state and to perform a particular action in that given state.

State-Value Function:

The state-value function, V^π(s) is the expected return when starting from state s and following policy π for all subsequent states or mathematically written as:

This function quantifies the long-term value of being in current state s under the policy π.

Action-Value Function:

The action-value function Q^π(s, a) is the expected return after taking action a in state s and subsequently following policy π for all remaining states:

This function quantifies the value of performing action a in current state s under policy π.

The Bellman Equation

These value functions are recursive where the value of any state/action is essentially the reward for the immediate action followed by the value of the new state. These recursive relationships are known as the Bellman equations.

Bellman equation for state-value function (V^π (s)):

For each possible action, a the agent can take in state s, weighted by the probability π(a∣s) of taking that action under policy π:

· Immediate Reward: R(s,a) is the expected reward for taking action a in state s

· Future Value: γ∑P(s′∣s,a)V^π(s′) is the discounted value of the next state s′, weighted by the probability P(s′∣s,a) of transitioning to s′

In simple terms, this equation essentially says the value of state 𝑠 is the expected immediate reward plus the expected discounted value of the next state, considering all possible actions and transitions.

Optimal Policy and Value Functions

An optimal policy π* yields the highest value for all states:

Similarly, the optimal action-value function is:

The goal of most RL algorithms is to find Q(s,a) or V(s), which then define the optimal policy. For example a deterministic optimal policy can be, for any given state, the agent selects the action that has the maximum action-value function:

But clearly we can take two different approaches, optimise the state and action value functions or optimise the parametrised policy directly. We’ll brief discuss both.

Value-Function: Temporal Difference Learning

Like normal neural network training with backpropagation after each batch, RL needs to also update the value functions based on the consequences of the actions taken. Depending on the nature of the problem at hand, there are different approaches to this. A popular approach is Temporal Difference (TD) Learning, which can be used in both episodic tasks (with a clear start and terminal states) and continuing tasks (without a terminal state). TD updates the value estimates after each step. TD learning updates the value estimates based on the difference between predicted and actual rewards. This can be written as

· r_t+1​: Reward received after taking action A_t​ in state S_t.

· V(S_t+1): Estimated value of the next state.

· V(S_t): Current estimate of the value of state S_t

The TD error δt measures the difference between the expected value and the observed value of the current state. The update rule for the value function is:

where α is the learning rate. The value estimate V(S_t) is updated towards the observed reward plus the discounted value of the next state. This method allows the agent to learn value functions directly from raw experience without a model of the environment.

Policy Gradient Methods

Whilst value-based methods focus on estimating value functions, the aim of policy gradient methods is to optimise the parametrised policy directly. The policy is parameterised by θ, and the objective is to maximise the expected return:

The policy parameters are then updated using the standard gradient descent (here we are maximising so its actually gradient ascent!) method:

But calculating ∇_θ J(θ) is tricky. Since J(θ) involves an expectation over all possible action sequences, it’s not straightforward to compute this directly. This is where the Policy Gradient Theorem provides a practical way to compute this gradient:

The term ∇_θ​lnπ_θ​(a∣s) is the gradient of the log-probability of taking action a in state s under policy π_θ. This equation means the parameters of the policy are adjusted to increase the probability of actions that lead to higher rewards.

Proximal Policy Optimization (PPO)

In practice, methods like Proximal Policy Optimization (PPO) are used to stabilise training and improve performance. The PPO objective function is:

Where r_t (θ) is the probability ratio between the new and the old policies:

​Ā is the advantage estimate at time t, measuring how much better action a_t is compared to the average action in state s_t. ϵ is a hyperparameter that controls the clipping range. The clipping function ensures that the update doesn’t deviate too much from the previous policy, which helps maintain stable learning. This clipped surrogate objective essentially prevent large updates that could destabilise training.

The advantage function Ā_t represents how much better an action is compared to the average. It’s calculated as:

• If Ā_t>0, the action is better than expected, and the policy should increase its probability.
• If Ā_t <0, the action is worse than expected, and the policy should decrease its probability.

Applying RL to Power Plant Operations

With a simple foundational understanding of reinforcement learning and its mathematical underpinnings, let’s apply reinforcement learning to teach an agent to optimally operate a power plant. The operation of a power plant involves making decisions that balance multiple objectives, such as safety, meeting power demand, minimising operational costs, adhering to emissions regulations, and scheduling maintenance. I appreciate this is a massive oversimplification of operating a power plant but this is just intended as a simple example for demonstrating RL.

Defining the Environment

The states (s) is a combination of external factors and internal states. The state s_t ​includes:

• External Factors:
• Demand forecast (d_t ​)
• Fuel prices (p_coal,t ; p_gas,t​)
• Efficiency parameters (e_boiler,t e_turbine,t)
• Other external variables; a list is shown in the figure below.
• Internal States:
• Power output (P_t)
• Fuel consumption rate (F_t)
• Emissions levels (E_t)
• Maintenance status (M_t​)
• Energy storage levels (S_t​)

The external variables are synthetic data, a few examples of these variables are shown in the image below. The code for the synthetic data generation was generated to encapsulate the seasonal behaviour of these variables in the real-world by time of day and time of year. I won’t include the code here but it is fairly straightforward to go through and accessible on github.

The actions (a) are the control variables such as turbine outputs, fuel mixture ratios, and emissions control settings. The action list at time t includes:

• Main turbine output (a1,t)
• Secondary turbine output (a2,t)
• Fuel mixture ratio (a3,t)
• Generator excitation level (a4,t​)
• Emissions control intensity (a5,t)

These actions are continuous variables bounded within operational limits.

Let’s begin writing the environment code. we’ll use gymnasium library to create our environment ‘gym’. This part of the code just contains the initialisation of all the parameters of the class which is also the parameters used at the beginning of the run.

import gymnasium as gym
from gymnasium import spaces
import numpy as np
import pandas as pd
from pydantic import Field, BaseModel, ConfigDict
from src.utils.logger import logger
from typing import Dict, List

class EnvironmentConfig(BaseModel):
    Data: pd.DataFrame = Field(..., description="Input data of external factors")
    actions_list: Dict = Field(...,description="list of possible actions that can be taken")
    internal_states_initialValue: Dict = Field(...,description="all internal states")
    environment_varibales: Dict = Field(..., description="any other variables that are not included in the internal or external states")

    model_config = ConfigDict(arbitrary_types_allowed=True)

class TheEnvironment(gym.Env):
    def __init__(self, config:EnvironmentConfig):
        super(TheEnvironment, self).__init__()
        self.config = config
        # number of features
        self.external_conds_data = config.Data
        self.ExternalDfeatures = len(self.external_conds_data.columns)

        # Environment variables
        self.EnvironVariables = config.environment_varibales

        # DEFINING THE ACTION SPACE AND OBSERVATION SPACES:
        possible_actions = config.actions_list
        Number_actions = len(possible_actions)
        self.action_space = spaces.Box(
            low=0,high=1,
            shape=(Number_actions,),
            dtype = np.float32
            )
        self.internal_states_initial_value = config.internal_states_initialValue
        self.obs_shape = self.ExternalDfeatures + len(self.internal_states_initial_value)
        self.observation_space = spaces.Box(
            low = -np.inf,
            high = np.inf,
            shape = (self.obs_shape,), dtype = np.float32
            )

        #current_stuep initialisation
        self.current_step = 0
        self.max_steps = len(self.external_conds_data)-1

        # internal states initialisation

        self.current_power_output = self.internal_states_initial_value['current_power_output']
        self.fuel_consumption_rate = self.internal_states_initial_value['fuel_consumption_rate']
        self.emissions_levels = self.internal_states_initial_value['emissions_levels']
        self.current_operating_costs = self.internal_states_initial_value['current_operating_costs']
        self.emissions_quota = self.internal_states_initial_value['emissions_quota']
        self.hours_main_turbine_since_maintenance = self.internal_states_initial_value['hours_main_turbine_since_maintenance']
        self.hours_secondary_turbine_since_maintenance = self.internal_states_initial_value['hours_secondary_turbine_since_maintenance']

        # ACTIONS INITIALISATION
        self.current_fuel_mixture_ratio = self.internal_states_initial_value['current_fuel_mixture_ratio']

        # Environment variables:
        self.Main_turbineOffCount = 0
        self.SecondTurbineOffCount =0
        self.EnergyStorage = self.internal_states_initial_value['Initial_storage']

        self.Total_reward = 0

Action to State Mapping

To evaluate the actions of RL agents and correctly map the state transitions, we need to define a model. This model will be able to predict or map the actions to the next states. The environment transitions to the next state s_(t+1) based on the current state and action:

where f represents the deterministic part of the environment dynamics (i.e. the model), and ϵ_t​ accounts for stochastic variations. For example, the internal state, Power output (P_t) can be defined in terms of the actions as follows:

We have defined a similar relationship between all actions and internal states.

Continuing from the code above, the ObservationSpace function takes a snap shot of the current state; it places all the external variables at the current step as well as the internal variables such as current power output and amount of energy in storage etc into a list which will be used by the agent to predict the actions to take.

    def ObservationSpace(self):
        """getting the current state of the environment"""

        observation_frame = []

        # EXTERNAL DATA
        for variable in self.external_conds_data.columns:
            df = self.external_conds_data[variable]
            min_val = min(df)
            max_val = max(df)
            mean_val = np.mean(df)

            if self.current_step < len(df):
                # print(f"column: {variable}, value: {df.iloc[self.current_step]}, type: {type(df.iloc[self.current_step])}")
                para_scaled = (df.iloc[self.current_step]-mean_val)/(max_val-min_val)
            # if the current step is longer than the dataframe
            # then add the last given data
            else:
                para_scaled = (df.iloc[-1]-mean_val)/(max_val-min_val)
            observation_frame.append(df.iloc[-1])

        # INTERNAL STATES
        # these are scaled approximately for now but needs to be done properly later
        observation_frame.append(self.current_power_output/400)
        observation_frame.append(self.fuel_consumption_rate/400)
        observation_frame.append(self.current_fuel_mixture_ratio)
        observation_frame.append(self.emissions_levels/500)
        observation_frame.append(self.current_operating_costs/1000)
        observation_frame.append(self.emissions_quota/6000)
        observation_frame.append(self.hours_main_turbine_since_maintenance/10)
        observation_frame.append(self.hours_secondary_turbine_since_maintenance/10)
        observation_frame.append(self.EnergyStorage/6000)

        return np.array(observation_frame)

Now we define the step function which given a set of actions by the agent, it will take a single step of those actions to observe the consequence of the actions i.e. the reward. It will also calculate all the internal states and observeration space (from the code above) that will be used to calculate the next set of actions.

def step(self, actions):
        self.current_step += 1

        # check if the number of steps is already above the max number of steps allowed
        done = self.current_step >= self.max_steps

        if done:
            reward, accum_reward = self.RewardCalculation()
            return self.ObservationSpace(), reward, done, False, {}
            # observation, reward, done, info

        # All the actions
        self.main_turbine_output = (actions[0]+ self.config.actions_list['main_turbine_output'][0])*self.config.actions_list['main_turbine_output'][1]
        self.secondary_turbine_output = (actions[1]+ self.config.actions_list['secondary_turbine_output'][0])*self.config.actions_list['secondary_turbine_output'][1]
        self.current_fuel_mixture_ratio = (actions[2]+ self.config.actions_list['current_fuel_mixture_ratio'][0])*self.config.actions_list['current_fuel_mixture_ratio'][1]
        self.generator_excitation = (actions[3]+ self.config.actions_list['generator_excitation'][0])*self.config.actions_list['generator_excitation'][1]
        self.emissions_control_intensity = (actions[4]+ self.config.actions_list['emissions_control_intensity'][0])*self.config.actions_list['emissions_control_intensity'][1]

        # calculate the resultant internal states:
        self.CalculateInternalStates()
        observation = self.ObservationSpace()
        reward, accum_reward = self.RewardCalculation()

        info = {
            "power_difference": self.current_power_demand - self.current_power_output,
            "emissions_quota": self.emissions_quota,
            "energy_storage": self.EnergyStorage,
            "reward": reward
        }

        return observation, reward, done, False, info # False is for truncated

Given the observation space, a set of actions are determined by the agent which in turn determine the state. To map the effect of the actions on the internal states variable, a function CalculateInternalStates is defined to update the internal states after a step is taken. These updates are defined based on simple rules of conservation.

# environment code for state transition
    def CalculateInternalStates(self):

        # Powerout
        main_T_eff = self.external_conds_data['main_turbine_efficiency'].iloc[self.current_step]
        second_T_eff = self.external_conds_data['secondary_turbine_efficiency'].iloc[self.current_step]
        base_power = self.main_turbine_output*main_T_eff + self.secondary_turbine_output*second_T_eff
        power_generated = base_power*self.generator_excitation


        if self.current_step < self.max_steps:
            # print(f"current step: {self.current_step}, max steps: {self.max_steps}, len of data: {len(self.external_conds_data['demand'])}")
            power_demand = self.external_conds_data['demand'].iloc[self.current_step+1]
        else:
            power_demand = self.external_conds_data['demand'].iloc[-1]

        # surplus goes to storage and deficit is made up from storage
        self.current_power_demand = power_demand
        power_difference = power_generated - power_demand
        # storage
        storage_capacity = self.EnvironVariables['Energy_storage_capacity']

        if power_difference >= 0:
            if self.EnergyStorage < storage_capacity:
                self.EnergyStorage += power_difference
                self.current_power_output = power_demand

        else:
            if self.EnergyStorage+power_difference >= 0: # i.e. there is actually enough left
                self.EnergyStorage -= power_difference
                self.current_power_output = power_demand

            else: # i.e. there is not enough left in storage
                self.current_power_output = self.EnergyStorage + power_generated
                self.EnergyStorage = 0

        # Fuel Consumption
        boiler_eff = self.external_conds_data['boiler_efficiency'].iloc[self.current_step]
        power_generated = self.main_turbine_output+self.secondary_turbine_output
        gas_consumption = power_generated*(1-self.current_fuel_mixture_ratio)/ boiler_eff
        coal_consumption = power_generated*self.current_fuel_mixture_ratio/boiler_eff
        self.fuel_consumption_rate = gas_consumption+coal_consumption

        # Emissions
        base_emissions = self.fuel_consumption_rate * (
            self.current_fuel_mixture_ratio*2 +
            (1-self.current_fuel_mixture_ratio))
        self.emissions_levels = base_emissions*(1-self.emissions_control_intensity/100)

        self.emissions_quota -= self.emissions_levels

        if self.current_step % (24*28) == 0: # i.e. every 4 weeks reset emissions quota
            self.emissions_quota = self.internal_states_initial_value['emissions_quota']

        # operating costs
        coal_price = self.external_conds_data['coal_price'].iloc[self.current_step]
        gas_price = self.external_conds_data['gas_price'].iloc[self.current_step]
        self.current_operating_costs = self.fuel_consumption_rate*(
            self.current_fuel_mixture_ratio*coal_price + 
            (1-self.current_fuel_mixture_ratio)*gas_price)

        # update the turbine operating times
        if self.main_turbine_output > 0:
            self.hours_main_turbine_since_maintenance += 1


        # maintenance time required
        maintenance_time = self.EnvironVariables['Turbine_maintenance_time']
        if self.main_turbine_output == 0:
            self.Main_turbineOffCount += 1
            if self.Main_turbineOffCount >= maintenance_time:
                self.hours_main_turbine_since_maintenance = 0
                self.Main_turbineOffCount = 0

        if self.secondary_turbine_output > 0:
            self.hours_secondary_turbine_since_maintenance += 1

        if self.secondary_turbine_output == 0:
            self.SecondTurbineOffCount += 1
            if self.SecondTurbineOffCount >= maintenance_time:
                self.hours_secondary_turbine_since_maintenance = 0
                self.SecondTurbineOffCount =0

Reward Function

Let’s define a reward function that will encapsulate the operational objectives which include meeting power demand, operational costs, emissions costs, maintenance costs and emissions control costs.

For the power demand, we can penalise any deviations from the demand:

Operating costs, we’ll just include fuel consumption costs:

Where F is the fuel consumption rates and the superscript is the fuel source. Emissions costs, penalising the model if the emissions level exceeds the quotas:

To prevent the model from running any instruments for extended operation without maintenance, we can add a penalising term:

Cost of employing emissions control measures:

All of these can be combined to define the reward function that encapsulate all the different objects:

All of these equations are defined in the code as follows.

    def RewardCalculation(self):
        # Ensure meeting the power demand
        power_difference = self.current_power_demand - self.current_power_output

        # maintenance of turbines:
        Turbine_use_limit = self.EnvironVariables['Turbine_use_b4_maintenance']
        #Main turbine
        if self.hours_main_turbine_since_maintenance > Turbine_use_limit:
            outstanding_maintenance_MT = np.exp(self.hours_main_turbine_since_maintenance - Turbine_use_limit)
        else:
            outstanding_maintenance_MT =0

        # Secondary Turbine
        if self.hours_secondary_turbine_since_maintenance > Turbine_use_limit:
            outstanding_maintenance_ST = np.exp(self.hours_secondary_turbine_since_maintenance - Turbine_use_limit)
        else:
            outstanding_maintenance_ST =0

        # MEETING EMISSION QUOTA
        if self.emissions_quota < 0:
            extra_emission = -self.emissions_quota
        else:
            extra_emission = 0

        # costs associated with emissions control
        emissions_control_costs = self.emissions_control_intensity*1000*np.exp(self.emissions_control_intensity)


        # including operating and fuel costs
        electricity_price = self.external_conds_data['electricity_price'].iloc[self.current_step]
        reward = (5*self.current_power_output*electricity_price -
                  self.current_operating_costs - power_difference*1000 -
                  outstanding_maintenance_MT * 20000 - outstanding_maintenance_ST* 20000 -
                  extra_emission * 5 -
                  emissions_control_costs)

        self.current_step_reward = reward/1e6 # reward scaled for numerical stability
        self.Total_reward += self.current_step_reward

        return reward, self.Total_reward

Defining The Agent

We can now start defining the agent with stable_baseline3 library. The agent is defined as part of an overall class with various functions that allow for the agent to be trained, used for inference predictions and also run on the validation data to check for over fitting during training.

class AgentConfig(BaseModel):
    total_timesteps: int = Field(..., description="total number of steps")
    environment: DummyVecEnv = Field(..., description="Environment for the agent to run")
    validation_timesteps: int = Field(..., description="Number of steps for the validation")
    train_timesteps:int = Field(default=100, description="ONLY FOR TESTING!! USED TO UPLOAD SAVED MODEL")
    model_config = ConfigDict(arbitrary_types_allowed=True)

class PPOAgent:
    def __init__(self, config: AgentConfig):
        self.callback = PolicyGradientLossCallback()
        self.Agent = PPO(
            "MlpPolicy", 
            config.environment,
            verbose = 1,
            learning_rate=1e-5,
            gamma=0.99) # batch_size=256
        self.config = config

    def train(self):
        self.Agent.learn(total_timesteps=self.config.total_timesteps, callback = self.callback)
        logger.info("Finished training")

    def predict(self, observation):
        action,_ = self.Agent.predict(observation)
        return action

    def validate(self):
        environment = self.config.environment
        observation = environment.reset()
        Rewards = []
        rewards_accum =0
        accumulative_reward = np.zeros(self.config.validation_timesteps)
        for ii in range(self.config.validation_timesteps):
            actions, _ = self.Agent.predict(observation)
            observation, reward, done, _ = environment.step(actions) # Unpacking only 4, even though 5 is returned - issue with current version of DummyVecEnv
            Rewards.append(reward)
            rewards_accum += reward
            accumulative_reward[ii] = rewards_accum

            # check if the simulation is complete
            if done:
                observation = environment.reset()
        print(f"Accumulative Reward at the end of {self.config.validation_timesteps} steps validation: {accumulative_reward[-1]}")
        logger.info(f"Accumulative Reward at the end of {self.config.validation_timesteps} steps validation: {accumulative_reward[-1]}")

        return np.array(Rewards), accumulative_reward

Training the Agent

We can now train and test the agent. The code for train and test functions are very similar; it is bringing the agent code above and the defined environment together and training it for a predefined number of steps followed by validation or testing of the agent.

class ModelEvaluationConfig(BaseModel):
    # Data paths
    training_data_path: str = Field(default="data/processed/train.csv", description="path to training data")
    val_data_path: str = Field(default = "data/processed/val.csv", description="path to validation data")
    test_data_path: str = Field(default="data/processed/test.csv", description="path to testing data")

    # agent environment variables
    agent_possible_actions: Dict = Field(..., description="all possible actions and value range")
    train_total_timesteps: int = Field(default=20000, description="total number of steps during training")
    validation_timesteps: int = Field(default=300, description="Number of steps for the validation")
    test_timesteps: int = Field(default=300,description="Number of steps during testing")

    # Environment variables:
    internalStates_InitialVal: dict = Field(..., description="a dictionary on all internal states and their initial value")
    environment_variables: dict = Field(..., description="a dictionary of any other environmental variables")

    # loading pretrained agent during testing
    use_pretrained_agent: bool = Field(default= True, description = "if true, it will load pretrained agent before testing")

    model_config=ConfigDict(arbitrary_types_allowed=True)

class Model_train:
    def __init__(self, config:ModelEvaluationConfig):
        self.config = config

    def train_agent(self):
        ## TRAINING
        #training data
        train_data = pd.read_csv(Path(self.config.training_data_path))
        # Create environment config
        env_config = EnvironmentConfig(
            Data=train_data,
            actions_list=self.config.agent_possible_actions,
            internal_states_initialValue=self.config.internalStates_InitialVal, 
            environment_varibales=self.config.environment_variables  
        )
        train_environment = DummyVecEnv([lambda: TheEnvironment(env_config)])
        agent_config = AgentConfig(total_timesteps=self.config.train_total_timesteps,
                                   environment=train_environment,
                                   validation_timesteps=self.config.validation_timesteps
                                   )

        agent = PPOAgent(agent_config)
        # training the agent:
        agent.train()

        # VALIDATION OF THE AGENT
        val_data = pd.read_csv(Path(self.config.val_data_path))
        val_env_config = EnvironmentConfig(
            Data=val_data,
            actions_list=self.config.agent_possible_actions,
            internal_states_initialValue=self.config.internalStates_InitialVal, 
            environment_varibales=self.config.environment_variables  
        )
        val_environment = DummyVecEnv([lambda: TheEnvironment(val_env_config)])

        # evaluate the agent
        Reward, accumulative_reward = agent.validate()

        # save trained agent:
        agent.save_trained_agent()

        return train_environment, val_environment, agent, Reward, accumulative_reward

    def test_agent(self):

        test_data = pd.read_csv(Path(self.config.test_data_path))
        # Create environment config
        env_config = EnvironmentConfig(
            Data=test_data,
            actions_list=self.config.agent_possible_actions,
            internal_states_initialValue=self.config.internalStates_InitialVal, 
            environment_varibales=self.config.environment_variables  
        )
        test_environment = DummyVecEnv([lambda: TheEnvironment(env_config)])

        agent_config = AgentConfig(total_timesteps=self.config.test_timesteps,
                                   environment=test_environment,
                                   validation_timesteps=self.config.validation_timesteps,
                                   train_timesteps=self.config.train_total_timesteps
                                   )

        agent = PPOAgent(agent_config)
        # load pretrained agent
        if self.config.use_pretrained_agent:
            agent.load_trained_agent()

        accum_reward = np.zeros(self.config.test_timesteps)
        Rewards = np.zeros(self.config.test_timesteps)
        accum_reward_perstep = 0 
        observation_space = []
        obs = test_environment.reset()
        for ii in range(self.config.test_timesteps):
            action = agent.predict(obs)
            obs, reward, done, info = test_environment.step(action)
            Rewards[ii] = reward
            accum_reward_perstep += reward
            accum_reward[ii] = accum_reward_perstep
            observation_space.append(obs)

            if done:
                obs = test_environment.reset()

        return accum_reward,Rewards, observation_space

The complete code and pipeline is available on github. We can test the agents ability to operate on testing data by using an agent only trained for 10k steps as a baseline and compared to performance of an agent trained for 500k steps. Below is an image of the cumulative reward which shows a significance difference. The agent trained only for 10k steps may as well take random actions, the cumulative reward is increasingly dropping. Where as an agent trained for 500k steps has learned to operate to increase the cumulative reward so that the powerplant is operating on profit and not losses.

Despite a lack of fine tuning the model hyperparameters, it has done a decent job actually, we could tune the model further to get improved performance.

This brings us to the end of this article. Reinforcement learning is one of the most exciting and less developed branches of AI, it will be very exciting to see the developments of this field but also it’s application to different problems. Finally, thank you for taking the time to read this article, I hope you found it useful in your understanding of reinforcement learning or their mathematical background. This article is only a humble introduction, there is a lot more to read about the fundamental concepts that underpins reinforcement learning.

Unless otherwise noted, all images are by the author