Discrete Markov chains

The discrete case, generally known as a Markov chain, is discussed on this page.

The Markov approach can be applied to the random behaviour of systems that vary discretely or continuously with respect to time and space. This discrete or continuous random variable is known as a stochastic process. Not all stochastic processes can be modelled using the basic Markov approach although there are techniques available for modelling some additional stochastic processes using extensions of this basic method.

Discrete Markov chains

In order for the basic Markov approach to be applicable, the behaviour of the system must be characterized by a lack of memory, that is, the future states of a system are independent of all past states’ except the immediately preceding one. Therefore the future random behaviour of the system only depends on where it is at present, not on where it has been in the past. In addition, the process must be stationary, sometimes called homogeneous, for the approach to be applicable. This means that the behaviour of the system must be the same at all points of time irrespective of the point of time being considered, i.e., the probability of making a transition from one given state to another is the same (stationary) at all times in the past and future. It is evident from these two aspects, lack of memory and being stationary, that the Markov approach is applicable to those systems whose behaviour can be described by a probability distribution that is characterized by a constant hazard rate, i.e., Poisson and exponential distributions since only if the hazard rate is constant does the probability of making a transition between two states remain constant at all points of time. If this probability is a function of time or the number of discrete steps, then the process is non-stationary and designated as non-Markovian. The discrete case, generally known as a Markov chain, The continuous case, generally known as a Markov process.

Modelling concepts

The basic concepts of Markov modelling can be illustrated by considering

the simple system is shown in Figure

In this system two system states are identifiable, being designated 1 and 2. The probabilities of remaining in or leaving a particular state in a finite time are also shown in Figure, and these probabilities are assumed to be constant for all times into the future. This is a discrete Markov chain since the system is stationary and the movement between states occurs in discrete steps.

The stochastic transitional probability matrix.

These transition probabilities can be represented by the following matrix P