Skip to main content

$$ \newcommand{\R}{\mathbb{R}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} $$

Lecture 10: Markov processes

Goal: mathematical model for control systems with random dynamics.

References

• Chapter 2: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).
• Chapter 3: Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction. (The MIT Press, 1998).

Mathematical model

To generalize from determinstic to stochastic dynamics, we replace \(x^+ = F(x,u)\) with \(x^+\sim P(x,u)\) where \(x\in X\), \(u\in U\), \(P:X\times U\rightarrow\Delta(X)\), and \(\Delta(X)\) is the set of probability distributions over \(X\).

Recall from the previous lecture that the formalization of probabilities differs depending on whether \(X\) is a finite set (\(\#(X) < \infty\)) or an infinite set (\(\#(X) = \infty\), e.g. \(X = \R^n\)).

When \(X\) is finite, we can regard \(p\in\Delta(X)\) as a function \(p:X\rightarrow[0,1]\) with the property that \(\sum_x p(x) = 1\).

When \(X\) is infinite, e.g. \(X = \R^n\), in general we need to be careful about “measurability” but in this class we will restrict to Gaussian random variables.

State

Q: what is a “state”?

the state of a dynamical system is all and only the information needed to predict the future.

Q: what is the state of \(x^+ \sim P(x,u)\)?

The state is not a single \(x\in X\). Instead, the state is a probability distribution \(p\in\Delta(X)\).

from imports import *

# Dynamics: stable 2D linear system
A = np.array([[0.9, -0.3],
              [0.2,  0.85]])
x0 = np.array([1.0, 0.0])
T = 30
sigma = 0.15
N = 200  # number of stochastic trajectories

# --- Deterministic trajectory ---
x_det = np.zeros((T + 1, 2))
x_det[0] = x0
for t in range(T):
    x_det[t + 1] = A @ x_det[t]

# --- Stochastic trajectories ---
x_sto = np.zeros((N, T + 1, 2))
x_sto[:, 0, :] = x0
for t in range(T):
    noise = np.random.randn(N, 2) * sigma
    x_sto[:, t + 1, :] = (A @ x_sto[:, t, :].T).T + noise

# Compute median and IQR across trajectories
x_med = np.median(x_sto, axis=0)          # (T+1, 2)
x_q25 = np.percentile(x_sto, 25, axis=0)  # (T+1, 2)
x_q75 = np.percentile(x_sto, 75, axis=0)  # (T+1, 2)

time = np.arange(T + 1)
colors = ['tab:blue', 'tab:orange']
labels = [r'$x_1$', r'$x_2$']

fig, axes = plt.subplots(1, 2, sharey=True)

# --- Left: deterministic ---
ax = axes[0]
for i in range(2):
    ax.plot(time, x_det[:, i], color=colors[i], label=labels[i])
ax.set_title(r'$x^+ = F(x,u)$')
ax.set_xlabel(r'$t$')
ax.set_ylabel(r'$x$')
ax.legend()

# --- Right: stochastic distribution ---
ax = axes[1]
for i in range(2):
    ax.plot(time, x_med[:, i], color=colors[i], label=labels[i])
    ax.fill_between(time, x_q25[:, i], x_q75[:, i],
                    color=colors[i], alpha=0.3)
ax.set_title(r'$x^+ \sim P(x,u)$')
ax.set_xlabel(r'$t$')
ax.legend()

fig.tight_layout()
show_with_alt(fig,
    "Two side-by-side plots of x vs time for a stable 2D linear system. "
    "Left: deterministic trajectory x^+ = F(x,u) showing x1 and x2 converging smoothly to zero. "
    "Right: stochastic distribution x^+ ~ P(x,u) showing the median of 200 sample trajectories "
    "with interquartile range shaded, illustrating the spread introduced by process noise."
)

Markov processes (MP)

So \(x^+ \sim P(x,u)\) is a stochastic (i.e. random) process wherein knowing the current state distribution \(p\in\Delta(X)\) suffices to predict the future evolution of the state distribution. This type of process is called Markovian.

Definition: a Markov process (MP) is specified by \((X,U,P)\) where \(X\), \(U\) are sets and \(P:X\times U\rightarrow\Delta(X)\). It is termed a Markov chain (MC) if \(U = \emptyset\) or if \(u\sim\pi(x)\) where \(\pi:X\rightarrow\Delta(U)\) is given.