Skip to main content
$$ \newcommand{\R}{\mathbb{R}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} $$

Lecture 11: Solution of finite MP

Goal: characterize dynamics of finite Markov Processes.

References

  • Chapter 2: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).
  • Chapter 3: Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction. (The MIT Press, 1998).

Dynamics of Markov processes

Let \((X,U,P)\) be a finite Markov Process, i.e. \(\#(X),\#(U) < \infty\), that defines a stochastic difference equation \(x^+\sim P(x,u)\).

Q: starting from state \(p\in\Delta(X)\) and applying policy \(\pi:X\rightarrow\Delta(U)\), determine the next state \(p^+\in\Delta(X)\). It may help to regard \(p\) as a function \(p:X\rightarrow[0,1]\) and then determine \(p^+(x^+)\) for all \(x^+\in X\).

We can compute \(p^+(x^+)\) for each \(x^+\in X\) using the rules of probability: \[ p^+(x^+) = \sum_{x\in X}\left( p(x)\left[ \sum_{u\in U}\pi(u|x)P(x^+|x,u)\right]\right). \]

Note that this equation defines a deterministic (!) difference equation \(p^+ = F(p)\) where \((p:X\rightarrow[0,1])\in\Delta(X)\) can be regarded as a vector \(p\in\R^N\) where \(N = \#(X)\) by choosing an indexing for the elements of \(X\).

Q: show that \(F\) is linear (!!). Specifically: find \(\Gamma\in\R^{N\times N}\) such that \(F(p) = \Gamma p\).

Choose an indexing for \(X\) so that \(x\in X\) can be used to index a row or column of an \(N\times N\) matrix. Then \([\Gamma]_{x^+,x} = \sum_{u\in U}\pi(u|x) P(x^+|x,u)\) yields \(p^+ = \Gamma p\).

Asymptotics

Since the dynamics of Markov processes are linear, we can use linear systems theory to characterize asymptotic properties. Specifically, we’re studying a discrete-time linear time-invariant difference equation (DT LTI DE) \(p^+ = \Gamma p\).

First note that trajectories are given by \(p_t = \Gamma^t p_0\) for any \(t\in\N\) where \(\Gamma^t = \underbrace{\Gamma\cdots\Gamma}_{t\ \text{times}}\).

Second note that \(\Gamma\) has special properties arising from the probability distributions used to define it: \(1^\top\Gamma = 1^\top\) where \(1^\top = (1,\dots,1)\in\R^{1\times N}\), which means \(\Gamma\) is termed left-stochastic. Such matrices have \(1\in\operatorname{spec}\Gamma\) (from the preceding calculation) and \(\forall\lambda\in\operatorname{spec}\Gamma : |\lambda| \leq 1\) (from the Gershgorin circle theorem).

Fact: if \(\Gamma\) is irreducible (i.e. from every \(x\) it’s possible to reach every other \(x'\)) and aperiodic (i.e. states are not forced to cycle with fixed periods) then \(\lim_{t\rightarrow\infty} p_t = \bar{p}\) where \(\Gamma \bar{p} = \bar{p}\) is the unique right-eigenvector with eigenvalue \(1\). (A sufficient condition for irreducibility and aperiodicity is \([\Gamma]_{x,x^+} > 0\) for all \(x,x^+\).)