Lecture 15: model-based algorithms for finite MDP
Goal: derive algorithms to solve MDP when the model is known.
References
- Chapter 2: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).
Problem formulation
Consider the finite MDP \((X,U,P,c)\) with infinite-horizon exponentially-discounted cost
\[ \min_u \text{E}[c(x,u)]\ \ \text{s.t.}\ \ x^+\sim P(x,u),\ \ c(x,u) = \sum_{t=0}^\infty \gamma^t \mathcal{L}(x_t,u_t) \]
where \(\gamma\in(0,1)\).
Value iteration
Consider the Bellman operator \(T:\R^X\rightarrow\R^X\) defined by
\[ \forall x : (T v)(x) = \min_{u\in U}\sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v(x^+)). \]
Q: use the fact that \(\|Tv - Tw\| \leq \gamma \| v - w \|\) for all \(v,w\in\R^X\) to propose an iterative algorithm to approximate the optimal value \(v^*\) and assess the computational complexity.
Consider what happens with respect to the optimal policy \(w = v^*\) when we repeatedly apply the operator: \(v \mapsto Tv \mapsto T^2 v \mapsto \cdots \mapsto T^k v\): \[ \| T^k v - v^* \| \leq \gamma ^k \| v - v^* \| \implies \lim_{k\rightarrow\infty} T^k v = v^*. \] Each evaluation of \(T\) requires \(O(\#(X)\#(U))\) operations, but in general an infinite number of iterations are needed to obtain \(v^*\) exactly.
This algorithm is termed value iteration. Given the optimal value \(v^*\) it is trivial to determine an optimal deterministic policy \(\pi:X\rightarrow U\) as
\[ \forall x \in X : \pi^*(x) = \arg\min_{u\in U}\sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^*(x^+)). \]
Policy iteration
Consider the Bellman operator \(T_\pi:\R^X\rightarrow\R^X\) defined by
\[ \forall x : (T_\pi v)(x) = \sum_{u\in U}\pi(u | x) \sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v(x^+)). \]
Given policy \(\pi:X\rightarrow\Delta(U)\) we can compute \(v^\pi\) by solving the affine equation \(v^\pi = T_\pi v^\pi\) in \(O(\#(X)^3)\) operations (the exponent can be decreased to less than 3 with some effort, but cannot be less than or equal to 2).
Q: given policy \(\pi\) with value \(v^\pi\), propose an algorithm to improve the policy.
One choice is the “greedy” update \[ \forall x \in X : \pi^+(x) = \arg\min_{u\in U}\sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^\pi(x^+)). \]
It turns out that: \(v^{\pi^+}(x) \leq v^\pi(x)\) for all \(x\in X\); \(\exists x^* : v^{\pi^+}(x^*) < v^\pi(x^*)\); and \(\pi \mapsto \pi^+ \mapsto \pi^{++} \mapsto \cdots \mapsto \pi^*\), i.e. this iteration converges to an optimal deterministic policy in a finite number of iterations!