Lecture 13: Bellman equation for finite MDP
Goal: derive Bellman equation for finite MDP.
References
- Chapter 2: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).
- Chapter 3: Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction. (The MIT Press, 1998).
Problem formulation
Consider the finite MDP \((X,U,P,c)\) with infinite-horizon exponentially-discounted cost
\[ \min_u \text{E}[c(x,u)]\ \ \text{s.t.}\ \ x^+\sim P(x,u),\ \ c(x,u) = \sum_{t=0}^\infty \gamma^t \mathcal{L}(x_t,u_t) \]
where \(\gamma\in(0,1)\).
Bellman equation
Given a policy \(\pi:X\rightarrow\Delta(U)\), we define the value function \(v^\pi:X\rightarrow\R\) as
\[ \forall x\in X : v^\pi(x) = \text{E}\left[ c(x,u) \mid x_0 = x, u_t\sim\pi(x_t)\right]. \]
Q: show that \(v^\pi\) satisfies the Bellman equation
\[ \forall x : v^\pi(x) = \sum_{u\in U}\pi(u | x) \sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^\pi(x^+)) \]
where \(\pi(u | x)\) is the probability of choosing \(u\in U\) from \(x\in X\) and \(P(x^+ | x,u)\) is the probability of reaching \(x^+\in X\) by choosing \(u\in U\) from \(x\in X\).
Expanding the definition of \(v^\pi\) and regrouping terms we have \[\begin{align*} v^\pi(x) & = \text{E}\left[ c(x,u) | x_0 = x, u_t\sim\pi(x_t)\right] \\ & = \text{E}\left[ \gamma^0 \mathcal{L}(x_0,u_0) + \sum_{t=1}^\infty \gamma^t \mathcal{L}(x_t,u_t) | x_0 = x, u_t\sim\pi(x_t), x_1\sim P(x_0,u_0) \right] \\ & = \sum_{u_0\in U}\pi(u_0 | x_0)\sum_{x^+\in X}P(x^+ | x_0,u_0) \mathcal{L}(x_0,u_0) \\ & \quad + \sum_{u_0\in U} \pi(u_0 | x_0) \sum_{x^+\in X} P(x^+ | x_0, u_0) \text{E}\left[\sum_{t=1}^\infty \gamma^t\mathcal{L}(x_t,u_t) | x_1 = x^+, u_t\sim\pi(x_t)\right]. \end{align*}\] But with \(\tau = t - 1\) the final expectation is equivalent to \(\gamma \text{E}\left[\sum_{\tau=0}^\infty \gamma^\tau\mathcal{L}(x_\tau,u_\tau) | x_0 = x^+, u_\tau\sim\pi(x_\tau)\right] = \gamma v^\pi(x^+)\). Combining terms gives the desired result.
Solution to Bellman equation
Note: the value \(v^\pi:X\rightarrow\R\) appears linearly in the Bellman equation!
Q: treating \(v^\pi \in \R^{\#(X)}\) as a vector, determine \(L^\pi,b^\pi\) so that \(L^\pi v^\pi = b^\pi\) is the Bellman equation.
Approach: move the \(v^\pi\) terms to the left-hand side of the Bellman equation and factor out coefficients that multiply them, treating \(x^+,x,u\) as indices (i.e. assuming the states and actions have been enumerated). \[ [L^\pi]_{x^+,x} = \delta_{x^+,x} - \gamma\sum_{u\in U}\pi(u|x)P(x^+|x,u), \] \[ [b^\pi]_x = \sum_{u\in U}\pi(u|x)\sum_{x^+\in X}P(x^+|x,u)\mathcal{L}(x,u). \]
Takeaway: the value of any policy can be computed by solving a system of linear equations.