Skip to main content

$$ \newcommand{\R}{\mathbb{R}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} $$

Lecture 14: Bellman operators for finite MDP

Goal: derive and analyze Bellman operators for finite MDP.

References

• Chapter 2: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).

Problem formulation

Consider the finite MDP \((X,U,P,c)\) with infinite-horizon exponentially-discounted cost

\[ \min_u \text{E}[c(x,u)]\ \ \text{s.t.}\ \ x^+\sim P(x,u),\ \ c(x,u) = \sum_{t=0}^\infty \gamma^t \mathcal{L}(x_t,u_t) \]

where \(\gamma\in(0,1)\).

Bellman operators

Given policy \(\pi:X\rightarrow\Delta(U)\) we know that the value \(v^\pi:X\rightarrow\R\) defined by \(v^\pi(x) = \text{E}\left[ c(x,u) \mid x_0 = x, u_t\sim\pi(x_t)\right]\) satisfies the Bellman equation

\[ \forall x : v^\pi(x) = \sum_{u\in U}\pi(u | x) \sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^\pi(x^+)). \]

Idea: use this equation to define an operator \(T_\pi:\R^X\rightarrow\R^X:v\mapsto T_\pi v = v^+\) by

\[ \forall x : (T_\pi v)(x) = \sum_{u\in U}\pi(u | x) \sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v(x^+)). \]

Note that the value \(v^\pi\) of policy \(\pi\) is a fixed point of this operator: \(T_\pi v^\pi = v^\pi\). We’ll soon see that the operator has extremely useful properties.

Analogously, noting that the optimal value \(v^*:X\rightarrow\R\) of the optimal policy \(\pi^*:X\rightarrow\Delta(U)\) satisfies

\[\begin{align*} \forall x : v^*(x) & = \sum_{u\in U}\pi^*(u | x) \sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^*(x^+)) \\ & = \min_{u\in U}\sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v^*(x^+)) \end{align*}\]

so we can define an operator \(T:\R^X\rightarrow\R^X : v\mapsto Tv = v^+\) by

\[ \forall x : (T v)(x) = \min_{u\in U}\sum_{x^+\in X} P(x^+ | x,u)(\mathcal{L}(x,u) + \gamma v(x^+)). \]

Operator properties

Q: what kind of operator is \(T_\pi\)? What about \(T\)?

\(T_\pi\) is affine! In fact \(T_\pi v = \gamma A^\pi v + b^\pi\) where \[\begin{align} [A^\pi]_{x,x^+} & = \sum_{u\in U}\pi(u|x)P(x^+|x,u), \\ [b^\pi]_x & = \sum_{u\in U}\pi(u|x) \sum_{x^+\in X}P(x^+|x,u)\mathcal{L}(x,u). \end{align}\]

\(T\) is nonlinear (in fact: nonsmooth!).

Given \(k\in\N\) let \(T_\pi^k = \underbrace{T_\pi\circ T_\pi\circ\cdots\circ T_\pi}_{k\ \text{times}}\), \(T^k = \underbrace{T\circ T\circ\cdots\circ T}_{k\ \text{times}}\) and endow \(\R^X\) with the max norm \(\|v\| = \max_{x\in X}|v(x)|\).

Theorem (Bellman operators are contractions; cf 2.5 in BT96):

\[\begin{align*} \forall v,w\in\R^X,\pi:X\rightarrow\Delta(U):\ & \|Tv - Tw\| \leq \gamma \| v - w \|, \\ & \|T_\pi v - T_\pi w\| \leq \gamma \| v - w \|. \end{align*}\]

Corollary (by Banach contraction mapping Theorem): 1. \(\forall v\in \R^X : \lim_{k\rightarrow\infty} T^k v = v^* = T v^*\) is the optimal value; 2. \(\forall v\in \R^X : \lim_{k\rightarrow\infty} T_\pi^k v = v^\pi = T_\pi v^\pi\) is the value of \(\pi\); 3. \(\pi\) is optimal if and only if \(T_\pi v^\pi = T v^\pi = v^\pi\).