Lecture 16: model-free algorithms for finite MDP
Goal: derive algorithms to solve MDP when the model is unknown.
References
- Chapter 5: Bertsekas, D. P. & Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific (1996).
Problem formulation
Consider the finite MDP \((X,U,P,c)\) with infinite-horizon exponentially-discounted cost
\[ \min_u \text{E}[c(x,u)]\ \ \text{s.t.}\ \ x^+\sim P(x,u),\ \ c(x,u) = \sum_{t=0}^\infty \gamma^t \mathcal{L}(x_t,u_t) \]
where \(\gamma\in(0,1)\).
Given a policy \(\pi:X\rightarrow\Delta(U)\), recall the value function \(v^\pi:X\rightarrow\R\) is defined by
\[ \forall x\in X : v^\pi(x) = \text{E}\left[ \mathcal{L}(x,u) + \gamma v^\pi(x^+) \right]. \]
This form of the Bellman equation suggests we can approximate \(v^\pi\) using a very simple algorithm: Robbins-Monroe / Monte Carlo estimation.
Robbins-Monroe / Monte Carlo estimation
Citation: Robbins, H. & Monro, S. A Stochastic Approximation Method. Ann. Math. Stat. 22, 400–407 (1951).
Suppose you want to find the root of a function \(r:\R^n\rightarrow\R^n\), that is, a vector \(v\in\R^n\) such that \(r(v) = 0\). However, suppose you are only able to make a query \(v\) and obtain a noisy observation \(y = r(v) + \eta\) where \(\eta\) is a random variable with \(\text{E}[\eta] = 0\).
Idea: iteratively update the query \(v^+ = v - \alpha y\) where \(\alpha > 0\) is a step size parameter.
This very simple idea is guaranteed to converge to the root so long as the function satisfies some regularity properties and the step sizes decrease (i) fast but (ii) not too fast: letting \(\{\alpha_n\}_{n=1}^\infty\) denote the sequence of step sizes, we require
\[ \sum_{n=1}^\infty \alpha_n = \infty,\ \ \sum_{n=1}^\infty \alpha_n^2 < \infty. \]
Temporal Difference (TD) learning
A straightforward application of the Robbins-Monroe algorithm to the Bellman equation along a trajectory \((x,u):T\rightarrow X\times U\) yields
\[ v^+(x_t) = v(x_t) - \alpha \left[ v(x_t) - \mathcal{L}(x_t,u_t) - \gamma v(x_{t+1}) \right]. \]
To guarantee convergence the step sizes \(\alpha\) must decrease (i) fast but (ii) not too fast as defined in the preceding section.
Q learning
Given a policy \(\pi:X\rightarrow\Delta(U)\) with value \(v^\pi : X\rightarrow\R\), consider the state-action policy quality \(q^\pi:X\times U\rightarrow\R\) defined by
\[ q^\pi(x,u) = \sum_{x^+\in X} P(x^+ | x,u) (\mathcal{L}(x,u) + \gamma v^\pi(x^+)). \]
This function computes the expected cost of (1.) choosing action \(u\) in state \(x\) and then (2.) using \(\pi\) from then on.
Note that the optimal quality function (i.e. the quality of the optimal policy) satisfies a Bellman equation
\[ q^*(x,u) = E[\mathcal{L}(x,u) + \gamma\min_{w\in U} q^*(x^+,w)] \]
because \(v^*(x) = \min_{u\in U} q^*(x,u)\) so a straightforward application of the Robbins-Monroe algorithm yields an algorithm for approximating \(q^*\) from data:
\[ q^+(x,u) = q(x,u) - \alpha\left( q(x,u) - \mathcal{L}(x,u) - \gamma\min_{w\in U} q(x^+,w) \right). \]
Note that \(u^*(x) = \min_{u\in U}q^*(x,u)\), so the optimal policy can be directly obtained from the optimal \(q\) function.