Lecture 5: dynamic programming
Goal: apply nonlinear programming techniques to solve multi-stage optimization problems.
References
- Section 3.4: Stengel, R. F. Optimal Control and Estimation. (Dover, New York, 1994).
- Chapter 6: Lewis, F. L., Vrabie, D. & Syrmos, V. L. Optimal Control. (John Wiley & Sons, 2012).
Problem formulation
Suppose we are given a discrete-time difference equation (DT DE) \(x^+ = F(x,u), x\in\mathbb{R}^d, u\in\mathbb{R}^m\) and we wish to choose inputs over time \(u:[0,t)\rightarrow\mathbb{R}^m\) to minimize
\[ c(x,u) = \ell(t,x(t)) + \sum_{s = 0}^{t - 1}\mathcal{L}(s,x(s),u(s)). \]
Bellman equation
Idea: the optimal control \(u(t)\) to apply at time \(t\) depends only on \(x(t)\) – not on previous states or inputs.
This observation leads naturally to working backward from the final time: letting \(v_t^*(x(t))\) denote the lowest (i.e. optimal) cost achievable from state \(x(t)\) at time \(t\), we have
\[ v_s^*(x(s)) = \min_{u(s)} \left[ \mathcal{L}(s,x(s),u(s)) + v_{s+1}^*(x(s+1)) \right]. \]
where \(v_s^*\) is referred to as the value function. This is referred to as a Bellman equation after Richard Bellman who published this key insight in 1957.
Importantly, the Bellman equation is an NLP at every time \(s\). So in principle the optimal control inputs can be obtained by solving a sequence of NLP backward in time!
Since the optimal trajectory is not known in advance, a key limitation is that this NLP must be solved for every possible trajectory, which is impractical in general.
Linear dynamics, quadratic cost
Q: consider \(x^+ = a x + b u,\ x,u\in\mathbb{R},\ c_\tau(x,u) = \sum_{s = \tau}^{t} q(s) x(s)^2 + r(s) u(s)^2\). Determine the optimal input and value \(v^*\) at \(\tau = t\).
\(c_t(x,u) = q(t) x(t)^2 + r(t) u(t)^2\)
\(v_t^*(x(t)) = q(t) x(t)^2\) is optimal with input \(u^*(t) = 0\).
Q: determine the optimal input and value at \(\tau = t - 1\) using Bellman’s equation.
\[\begin{align*} v_{t-1}^* & = \min_{u(t-1)}\left[ q(t-1) x(t-1)^2 + r(t-1) u(t-1)^2 + q(t) x(t)^2 \right] \\ & = \min_{u_{t-1}}\left[ q_{t-1} x_{t-1}^2 + r_{t-1} u_{t-1}^2 + q_t (a x_{t-1} + b u_{t-1})^2 \right]. \end{align*}\] Differentiating with respect to \(u_{t-1}\) yields \(2 r_{t-1} u_{t-1} + 2 q_t(a x_{t-1} + b u_{t-1}) b = 0\) \(\iff\) \(u_{t-1} = \frac{-a b q_t}{b^2 + r_{t-1}} x_{t-1}\) \(\leftarrow\) – linear in \(x_{t-1}\)!
We’ll generalize this very special case of linear dynamics and quadratic cost to the multidimensional case in the next lecture.
Takeaway
We’ll use Bellman equations both analytically (to verify optimality) and computationally (to synthesize optimal control inputs).