Lecture 21: model-predictive control
Goal: solve non-linear optimal control problems on-line.
References
- Hewing, L., Wabersich, K. P., Menner, M. & Zeilinger, M. N. Learning-based model predictive control: Toward safe learning in control. Annu. Rev. Control Robot. Auton. Syst. 3, 269–296 (2020).
- Aswani, A., Gonzalez, H., Sastry, S. S. & Tomlin, C. Provably safe and robust learning-based model predictive control. Automatica 49, 1216–1226 (2013).
- Hsu, K.-C., Hu, H. & Fisac, J. F. The safety filter: A unified view of safety-critical control in autonomous systems. Annu. Rev. Control Robot. Auton. Syst. 7, 47–72 (2024).
Problem formulation
Consider the stochastic optimal control problem
\[\begin{array}{rll} \min_{\pi}\ & E\left[\sum_{t=0}^{T} \mathcal{L}_{t}\left(x_{t}, u_{t}\right)\right] & \\ \text{s.t.}\ & x^+ = f(x,u,w;\theta) & \text{-- dynamics} \\ & u_{t}=\pi\left(x_{0}, \cdots, x_{t}\right) & \text{-- policy } \\ & P(x \in X) \geqslant p & \text{-- safe states w/ high probability (w.h.p.)} \\ & P(u \in U) \geqslant g & \text{-- safe inputs w.h.p.} \end{array}\]where \(w \in W\) is random variable and \(\theta \in \Theta\) is parameter
Idea: solve and re-solve problem on shorter time horizon (“receding-horizon” control). Let \(z_{\tau \mid t}\) denote the “\(\tau\)-step-ahead” prediction of \(z_t\) and consider the optimal control problem
\[\begin{array}{ll} \min_{u} \ & \ell\left(x_{T \mid t}, u_{T \mid t}\right) + \sum_{\tau=0}^{T-1} \mathcal{L}_{t+\tau}\left(x_{\tau \mid t}, u_{\tau \mid t}\right) \\ \text { s.t. } \ & x_{\tau+1 \mid t}=\widetilde{f}\left(x_{\tau \mid t}, u_{\tau \mid t}\right) \\ & x \in X, u \in U, x_{0 \mid t}=x_{t}, x_{T \mid t} \in X_{f} \end{array}\]where \(x_{T \mid t} \in X_{f}\) is called the terminal state constraint.
There are two key properties sought in MPC theory: 1. feasibility: if \(x_{t}\) is feasible then \(x_{t+1}\) generated by \(u_{0 \mid t}\) is feasible 2. stability: optimal value \(v^{*}\) is a Lyapunov function for the closed-loop dynamics
Learning-based MPC
We highlight two key ways MPC can interact with learning.
Learning dynamics
Aswani et al. 2013 considered the problem of learning the dynamics while implementing MPC.
Assume \(x^{+}=f(x, u)=A x+B u+g(x, u)\) where \(x \in X, u \in U, g(x, u) \in W\), \(X\), \(U\), and \(W\) are polytopes (i.e. intersections of half-spaces).
Then use the robustness inherent in MPC to guarantee feasibility even with incorrect model \(\left(x^{+}=A x+B u\right)\).
Finally, learn the model \(\tilde{g}\) using basis functions, neural networks, or nonparametric estimators that guarantee boundedness.
Example (Nadaraya-Watson): given parameters \(\lambda, \eta>0\) let \(z_{i}=\left[\begin{array}{l}x_{i} \\ u_{i}\end{array}\right]\), \(y_{i}=x_{i+1}-\left(A x_{i}-B u_{i}\right)\), \(\xi_{i}(z)=\left\|z-z_{i}\right\|^{2} \frac{1}{\eta}\) and \(\kappa: \mathbb{R} \rightarrow \mathbb{R}_{\geq 0}\) be smooth, even, with finite support. Then the following estimate guarantees the dynamics are learned while executing MPC.
\[\begin{aligned} \tilde{g}(x, u) & = \frac{1}{\lambda+\sum_{i} \kappa\left(\xi_{i}(x, u)\right)} \sum_{i} y_{i} \kappa\left(\xi_{i}(x, u)\right) \\ & =\underset{d}{\arg \min } \sum_{i} \kappa\left(\xi_{i}(x, u)\right)\left\|y_{i}-d\right\|^{2}+\lambda\|d\|^{2} \end{aligned}\]Guaranteeing safety
Hsu et al. 2024 considered how to augment a learned policy to ensure safety constraints are satisfied.
Idea: given learned policy \(\pi_{0}\), pass through a “safety filter” MPC problem
\[ \begin{aligned} \min_{\pi}\ & \left\|\pi-\pi_{0}\right\| \\ \text{s.t.}\ & x^{+}=f(x, u, w ; \theta) \\ & u_{t}=\pi\left(x_{0}, \cdots, x_{t}\right) \\ & P(x \in x) \geq p \\ & P(u \in U) \geq q \end{aligned} \]
The controller is safe if the problem is feasible, and the resulting controller is close to the learned controller.