Skip to main content

$$ \newcommand{\R}{\mathbb{R}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} $$

Lecture 6: linear quadratic regulation

Goal: derive optimal controller for linear DE with quadratic cost: the linear-quadratic regulator (LQR).

References

• Section 3.7: Stengel, R. F. Optimal Control and Estimation. (Dover, New York, 1994).
• Chapter 6: Lewis, F. L., Vrabie, D. & Syrmos, V. L. Optimal Control. (John Wiley & Sons, 2012).

Problem formulation

We consider the discrete-time linear-quadratic regulator:

\[ \min c_\tau(x,u)\ \text{s.t.}\ x_{s+1} = A_s x_s + B_s u_s, \]

\[ c_\tau(x,u) = \frac{1}{2} x_t^\top P_t x_t + \frac{1}{2} \sum_{s = \tau}^{t - 1} x_s^\top Q_s x_s + u_s^\top R_s u_s \]

where \(\ell(t,x_t) = \frac{1}{2} x_t^\top P_t x_t\) is the final cost and \(\mathcal{L}(s,x_s,u_s) = x_s^\top Q_s x_s + u_s^\top R_s u_s\) is the running cost.

Assume \(P_t^\top = P_t > 0\), \(Q_s^\top = Q_s > 0\), \(R_s^\top = R_s > 0\).

Bellman equation

We’ll use the Bellman equation to determine the optimal control input. Let \(v_\tau^*:\mathbb{R}^d\rightarrow\mathbb{R}\) be the optimal value function that satisfies

\[ v_\tau^*(x_\tau) = \min_{u_\tau} \left[ \mathcal{L}(\tau,x_\tau,u_\tau) + v_{\tau+1}^*(x_{\tau+1}) \right]. \]

Q: use the Bellman equation to determine the optimal control and value at \(\tau = t\).

Since \(c_t = \frac{1}{2} x_t^\top P_t x_t\), \(u_t^* = 0\) (any choice is valid / optimal) and \(v_t^*(x_t) = \frac{1}{2} x_t^\top P_t x_t\).

Q: use the Bellman equation to determine the optimal control and value at \(\tau = t-1\).

The Bellman equation reads \(v_{t-1}(x_{t-1}) = \min_{u_{t-1}}\frac{1}{2} \left[ x_{t-1}^\top Q_{t-1} x_{t-1} + u_{t-1}^\top R_{t-1} u_{t-1} + x_t^\top P_t x_t \right]\).

Substituting \(x_t = A_{t-1} x_{t-1} + B_{t-1} u_{t-1}\) and differentiating with respect to \(u_{t-1}\) yields \(D_{u_{t-1}} v_{t-1}^* = u_{t-1}^\top R_{t-1} + (A_{t-1} x_{t-1} + B_{t-1} u_{t-1})^\top P_t B_{t-1}\).

Setting this equation equal to zero (necessary for optimality) and solving for \(u_{t-1}\) yields \(u_{t-1}^* = - (B_{t-1}^\top P_t B_{t-1} + R_{t-1})^{-1}B_{t-1}^\top P_t A_{t-1} x_{t-1}\).

Differentiating \(v_{t-1}^*\) again yields \(D_{u_{t-1}}^2 v_t^* = (B_{t-1}^\top P_t B_{t-1} + R_{t-1})\). Since this is the sum of two symmetric positive-semidefinite matrices, it is symmetric and PSD. Since \(R_{t-1}\) is positive-definite, the sum is positive definite. This is important to ensure it is invertible in the expression for \(u_{t-1}^*\) and because it means the second-order sufficient condition certifying \(u_{t-1}^*\) is a local minimum is satisfied. (Since the cost is convex, it is \(u_{t-1}^*\) is the global minimum).

Solution

For simplicity, define \(K_{t-1} = (B_{t-1}^\top P_t B_{t-1} + R_{t-1})^{-1} B_{t-1}^\top P_t A_{t-1}\) so that \(u_{t-1}^* = -K_{t-1} x_{t-1}\), i.e. the optimal control is a linear function of state!

The optimal value is \(v_{t-1}^* = \frac{1}{2} x_{t-1}^\top P_{t-1} x_{t-1}\) where

\[ P_{t-1} = (A_{t-1} - B_{t-1} K_{t-1})^\top P_t (A_{t-1} - B_{t-1} K_{t-1}) + K_{t-1}^\top R_{t-1} K_{t-1} + Q_{t-1}. \]

At time \(\tau = t - 2\) we’ll perform analogous calculations yielding the following backward-time recursion:

\[\begin{align*} K_s & = (B_s^\top P_{s+1} B_s + R_s)^{-1} B_s^\top P_{s+1} A_s, \\ P_s & = (A_s - B_s K_s)^\top P_{s+1} (A_s - B_s K_s) + K_s^\top R_s K_s + Q_s, \\ u_s^* & = - K_s x_s, \\ v_s^* & = \frac{1}{2} x_s^\top P_s x_s. \end{align*}\]

Critically, we have solved for the optimal control input for any state. This is a very special outcome that occurs only for very special systems.

As a final note on terminology: the equation for \(P_s\) is termed a Riccati difference equation.