Lecture 1: learning-based control
Goal: overview of course topics and objectives.
Optimal control
This course will focus on the following stochastic optimal control problem (SOCP):
\[ \text{minimize}\ \text{cost}(x,u)\ \text{subject to}\ x_{t+1}\sim\text{dynamics}(t,x_t,u_t) \]
where \(x_t\in X\) is the state, \(u_t\in U\) is the control input, \(x:[0,T)\rightarrow X\), \(u:[0,T)\rightarrow U\), and \(a\sim b\) means “\(a\) is a random variable drawn from distribution \(b\)”.
Example: it is common for cost to be accrued at every timestep (“running cost”) and for the dynamics to be deterministic,
\[ \min_u \sum_t c(x_t,u_t)\ \text{s.t.}\ x_{t+1} = F(x_t,u_t). \]
Q: these are clearly optimization problems; where is the learning?
In the real world, the cost and dynamics aren’t known exactly and must be “learned”, either by expert humans or expert algorithms.
Who is the expert?
Different problem formulations, solution concepts, and methods / algorithms are brought to bear depending on whether we are leveraging human or algorithmic expertise. We consider three dimensions.
| human | algorithm | |
|---|---|---|
| 1. prior knowledge | - Newton’s laws - Maxwell’s Equations - Navier-Stokes |
- historical logs - automated experiments - simulations |
| 2. domain expertise | - operating regime - parameter estimate |
- feature selection - neural network architecture |
| 3. computational approach | leverage the model \(u^+ = u - \gamma Dc(u)\) |
leverage the data \(u^+ = u - \gamma g(u),\ E[g(u)] = Dc(u)\) |
A couple of notes:
- in 1., the human’s knowledge just summarizes the algorithm’s data;
- in 2., the entries in both columns are essentially wisdom gained through trial-and-error with no underlying principles for their design;
- in 3., both computational approaches reduce to some form of (stochastic) gradient descent.
Takeaway: these two approaches are closely related – neither is fundamentally better than the other – and the best solution usually bridges both.