Lecture 2: general-purpose algorithms

Goal: define general-purpose optimization / learning algorithms for control.

References

Review: Recht, B. A Tour of Reinforcement Learning: The View from Continuous Control. Annu. Rev. Control Robot. Auton. Syst. 2, 253–279 (2019).
Original reference: Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992).
Modern theory: Nesterov, Y. & Spokoiny, V. Random Gradient-Free Minimization of Convex Functions. Found Comput Math 17, 527–566 (2017).
Comparison of sampling and descent algorithms: Ma, Y.-A., Chen, Y., Jin, C., Flammarion, N. & Jordan, M. I. Sampling can be faster than optimization. Proc. Natl. Acad. Sci. U. S. A. 116, 20881–20885 (2019).

Problem formulation

Consider the unconstrained nonlinear program (NLP)

\[ \min c(u) \ \text{s.t.}\ u\in\mathbb{R}^m \]

where $c : \mathbb{R}^m\rightarrow\mathbb{R} : u \mapsto c(u)$ is the cost and $u$ has the decision variable(s). This problem is termed a nonlinear program to contrast with a linear program where the cost is $c^\top u$ for some vector $c\in\mathbb{R}^m$.

We want $u^*$ such that $c(u^*) < c(u)$ for all $u\neq u^*$; in this case $u^*$ is a global minimizer. If we restrict to $|u - u^*| < \varepsilon$ then $u^*$ is a local minimizer.

Iterative algorithms

Idea: iteratively choose $u^+$ such that $c(u^+) < c(u)$ and hope that $u \mapsto u^+ \mapsto u^{++} \mapsto \cdots \mapsto u^*$.

Usually need to assume some regularity property on $c$, e.g. continuity or differentiability.

Question: how to choose $u^+$?

There are two main strategies: randomize or descend (referred to as sampling and optimization in Ma et al. 2019).

	randomize	descend
algorithm	$u^+ \sim \rho(u)$	$u^+ = u - \gamma Dc(u)^\top$
notes	$u^+$ is a random variable with probability distribution $\rho$	need to have access to gradient $Dc$, exact or approximate
examples	* genetic / swarm algorithms * simulated annealing * Markov Chain Monte Carlo	* gradient descent * Newton-Raphson * zero-th order methods
properties	+ approximates global $u^$ + easy to implement - sample-inefficient needs continuous $c$	- approximates local $u^$ + easy[-ish] implementation + sample-efficient needs differentiable $c$

A general-purpose algorithm

Instead of

\[ \min c(u) \ \text{s.t.}\ u\in\mathbb{R}^m \]

consider

\[ \min E[c(u)] \ \text{s.t.}\ u\sim\rho \]

where $\rho$ is a probability distribution.

Q: how do minimizers of these two problems relate?

If “$\delta$” distributions are allowed, then minimizers coincide (and, since $\int\rho = 1$, the second problem’s cost can’t be lower than the first).

But there are (uncountably) infinitely many “$\delta$” distributions, so we need a practical way to search over probability distributions.

Log-likelihood trick

In practice, we go one step further: instead of

\[ \min E[c(u)] \ \text{s.t.}\ u\sim\rho \]

consider

\[ \min E[c(u)] \ \text{s.t.}\ u\sim\rho_\theta \]

where $\rho_\theta$ is a probability distribution parameterized by $\theta\in\Theta$.

The richness of the parameterization of $\rho_\theta$ trades off tractability with suboptimality.

Idea: to descend this modified cost, we employ the “log-likelihood trick”.

\[\begin{align}[rll] D_\theta E[c(u)] & = D_\theta\int c(u)\rho_\theta(u) du & \text{def. of expectation} \\ & = \int c(u) D_\theta\rho_\theta(u) du & \text{assuming $D_\theta$ $\&$ $\int$ commute} \\ & = \int c(u) D_\theta\rho_\theta(u) \frac{\rho_\theta(u)}{\rho_\theta(u)} du & \text{assuming $\rho_\theta(u) \neq 0$} \\ & = \int\left[c(u)D_\theta\log\rho_\theta(u)\right]\rho_\theta(u) du & \text{$D_x\log f(x) = D_x f(x) / f(x)$} \\ & = E\left[ c(u) D_\theta\log\rho_\theta(u) \right] & \text{def. of expectation} \end{align}\]

Takeaway: if we sample $u\sim\rho_\theta(u)$, compute $D_\theta\log\rho_\theta(u)$, and evaluate $c(u)$, then by averaging samples we can approximate derivative of $E[c(u)]$ without the derivative $Dc$!