Lecture 3: descent algorithms

Goal: deterministic gradient descent algorithms for unconstrained optimization.

References

Section 3.6: Stengel, R. F. Optimal Control and Estimation. (Dover, New York, 1994).
Chapter 1: Bertsekas, D. P. Nonlinear Programming. (Athena Scientific, 1999).

Problem formulation

Consider the nonlinear program (NLP)

\[ \min_{u\in\mathbb{R}^m} c(u) \]

and assume $c$ is smooth, i.e. continuously differentiable as many times as we need.

What the gradient tells us

Suppose we start with a “guess” $u\in\mathbb{R}^m$ for the minimizer and observe that $Dc(u) \neq 0$.

Q: how should we modify our “guess” to get closer to a (local) minimizer?

If we regard $Dc(u)$ as a function $Dc(u):\mathbb{R}^m\rightarrow\mathbb{R}:v\mapsto\langle Dc(u),v\rangle$, we see that the gradient tells us, for any given direction $v\in\mathbb{R}^m$, how quickly $c$ changes in the $v$ direction:

\[ c(u + \gamma v) \approx c(u) + \gamma \langle Dc(u),v\rangle. \]

Importantly, any $v\mathbb{R}^m$ such that $\langle Dc(u),v\rangle < 0$ is a valid descent direction. As a particular choice that always works, $v = -Dc(u) \implies \langle Dc(u), -Dc(u)\rangle = -|Dc(u)|^2 < 0$. But there’s an entire half-space of valid descent directions!

Code

from imports import *

fig, ax = plt.subplots(figsize=(5, 5))

xlim, ylim = [-3, 3], [-3, 3]
dc = np.array([1.0, 2.0])  # Dc(u) direction: up and to the right

# kernel line: dc[0]*u1 + dc[1]*u2 = 0  =>  u2 = -(dc[0]/dc[1]) * u1
slope = -dc[0] / dc[1]
u1_fill = np.array([xlim[0] - 1, xlim[1] + 1])  # extend past plot edges
u2_line = slope * u1_fill

# fill half-planes with both color and hatch pattern for color-blind accessibility
ax.fill_between(u1_fill, u2_line, ylim[1] + 1, color='moccasin',   alpha=0.65, hatch='///', zorder=0)  # above: descent half-space
ax.fill_between(u1_fill, ylim[0] - 1, u2_line, color='thistle',    alpha=0.65, hatch='\\', zorder=0)  # below: non-descent half-space

# kernel line
u1_line = np.linspace(*xlim, 200)
ax.plot(u1_line, slope * u1_line, 'k-', linewidth=1.8, zorder=2)

# Dc(u) vector arrow
ax.annotate('', xy=dc, xytext=(0, 0),
            arrowprops=dict(arrowstyle='->', color='black', lw=2.0,
                            mutation_scale=18), zorder=3)

# labels
ax.text(dc[0] + 0.15, dc[1] + 0.1, r'$Dc(u)$', fontsize=14, va='bottom')
# place ker label along the kernel direction, offset slightly
ker_pt = np.array([-1.4, -slope * 1.4])
ax.text(ker_pt[0] - 1.0, ker_pt[1] + 0.2, r'$\ker Dc(u)$', fontsize=14,
        rotation=np.degrees(np.arctan(slope)), rotation_mode='anchor')

# axes through origin
for spine in ['left', 'bottom']:
    ax.spines[spine].set_position('zero')
for spine in ['right', 'top']:
    ax.spines[spine].set_visible(False)
ax.set_xticks([])
ax.set_yticks([])
ax.set_xlim(xlim)
ax.set_ylim(ylim)
ax.set_xlabel(r'$u_1$', fontsize=14, labelpad=2)
ax.set_ylabel(r'$u_2$', fontsize=14, labelpad=2, rotation=0)
ax.xaxis.set_label_coords(1.02, 0.5)
ax.yaxis.set_label_coords(0.5, 1.02)
ax.set_aspect('equal')

plt.tight_layout()
#plt.show()
show_with_alt(fig, r'two-dimensional decision vector space with $Dc(u)$ and $\ker Dc(u)$ overlaid showing a half-space of descent directions. The descent half-space (above the kernel line) is filled with a forward-diagonal hatch pattern; the non-descent half-space (below) uses a backward-diagonal hatch pattern.')

two-dimensional decision vector space with $Dc(u)$ and $\ker Dc(u)$ overlaid showing a half-space of descent directions. The descent half-space (above the kernel line) is filled with a forward-diagonal hatch pattern; the non-descent half-space (below) uses a backward-diagonal hatch pattern.

Gradient descent algorithm

Idea: to select a descent direction, we can formulate another optimization problem.

Q: solve $\min_{v\in\mathbb{R}^m} \langle Dc(u), v \rangle$ s.t. $|v|_2 \leq |Dc(u)|_2$.

Recalling that $\langle x, y \rangle = |x|_2|y|_2\cos\theta$ where $\theta$ is the angle between $x$ and $y$, by inspection the solution is $v^* = -Dc(u)^\top$.

This solution leads naturally to the gradient descent algorithm $u^+ = u - \gamma Dc(u)^\top$ with step size $\gamma > 0$.

Convergence analysis

Note: the algorithm defines a difference equation (DE) which is a dynamical system in and of itself. Local minimizers of the original NLP are equilibria of the DE. We can use stability analysis for DE to find the largest valid choice for $\gamma > 0$ that guarantees convergence of the gradient descent algorithm for NLP.

Q: linearize the DE about a local minimizer for the NLP.

Differentiating the DE with respect to $u$ yields $D_u\left[ u - \gamma Dc(u)^\top \right] = I - \gamma D^2 c(u) = A(\gamma)$. Evaluating at $u^*$ yields $(u - u^*)^+ \approx A(\gamma)(u - u^*)$.

For stability of the DE we need all eigenvalues $\lambda$ of $A(\gamma)$ to have magnitude smaller than 1. The Spectral Mapping Theorem tells us that $\sigma\in D^2c(u)\implies \lambda = 1 - \gamma\sigma \in \text{spec} I - \gamma D^2 c(u) = \text{spec} A(\gamma)$, so we need $-1 < 1 - \gamma\sigma < +1 \iff 0 < \gamma < \frac{2}{\sigma}$ for stability of the DE / convergence of gradient descent for the NLP.