We've discussed three iterative methods:
For Bisection, we saw that $p_n = p + O(1/2^n)$. For, fixed-point iteration we have $p_n = p + O(K^n)$ where $K$ is the upper bound on the absolute value of the derivative. And for Newton's method $p_n = p + O(K^n)$ for all $0 < K < 1$.
We introduce some definitions to classify these methods.
Suppose $\{p_n\}_{n=0}^\infty$ is a sequence that converges to $p$ with $p_n \neq p$ for all $n$. If positive constants $\lambda$ and $\alpha$ exist such that
$$ \lim_{n \to \infty} \frac{|p_{n+1}-p|}{|p_{n}-p|^\alpha} = \lambda,$$then $\{p_n\}_{n=0}^\infty$ converges to $p$ to order $\alpha$, with asymptotic order constant $\lambda$.
An iterative method defined by $p_n = g(p_{n-1})$ is said to be of order $\alpha$ if the sequence $\{p_n\}_{n=0}^\infty$ converges to the solution of $p=g(p)$ to order $\alpha$.
Two cases are of particular interest:
To see why quadratic convegence is faster: Assume $p_n \to p$, linearly, with constant $\lambda = .5$. Also assume $q_n \to p$, quadratically, with the same constant. Then
$$ \lim_{n \to \infty} \frac{|p_{n+1} - p|}{|p_{n}-p|} = .5, \quad \lim_{n \to \infty} \frac{|q_{n+1} - p|}{|q_{n}-p|^2} = .5.$$Let's assume that $|p_0 -p|, ~|q_0-q| < 1$. Then
$$ \frac{|p_{n+1} - p|}{|p_{n}-p|} \approx .5, \quad \frac{|q_{n+1} - p|}{|q_{n}-p|^2} \approx .5$$Then
$$|p_{n+1} - p|\approx (.5){|p_{n}-p|}, \quad |q_{n+1} - p|\approx (.5){|q_{n}-p|^2}.$$$$|p_{n+1} - p|\approx (.5){|p_{n}-p|} \approx (.5)^2{|p_{n-1}-p|} \\\approx \cdots \approx (.5)^{n+1}{|p_{0}-p|}$$$$|q_{n+1} - p|\approx (.5){|q_{n}-p|^2} \approx (.5)^3{|q_{n-1}-p|^4} \approx \\\cdots \approx (.5)^{2^{n}-1}{|p_{0}-p|^{2^n}}$$n = 5;format long;
[(.5)^(n+1) (.5)^(2^n-1)]
Let $g \in C[a,b]$ be such that $g(x) \in [a,b]$ for all $x \in [a,b]$. Suppose, in addition, that $g'$ exists on $(a,b)$ and there exists $0 \leq K < 1$ such that
$$ |g'(x)| \leq K, \quad \text{ for all } ~~ x \in [a,b]. $$If $p$ is the unique fixed point and $g'(p) \neq 0$ then for $p_0 \in [a,b]$ the sequence $p_n \to p$ linearly with asymptotic error constant
$$ \lambda = |g'(p)|$$(if $p_n \neq p$ for all $n$).
Based on the previous theorem for convergence of fixed-point iteration, we know that $p_{n} \to p$. It then follows that
$$ \lim_{n \to \infty} \frac{g(p_n)- g(p)}{p_n - p} = g'(p).$$But
$$ \lim_{n \to \infty} \frac{|g(p_n)- g(p)|}{|p_n - p|} = \lim_{n \to \infty} \frac{|p_{n+1} - p|}{|p_n - p|} = |g'(p)|.$$It is immediately clear that something special happens when $g'(p) = 0$ for a fixed point $p$. This next theorem helps describe this behavior.
Let $p = (a+b)/2 \in (a,b)$ be a fixed point of $g \in C^2[a,b]$. Suppose that $g'(p) = 0$ and $|g'(x)| \leq K < 1$ for $x \in [a,b]$. For $p_0 \in [a,b]$, the sequence $p_n = g(p_{n-1})$ converges at least quadradically to $p$ with asymptotic rate constant $\lambda = |g''(p)|/2$.
Let $x \in [a,b]$. By the Mean-Value Theorem, for some $\xi$ between $x$ and $p$
$$ g(x) = g(p) + g'(\xi)(x-p) = p + g'(\xi)(x-p).$$Then using that $|x-p| \leq (b-a)/2$ $$g(x) = p + g'(\xi)(x-p) \leq p + |g'(\xi)||x-p| \leq (b+a)/2 + (b-a)/2 = b\\ g(x) = p + g'(\xi)(x-p) \geq p - |g'(\xi)||x-p| \geq (b+a)/2 - (b-a)/2 = a.$$
So, $g(x) \in [a,b]$ for $x \in [a,b]$. We know that $p_n \to p$, the unique fixed point.
We use Taylor's theorem to state that for some $\xi \in [a,b]$ (between $p$ and $x$ actually)
$$ g(x) = g(p) + g'(p)(x-p) + \frac{g''(\xi)}{2}(x-p)^2 \\= g(p) + \frac{g''(\xi)}{2}(x-p)^2.$$If $x = p_{n}$ we find for some $\xi_n$ between $p$ and $p_n$.
$$\frac{p_{n+1}-p}{(p_n-p)^2} = \frac{g''(\xi_n)}{2}.$$Then $\xi_n \to p$ as $n \to \infty$ (it is closer to $p$ than $p_n$ is) and so
$$\lim_{n \to \infty} \frac{|p_{n+1}-p|}{|p_n-p|^2} = \lim_{n \to \infty} \frac{|g''(\xi_n)|}{2} = \frac{|g''(p)|}{2}.$$This is something satisfied by Newton's method with $g(x) = x - f(x)/f'(x)$. Recall that
$$g'(x) = \frac{f(x)f''(x)}{[f'(x)]^2}.$$A function $f(x)$ as a zero of mutiplicity $m$ (or of order $m$) at $x = p$ if for $x \neq p$ we can write
$$ f(x) = (x-p)^m q(x)$$where $\lim_{x \to p} q(x) \neq 0$.
The following can be proved using Taylor's Theorem:
A function $f \in C^m[a,b]$ has a zero of multiplicity $m$ at $p \in (a,b)$ if and only if
$$ 0 = f(p) = f'(p) = \cdots = f^{(m-1)}(p), \quad \text{but} \quad f^{(m)}(p) \neq 0.$$Recall, the Newton's interation function
$$ g(x) = x - f(x)/f'(x)$$.
Assume we want to solve $f(x)= 0$, but $f'(x) = 0$. We construct a new function $\mu(x)$ out of $f(x)$ that has a simple zero. More generally, let $f(x)$ have a zero of multiplicity $m$ at $x = p$ and consider the function
$$\mu(x) = \frac{f(x)}{f'(x)} = \frac{(x-p)^m q(x)}{m(x-p)^{m-1} q(x) + (x-p)^m q'(x)} \\ = (x-p) \frac{q(x)}{m q(x) + (x-p) q'(x)}$$It follows that $\lim_{x \to p} \mu(x)/(x-p) \neq 0$. And so, to find a zero of $f(x)$ when $f$ has a zero of higher multiplicity we can apply Newton's method to $\mu(x)$:
$$g(x) = x - \frac{\mu(x)}{\mu'(x)} = x - \frac{f(x)f'(x)}{[f'(x)]^2 - f(x)f''(x)}.$$This can work well in some cases.
This is a method to accelerate convergence. Given a sequence $\{p_n\}_{n=1}^\infty$, we want to construct a new sequence $\{\hat p_n\}_{n=1}^\infty$ that converges faster.
Assume
$$ \frac{p_{n+1} - p}{p_n-p} \approx \frac{p_{n+2} - p}{p_{n+1}-p}.$$Solving for $p$ gives
$$p \approx \frac{p_{n+2}p_n - p_{n+1}^2}{p_{n+2} - 2 p_{n+1} + p_n} ~\Rightarrow~ \hat p_n = \frac{p_{n+2}p_n - p_{n+1}^2}{p_{n+2} - 2 p_{n+1} + p_n}$$p = @(n) cos(1/n);
hatp = @(n) (p(n+2)*p(n) - p(n+1)^2)/(p(n+2)-2*p(n+1) + p(n));
n = 200;
fprintf('n = %i \n',n)
fprintf('Original error = %0.5d \n',abs(p(n+2)-1))
fprintf('Aiken error = %0.5d \n',abs(hatp(n)-1))
To see the reason this method is named as it is, define the forward difference
$$ \Delta p_n = p_{n+1} - p_n.$$Define $\Delta^kp_n = \Delta(\Delta^{k-1} p_n)$.
Then, in going back to the defining equation for $\hat p_n$:
$$ \frac{p_{n+1} - \hat p_n}{p_n-\hat p_n} = \frac{p_{n+2} - \hat p_n}{p_{n+1}-\hat p_n}.$$Now, for notational purposes define $q_n = p_n - \hat p_n$ and we write everything in terms of $q_n$:
$$p_{n+1} - \hat p_n = p_{n+1} - p_n + p_n - \hat p_n \\= \Delta p_n + q_n\\ p_{n+2} - \hat p_n = p_{n+2} - p_n + p_n - \hat p_n \\ = p_{n+2} - p_{n+1} + p_{n+1}- p_n + p_n - \hat p_n \\ = \Delta p_{n+1} + \Delta p_n + q_n$$And then we have
$$ \frac{\Delta p_n + q_n}{q_n} = \frac{\Delta p_{n+1} + \Delta p_n + q_n}{\Delta p_n + q_n} \\ \frac{\Delta p_n}{q_n} + 1= \frac{\Delta p_{n+1}}{\Delta p_n + q_n} + 1 \\ \frac{\Delta p_n}{q_n}= \frac{\Delta p_{n+1}}{\Delta p_n + q_n}\\ \Delta p_n (\Delta p_n + q_n) = q_n \Delta p_{n+1} \\ q_n (\Delta p_{n+1} - \Delta p_n ) = (\Delta p_n)^2\\ q_n \Delta^2 p_n = (\Delta p_n)^2\\ \hat p_n = p_n - \frac{(\Delta p_n)^2}{\Delta^2 p_n}$$It can be shown that if $p_n$ is linearly convergent then $\hat p_n$ converges faster in the sense that
$$ \lim_{n\to \infty} \frac{|\hat p_n - p|}{|p_n - p|} = 0. $$