Reference: Casella and Berger Chapter 2 and 3.
For a function \(g(x)\), the expectation of \(g(X)\) is \[\mathbb{E}(g(X)) = \int g(x) dF(x) = \begin{cases} \int_{-\infty}^\infty g(x) p(x)dx, \quad &\mbox{if $X$ is continuous}\\ \sum_{x} g(x) p(x), \quad &\mbox{if $X$ is discrete} \end{cases}.\] In the simplest case \(g(x)=x\), \[\mathbb{E}(X) = \int x dF(x) = \begin{cases} \int_{-\infty}^\infty x p(x)dx, \quad &\mbox{if $X$ is continuous}\\ \sum_{x} x p(x), \quad &\mbox{if $X$ is discrete} \end{cases}.\] is known as the the expected value (or mean) of a random variable \(X\). Let \(\mu = \mathbb{E}(X)\), the variance of \(X\) is \({\sf Var}(X) = \mathbb{E}((X-\mu)^2)\). The mean is a common measure of the center of a distribution and the variance is a common measure of the spread of a distribution.
The \(m\)-th moment of a random variable \(X\) is \[\mathbb{E}(X^m).\] Let \(\mu= \mathbb{E}(X)\) be the mean/first moment of \(X\), the \(m\)-th centered moment of \(X\) is \[\mathbb{E}((X-\mu)^m).\] Thus, the variance is the second centered moment.
Example.
\(X\sim {\sf Binomial}(n,p)\). Then \(\mathbb{E}(X)= np\) and \({\sf Var}(X) = np(1-p)\).
\(X\sim {\sf Geometric}(p)\). Then \(\mathbb{E}(X)= 1/p\) and \({\sf Var}(X) = (1-p)^2/p\).
\(X\sim {\sf Poisson}(\lambda)\). Then \(\mathbb{E}(X)= \lambda\) and \({\sf Var}(X) = \lambda\).
\(X\sim {\sf Normal}(\mu,\sigma^2)\). Then \(\mathbb{E}(X)= \mu\) and \({\sf Var}(X) = \sigma^2\).
\(X\sim {\sf Exponential}(\lambda)\). Then \(\mathbb{E}(X)= 1/\lambda\) and \({\sf Var}(X) = 1/\lambda^2\).
\(X\sim {\sf Gamma}(\alpha,\lambda)\). Then \(\mathbb{E}(X)= \alpha/\lambda\) and \({\sf Var}(X) = \alpha/\lambda^2\).
\(X\sim {\sf Beta}(\alpha,\beta)\). Then \(\mathbb{E}(X)= \frac{\alpha}{\alpha+\beta}\) and \({\sf Var}(X) = \frac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\).
\(X\sim {\sf Uniform}(a,b)\). Then \(\mathbb{E}(X)= (a+b)/2\) and \({\sf Var}(X) = (b-a)^2/12\).
Expectations are linear operations: \[\mathbb{E}\left(\sum_{j=1}^k c_j g_j(X)\right) = \sum_{j=1}^k c_j \cdot\mathbb{E}(g_j(X_i)).\] Note that the above equality holds even if \(X_i\)’s are dependent.
If a set of random variables \(X_1,\dots,X_n\) are mutually independent, then \[\mathbb{E}\left(X_1\cdot X_2\cdots X_n\right) = \mathbb{E}(X_1)\cdot \mathbb{E}(X_2)\cdots \mathbb{E}(X_n).\] In fact, you can also prove that \[\mathbb{E}\left(g_1(X_1)\cdot g_2(X_2)\cdots g_n(X_n)\right) = \mathbb{E}(g_1(X_1))\cdot \mathbb{E}(g_2(X_2))\cdots \mathbb{E}(g_3(X_n)).\]
For two random variables \(X\) and \(Y\) with their mean being \(\mu_X\) and \(\mu_Y\) and variance being \(\sigma^2_X\) and \(\sigma^2_Y\). The covariance \[{\sf Cov}(X,Y) = \mathbb{E}((X-\mu_x)(Y-\mu_y)) = \mathbb{E}(XY) - \mu_x\mu_y\] and the (Pearson’s) correlation \[\rho(X,Y) = \frac{{\sf Cov}(X,Y)}{\sigma_x\sigma_y}.\] For any two random variables \(X\) and \(Y\), the variance of their sum or difference satisfies \[{\sf Var}(X\pm Y) = {\sf Var}(X) + {\sf Var}(Y) \pm 2{\sf Cov}(X,Y).\]
The independence implies the covariance (and correlation) is \(0\), i.e., \[X\perp Y \Rightarrow {\sf Cov}(X,Y) = 0.\] As a result, if \(X\perp Y\), \[{\sf Var}(X+Y) = {\sf Var}(X)+{\sf Var}(Y).\] A more general result is that for independent random variables \(X_1,\cdots,X_n\), we have \[{\sf Var}\left(\sum_{i=1}^n a_iX_i\right) = \sum_{i=1}^n a_i^2 \cdot {\sf Var}(X_i).\]
Example (Binomial). Here we illustrate how the above properties can be useful in computing the variance of some distributions. Consider \(X\sim {\sf Binomial}(n,p)\). By the definition of a Binomial distribution, we can rewrite \(X= Y_1+Y_2+\cdots+Y_n\), where each \(Y_i\) is an independent Bernoulli random variable with parameter \(p\). Thus, \[{\sf Var}(X) ={\sf Var}(Y_1+Y_2+\cdots+Y_n) = \sum_{i=1}^n {\sf Var}(Y_i) = n p(1-p).\]
The moment generating function (MGF) provides an alternative and often more tractable characterization of a probability distribution. The MGF of a RV \(X\) is \[M_X(t) = \mathbb{E}(e^{tX}).\] Note that \(M_X\) may not exist. When \(M_X\) exists in a neighborhood of \(0\), using the fact that \[e^{tX} = 1+tX+\frac{(tX)^2}{2!}+\frac{(tX)^3}{3!}+\cdots,\] we have \[M_X(t) = 1 + t\mu_1 + \frac{t^2\mu_2}{2!}+ \frac{t^3\mu_3}{3!}+\cdots,\] where \(\mu_j = \mathbb{E}(X^j)\) is the \(j\)-th moment of \(X\). Therefore, \[\mathbb{E}(X^j) = M^{(j)}(0) = \left. \frac{d^j M_X(t)}{dt^j}\right|_{t=0}\] This relationship illustrates how the MGF ‘generates’ the moments of \(X\).
For two random variables \(X,Y\), if their MGFs are the same, then the two random variables have the same CDF. Thus, MGFs can be used as a tool to determine if two random variables have the identical CDF. Note that the MGF is related to the Laplace transform (actually, they are the same) and this may give you more intuition why it is so powerful.
The MGF has some interesting properties:
Location-scale. \(M_{aX+b}(t) = \mathbb{E}(e^{(aX+b)t}) = e^{bt} \mathbb{E}(e^{atX}) = e^{bt}M_{X}(at)\).
Multiplicity. \(M_{X+Y}(t) = \mathbb{E}(e^{(X+Y)t}) = \mathbb{E}(e^{Xt} e^{Yt})\). Thus, \[X\perp Y \Rightarrow M_{X+Y}(t) = \mathbb{E}(e^{Xt} e^{Yt}) = \mathbb{E}(e^{Xt})\mathbb{E}( e^{Yt}) = M_X(t)M_Y(t).\]
Example (Bernoulli and Binomial). Let \(X\sim {\sf Ber}(p)\). Its MGF is \(M_X(t) = \mathbb{E}(e^{tX}) = pe^t + (1-p)\). Let \(Y\sim {\sf Bin}(n,p)\). This can be expressed as \(Y=X_1+\cdots+X_n\), where each \(X_i\) is an independent Bernoulli random variable with parameter \(p\). Its MGF is \[M_Y(t) = \prod_{i=1}^n M_{Z_i}(t) = ( pe^t + (1-p))^n.\]
Example (Poisson). Let \(X\sim {\sf Poisson}(\lambda)\). Then its MGF is \[\begin{align*} M_X(t) = \mathbb{E}(e^{tX}) = \sum_{x=0} e^{tx} \frac{\lambda^x e^{-\lambda}}{x!} = e^{-\lambda} \underbrace{\sum_x \frac{[\lambda e^t]^x }{x!}}_{= e^{\lambda e^t}} = e^{\lambda(e^t-1)}. \end{align*}\]
Example (Exponential). Let \(X\sim {\sf Exp}(\lambda)\). Then its MGF is \[M_X(t) = \mathbb{E}(e^{tX}) = \int e^{tx} \lambda e^{-\lambda x}dx = \frac{\lambda}{\lambda-t}\] for \(t<\lambda\).
Example (Normal). Let \(X\sim N(\mu,\sigma^2)\). Then you can show that (exercise) \[M_X(t) = e^{\mu t + \frac{1}{2}\sigma^2t^2}.\] One can leverage the uniqueness theorem of the MGF to demonstrate that any linear combination of independent normal random variables is itself normally distributed.
Remark (characteristic function). A more general function than MGF is the characteristic function. Let \(i\) denote the imaginary unit. The characteristic function of a RV \(X\) is \[\phi_X(t) = \mathbb{E}(e^{itX}).\] When \(X\) is absolutely continuous, the characteristic function is the Fourier transform of the PDF. The characteristic function always exists and when two RVs have the same characteristic function, the two RVs have identical distribution.
The MGF can be defined for a random vector. Consider \(X = (X_1,\cdots, X_d)\in \mathbb{R}^d\) be a random vector. Its MGF is then defined as a function of \(d\) arguments: \[M_X(t) = \mathbb{E}(e^{t^TX}),\] where \(t= (t_1,\cdots, t_d) \in \mathbb{R}^d\).
Example. Let \(X\) be a multivariate normal \(MVN(\mu,\Sigma)\), where \(\mu\in\mathbb{R}^d\) is the mean vector and \(\Sigma\in \mathbb{R}^{d\times d}\) is the covariance matrix. Namely, each component \(X_i\sim N(\mu_i, \Sigma_{ii})\) and the covariance \({\sf Cov}(X_i,X_j) = \Sigma_{ij}\). Then its MGF will be \[M_X(t) = e^{t^T\mu+\frac{1}{2}t^T\Sigma t}.\] Using this, you can show that a linear tranformation \[Z = b+AX \sim MVN(b+A\mu, A\Sigma A^T).\]
Example (Normal plus Normal). Here we illustrate that the MGF provides a straightforward mechanism to verify that the sum of two jointly normal random variables remains normal. Let \(X\) and \(Y\) be two random variables such that their joint distribution is a multivariate normal (MVN) with mean vector \((\mu_1,\mu_2)^T\) and covariance matrix \(\Sigma\). Consider \(Z=X+Y\). To see why \(Z\) is still normal, consider its MGF: \[M_Z(t) = \mathbb{E}(e^{tZ}) = \mathbb{E}(e^{tX+tY}) = M_{X,Y}(t,t),\] which is the MGF of the normal vector \((X,Y)\) with the augment \((t,t)\). Thus, \[M_Z(t) = M_{X,Y}(t,t) = e^{t(\mu_1+\mu_2) + \frac{1}{2}t^2(\Sigma_{11}+\Sigma_{22}+2\Sigma_{12})},\] which is the MGF of a normal random variable with mean \(\mu_1+\mu_2\) and variance \(\Sigma_{11}+\Sigma_{22}+2\Sigma_{12} = {\sf Var}(X)+{\sf Var}(Y)+2{\sf Cov}(X,Y)\).
Let \(F_1,\cdots,F_n, \cdots\) be the corresponding CDFs of \(Z_1,\cdots, Z_n,\cdots\). For a random variable \(Z\) with CDF \(F\), we say that \(Z_n\) converges in distribution (a.k.a. converge weakly or converge in law) to \(Z\) if for every \(x\), \[\lim_{n\rightarrow \infty}F_n(x) = F(x).\] In this case, we write \[Z_n\overset{D}{\rightarrow} Z,\quad \mbox{or } Z_n\overset{d}{\rightarrow} Z.\] Namely, the CDFs of the sequence of random variables converge pointwise to the CDF of a fixed random variable.
For a sequence of random variables \(Z_1,\cdots,Z_n,\cdots\), we say \(Z_n\) converges in probability to another random variable \(Z\) if for any \(\epsilon>0\), \[\lim_{n\rightarrow \infty}P(|Z_n-Z|>\epsilon) = 0\] and we will write \[Z_n\overset{P}{\rightarrow} Z\]
Remark (convergence almost surely). For a sequence of random variables \(Z_1,\cdots,Z_n,\cdots\), we say \(Z_n\) converges almost surely to a random variable \(Z\) if \[P(\lim_{n\rightarrow\infty} Z_n=Z) = 1\] or equivalently, \[P(\{\omega: \lim_{n\rightarrow\infty} Z_n(\omega)=Z(\omega)\}) = 1.\] We use the notation \[Z_n\overset{a.s.}{\rightarrow} Z\] to denote convergence almost surely. Note that almost surely convergence implies convergence in probability. Convergence in probability implies convergence in distribution.
Examples.
Let \(\{X_1,X_2,\cdots, \}\) be a sequence of random variables such that \(X_n\sim N\left(0, 1+\frac{1}{n}\right)\). Then \(X_n\) converges in distribution to \(N(0,1)\).
Let \(\{X_1,X_2,\cdots\}\) be a sequence of random variables such that \(X_i \sim N(0, 1/n)\). Then \(X_n\overset{P}{\rightarrow}0\), i.e., it converges in probability to \(0\). Also, the random variable \(\sqrt{n} X_n \overset{D}{\rightarrow} N(0,1)\).
Let \(\{X_1,X_2,\cdots\}\) be a sequence of random variables such that \[P(X_n =0) =1-\frac{1}{n},\quad P(X_n = 1) = \frac{1}{n}.\] Then \(X_n\overset{P}{\rightarrow}0\).
Sometimes, one may be thinking that the convergence in probability/distribution may imply convergence in expectation. But this is not true! Here is an example that it converges in probability to \(0\) but its expectation diverges.
Example (diverging expectation but convergence in probability). Consider a sequence of RVs \(X_1,X_2,\cdots,\) such that \[P(X_n =0) =1-\frac{1}{n},\quad P(X_n = n^2) = \frac{1}{n}.\] Then you can easily verify that \(X_n\overset{P}{\rightarrow}0\). However, if you compute the expectation, \[\mathbb{E}(X_n) = n \rightarrow \infty.\] So the expectation is in fact diverging. We will subsequently demonstrate that convergence in expectation implies convergence in probability, as a consequence of Markov’s inequality.
We write \(X_1,\cdots,X_n\sim F\) when \(X_1,\cdots,X_n\) are IID (independently, identically distributed) from a CDF \(F\). In this case, \(X_1,\cdots,X_n\) is called a random sample.
Theorem 1 (Markov’s inequality). Let \(X\) be a non-negative RV. Then for any \(\epsilon>0\), \[P(X\geq \epsilon)\leq \frac{\mathbb{E}(X)}{\epsilon}.\]
A feature of the Markov inequality is that it implies that converges in expectation \(\Rightarrow\) convergence in probability. Also, the Markov’s inequality implies the following useful result, known as the Chebyshev’s inequality.
Theorem 2 (Chebyshev’s inequality). Let \(X\) be a RV with finite variance. Then for any \(\epsilon>0\), \[P(|X-\mathbb{E}(X)|\geq \epsilon)\leq \frac{{\sf Var}(X)}{\epsilon^2}.\]
The proof of the Chebyshev’s inequality is a direct application of the Markov’s inequality. The Chebyshev’s inequality shows that for a sequence of random variables with equal mean but a vanishing variance, this sequence converges in probability to the mean. When applying to the sample mean, it becomes the famous (weak) law of large numbers.
Theorem 3 (Weak Law of Large Numbers). Let \(X_1,\cdots, X_n\sim F\) and \(\mu = \mathbb{E}(X_1)\). If \(\mathbb{E}|X_1|<\infty\) and \({\sf Var}(X_1) = \sigma^2<\infty\), the sample average \[\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i\] converges in probability to \(\mu\). i.e., \[\bar{X}_n\overset{P}{\rightarrow} \mu.\]
Using the property of sample mean, one can easily show that \[{\sf Var}(\bar{X}_n) = \frac{\sigma^2}{n}.\] Thus, by the Chebyshev’s inequality \[P(|\bar X_n - \mu|>t)\leq \frac{\sigma^2}{nt^2}\rightarrow 0,\] which completes the proof.
The above theorem is also known as Weak Law of Large Numbers. In fact, we do not need to assume the existence of variance–this condition can be relaxed (but the proof will become much more complicated). Note that there is something called the strong law of large number, which states the convergence in terms of ‘almost surely convergence’.
Theorem 4 (Central Limit Theorem). Let \(X_1,\cdots, X_n\) be IID random variables with \(\mu = \mathbb{E}(X_1)\) and \(\sigma^2 = {\sf Var}(X_1)<\infty\). Let \(\bar{X}_n\) be the sample average. Then \[\sqrt{n}\left(\frac{\bar{X}_n-\mu}{\sigma}\right) \overset{D}{\rightarrow} N(0,1).\] Note that \(N(0,1)\) is also called standard normal random variable.
Let \(Z = \sqrt{n}(\bar {X}_n - \mu)\). Proving the problem is equivalent to showing that \(Z\rightarrow N(0,\sigma^2)\).
Note that we can rewrite \(Z\) as \[Z = \sqrt{n}(\bar {X}_n - \mu) = \frac{1}{\sqrt{n}}\sum_{i=1}^n(X_i-\mu) = \frac{1}{\sqrt{n}}\sum_{i=1}^nY_i,\] where each \(Y_i\) has mean \(0\) and variance \(\sigma^2\) and are IID. Thus, the MGF of \(Z\) is \[\begin{equation} M_Z(t) = \mathbb{E}(e^{tZ}) = \mathbb{E}\left(e^{\frac{t}{\sqrt{n}} \sum_{i=1}^nY_i}\right)= \mathbb{E}\left(e^{\frac{t}{\sqrt{n}} Y_1}\right)^n = M_{Y_1}(t/\sqrt{n})^n. \label{eq::MGF1} \end{equation}\] Note that we use the fact that \(Y_1,\cdots, Y_n\) are IID in the last equality.
Now we analyze \(M_{Y_1}(t/\sqrt{n})\): \[M_{Y_1}(t/\sqrt{n}) = \mathbb{E}(e^{tY/\sqrt{n}}) = 1 +\frac{t}{\sqrt{n}}\underbrace{\mathbb{E}(Y)}_{=0}+\frac{t^2}{2{n}}\underbrace{\mathbb{E}(Y^2)}_{=\sigma^2} + \mbox{ smaller order term}.\] The remaining terms are smaller order because we are under \(n\rightarrow\infty\). Denoting the smaller order term as \(o(1),\) using the above expansion, we can see that \[M_{Y_1}(t/\sqrt{n})^n = \left(1+\frac{t^2 \sigma^2}{2n + o(1)}\right)^n\rightarrow e^{\frac{1}{2}t^2\sigma^2}.\] Thus, Equation \(\eqref{eq::MGF1}\) will be approaching \[M_Z(t) = M_{Y_1}(t/\sqrt{n})^n=\left(1+\frac{t^2 \sigma^2}{2n + o(1)}\right)^n\rightarrow e^{\frac{1}{2}t^2\sigma^2},\] which is the MGF of a normal random variable with mean \(0\) and variance \(\sigma^2\). So we have proved the desired result.
Note that there are other versions of central limit theorem that allows dependent RVs or infinite variance using the idea of ‘triangular array’ (also known as the Lindeberg-Feller Theorem). However, the details are beyond the scope of this course so we will not pursue it here.
Continuous mapping theorem: Let \(g\) be a continuous function.
If a sequence of random variables \(X_n\overset{D}{\rightarrow}X\), then \(g(X_n)\overset{D}{\rightarrow}g(X)\).
If a sequence of random variables \(X_n\overset{p}{\rightarrow}X\), then \(g(X_n)\overset{p}{\rightarrow}g(X)\).
Slutsky’s theorem: Let \(\{X_n:n=1,2,\cdots\}\) and \(\{Y_n:n=1,2,\cdots\}\) be two sequences of RVs such that \(X_n\overset{D}{\rightarrow} X\) and \(Y_n\overset{p}{\rightarrow}c\), where \(X\) is a RV \(c\) is a constant. Then \[\begin{align*} X_n+Y_n&\overset{D}{\rightarrow} X+c\\ X_nY_n&\overset{D}{\rightarrow} c X\\ X_n/Y_n&\overset{D}{\rightarrow} X/c \quad \mbox{(if $c\neq 0$)}. \end{align*}\]
We will invoke these two theorems frequently when discussing the asymptotic properties of the maximum likelihood estimator.
These modes of convergence are foundational in statistical inference. Convergence in probability forms the basis of statistical consistency. An estimator is statistically consistent if it converges in probability toward its target population quantity. The convergence in distribution is often used to construct a confidence interval or perform a hypothesis test.
In addition to the above two theorems, we often use the concentration inequality to obtain convergence in probability. Let \(\{X_n:n=1,2,\cdots\}\) be a sequence of RVs. For a given \(\epsilon>0\), the concentration inequality aims at finding the function \(\phi_n(\epsilon)\) such that \[P(|X_n -\mathbb{E}(X_n)|>\epsilon)\leq \phi_n(\epsilon)\] and \(\phi_n(\epsilon)\rightarrow0\). This automatically gives us convergence in probability. Moreover, the convergence rate of \(\phi_n(\epsilon)\) with respect to \(n\) is a central quantity that describes how fast \(X_n\) converges toward its mean.
Example: concentration of a Gaussian mean. The Markov’s inequality implies a useful bound on describing how fast the sample mean of a Gaussian converges to the population mean. For simplicity, we consider a sequence of mean \(0\) Gaussians: \(X_1,\cdots, X_n\sim N(0,\sigma^2)\). Let \(\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i\) be the sample mean. It is known that \({\bar X}_n\sim N(0,\sigma^2/n)\). Then \[\begin{align*} P(\bar{X}_n>\epsilon) & = P(e^{\bar{X}_n}> e^\epsilon)\\ & = P(e^{s\bar{X}_n}> e^{s\epsilon})\\ &\leq \frac{\mathbb{E}(e^{s \bar{X}_n})}{e^{s\epsilon}}\qquad \mbox{by Markov's inequality}\\ & \leq e^{\frac{1}{2n}\sigma^2s^2 - s\epsilon} \qquad \mbox{by the MGF of Gaussian} \end{align*}\] for any positive number \(s\). In the exponent, it is a quadratic function of \(s\) and the maximal occurs at \(s= \frac{n\epsilon}{\sigma^2}\), leading to \[P(\bar{X}_n>\epsilon)\leq e^{-\frac{n\epsilon^2}{2\sigma^2}}.\] The same bound holds for the other direction \(P(\bar{X}_n<-\epsilon)\leq e^{-\frac{n\epsilon^2}{2\sigma^2}}.\) So we conclude \[P(|\bar{X}_n|>\epsilon)\leq 2e^{-\frac{n\epsilon^2}{2\sigma^2}}\] or more generally, \[P(|\bar{X}_n - \mathbb{E}(X_1)|>\epsilon)\leq 2e^{-\frac{n\epsilon^2}{2\sigma^2}}.\] A bound like the above is often referred to as a concentration inequality.
Example (concentration of a maximum). Let \(X_1,\cdots, X_n\) be IID standard normal random variables \(N(0,\sigma^2)\). Define \(Z_n = \max\{|X_1|,\cdots,| X_n|\}\) be the maximal number among them. Intuitively, we know that when \(n\rightarrow\infty\), \(Z_n\) should be diverging since we are taking the maximum of more and more values. But it is possible to find an increasing sequence \(\gamma_n\rightarrow\infty\) such that \(Z_n/\gamma_n\) will not diverge (in probability). How do we find such \(\gamma_n\)? A simple approach is based on the concentration inequality. Using the result from previous example, we know that for a single random variable \(X_i\) (replace the sample mean by the mean of a single RV), we have \[P(|X_i|>\epsilon)\leq 2e^{-\frac{\epsilon^2}{2\sigma^2}}.\] With this, we can bound \[\begin{align*} P(Z_n>\epsilon) &= P(\max\{|X_1|,\cdots, |X_n|\}>\epsilon)\\ &\leq \sum_{i=1}^n P(|X_i|>\epsilon)\qquad\mbox{(maximum is over $\epsilon\Rightarrow$ one of them must hold)}\\ &\leq 2n e^{-\frac{\epsilon^2}{2\sigma^2}}. \end{align*}\] Thus, as long as we can choose a sequence \(\epsilon = \epsilon_n\) such that \[2n e^{-\frac{\epsilon_n^2}{2\sigma^2}}\rightarrow \delta\] for some constant \(0<\delta<1\), we can bound how fast \(Z_n\) diverge. Solving this gives us a single rule \(\epsilon_n =\sigma\sqrt{2\log (2n) - 2\log (\delta)}\). This leads to the choice of \(\gamma_n = \sigma\sqrt{2\log n}\), which gives a characterization on how fast \(Z_n\) diverges.
Let \(X_1,\cdots, X_n\sim F\) be a random sample such that \(\sigma^2 = {\sf Var}(X_1)\). Using the Chebyshev’s inequality, we know that the sample average \(\bar{X}_n\) has a concentration inequality: \[P(|\bar{X}_n-\mathbb{E}(\bar{X}_n)|\geq \epsilon)\leq \frac{\sigma^2}{n\epsilon^2}.\]
However, when the RVs are bounded, there is a stronger notion of convergence, as described in the following theorem.
Theorem 5 (Hoeffding’s inequality). Let \(X_1,\cdots, X_n\) be IID RVs such that \(0\leq X_1\leq 1\) and let \(\bar{X}_n\) be the sample average. Then for any \(\epsilon>0\), \[P(|\bar{X}_n-\mathbb{E}(\bar{X}_n)|\geq \epsilon)\leq 2e^{-2n\epsilon^2}.\]
Before proving the Hoeffding’s inequality, we first introduce the following lemma:
Lemma 6. Let \(X\) be a random variable with \(\mathbb{E}(X)=0\) and \(a\leq X \leq b\). Then \[\mathbb{E}(e^{tX})\leq e^{t^2 (b-a)^2/8}\] for any positive number \(t\).
We will use the fact that \(x\mapsto e^{tx}\) is a convex function for all positive \(t\). Recall that a function \(g(x)\) is a convex function if for any two point \(a<b\) and \(\alpha\in [0,1]\), \[g(\alpha a+(1-\alpha)b) \leq \alpha g(a) + (1-\alpha)g(b).\]
Because \(X \in [a,b]\), we define \(\alpha_X\) to \[X = \alpha_X b + (1-\alpha_X)a.\] This implies \[\alpha_X = \frac{X-a}{b-a}\]
Using the fact that \(x\mapsto e^{tx}\) is convex, \[e^{tX} \leq \alpha_X e^{tb} +(1-\alpha_X) e^{ta} = \frac{X-a}{b-a}e^{tb} + \frac{b-X}{b-a} e^{ta}.\] Now taking the expectation in both sides, \[\begin{equation} \mathbb{E}(e^{tX}) \leq \frac{\mathbb{E}(X)-a}{b-a}e^{tb} + \frac{b-\mathbb{E}(X)}{b-a} e^{ta} = \frac{b}{b-a} e^{ta} - \frac{a}{b-a} e^{tb} = e^{g(s)}, \label{eq::egs} \end{equation}\] where \(s = t(b-a)\) and \(g(s) = -\gamma s+\log(1-\gamma+\gamma e^{s})\) and \(\gamma = -a/(b-a)\). Note that \(g(0) = g'(0) = 0\) and \(g''(s)\leq 1/4\) for all positive \(s\). Using Taylor’s theorem, \[g(s) = g(0) + sg'(0) + \frac{1}{2}s^2g''(s^*)\] for some \(s^* \in [0,s]\). Thus, we conclude \(g(s) \leq \frac{1}{2}\times s^2\times \frac{1}{4}=\frac{1}{8}s^2\).
Then equation \(\eqref{eq::egs}\) implies \[\mathbb{E}(e^{tX}) \leq e^{g(s)} \leq e^{\frac{s^2}{8}} = e^{\frac{t^2(b-a)^2}{8}}.\]
Now we formally prove the Hoeffding’s inequality.
We first prove that \(P\left(\bar{X}_n-\mu\geq\epsilon\right)\leq e^{-2n\epsilon^2/(b-a)^2}\).
Let \(Y_i = X_i-\mu\). Because the exponential function is monotonic, for any positive \(r\), \[\begin{align*} P\left(\bar{X}_n-\mu\geq\epsilon\right) &=P\left(\bar{Y}_n\geq\epsilon\right)\\ & = P\left(\sum_{i=1}^n Y_i\geq n \epsilon\right)\\ & = P\left(e^{\sum_{i=1}^n Y_i}\geq e^{n\epsilon}\right)\\ & = P\left(e^{t\sum_{i=1}^n Y_i}\geq e^{tn\epsilon}\right)\\ &\leq \frac{\mathbb{E}(e^{t\sum_{i=1}^n Y_i})}{e^{tn\epsilon}}\quad \mbox{ by Markov's inequality}\\ &= e^{-tn\epsilon}\mathbb{E}(e^{tY_1}\cdot e^{tY_2}\cdots e^{tY_n})\\ &= e^{-tn\epsilon}\mathbb{E}(e^{tY_1})\cdot \mathbb{E}(e^{tY_2})\cdots \mathbb{E}(e^{tY_n})\\ &= e^{-tn\epsilon}\mathbb{E}(e^{tY_1})^n\\ &\leq e^{-tn\epsilon}e^{nt^2 (b-a)^2/8} \mbox{ by Lemma~\ref{lem::Gaussian_tail}}. \end{align*}\]
Because the above inequality holds for all positive \(t\), we can choose \(t\) to optimize the bound. To get the bound as sharp as possible, we would like to make it as small as possible. Thus, we need to find \(t\) such that \[-tn\epsilon + nt^2 (b-a)^2/8\] is minimized. Taking derivatives with respect to \(t\) and set it to be \(0\), we obtain \[t_* = \frac{4\epsilon}{(b-a)^2}\] and \[-t_*n\epsilon + nt_*^2 (b-a)^2/8 = -2n\epsilon^2/(b-a)^2.\] Thus, the inequality becomes \[\begin{align*} P\left(\bar{X}_n-\mu\geq\epsilon\right) \leq e^{-t_*n\epsilon}e^{nt^2_* (b-a)^2/8} = e^{-2n\epsilon^2/(b-a)^2}. \end{align*}\]
The same proof also applies to the case \(P\left(\bar{X}_n-\mu\leq \epsilon\right)\) and we will obtain the same bound. Therefore, we conclude that \[P\left(|\bar{X}_n-\mu|\geq\epsilon\right) \leq 2 e^{-2n\epsilon^2/(b-a)^2}.\]
Hoeffding’s inequality gives a concentration of the order of exponential (actually it is often called a Gaussian rate) so the convergence rate is much faster than the one given by the Chebyshev’s inequality. Obtaining such an exponential rate is useful for analyzing the property of an estimator. Many contemporary statistical topics, including high-dimensional inference, nonparametric methods, semi-parametric inference, and empirical risk minimization all rely on a convergence rate of this form.
Note that the exponential rate may also be used to obtain an almost sure convergence via the Borel-Cantelli Lemma.
Example: consistency of estimating a high-dimensional proportion. To see how the Hoeffding’s inequality is useful, we consider the problem of estimating the proportion of several binary variables. Suppose that we observe IID observations \[X_1,\cdots, X_n \in \{0,1\}^d.\] \(X_{ij}=1\) can be interpreted as the \(i\)-th individual responding ‘Yes’ to the \(j\)-th question. We are interested in estimating the proportion vector \(\pi \in[0,1]^d\) such that \(\pi_j = P(X_{ij} = 1)\) is the proportion of ‘Yes’ response in \(j\)-th question in the population. A simple estimator is the sample proportion \(\hat \pi = (\hat \pi_1,\cdots, \hat\pi_d)^T\) such that \[\hat \pi_j = \frac{1}{n}\sum_{i=1}^n X_{ij}.\] When \(d\) is much smaller than \(n\), it is easy to see that this is a good estimator. However, if \(d=d_n\rightarrow\infty\) with \(n\rightarrow\infty\), does \(\hat{\pi}\) remain a statistically reliable estimator of \(\pi\)? To define a good estimator, we mean that every proportion can be estimated accurately. A simple way to quantify this is the vector max norm: \[\|\hat \pi-\pi\|_{\max} = \max_{j=1,\cdots, d}|\hat \pi_j - \pi_j|.\] We consider the problem of estimating \(\pi_j\) first. It is easy to see that by the Hoeffding’s inequality, \[P(|\hat \pi_j - \pi_j|> \epsilon)\leq 2e^{-2n\epsilon^2}.\] Thus, \[\begin{equation} \begin{aligned} P(\|\hat \pi-\pi\|_{\max}>\epsilon)& = P\left(\max_{j=1,\cdots, d}|\hat \pi_j - \pi_j| >\epsilon\right)\\ &\leq \sum_{j=1}^d P(|\hat \pi_j - \pi_j|> \epsilon)\\ &\leq 2d e^{-2n\epsilon^2}. \end{aligned} \label{eq::unif} \end{equation}\] Thus, as long as \(2d e^{-2n\epsilon^2}\rightarrow 0\) for any fixed \(\epsilon\), we have the statistical consistency. This implies that we need \[\frac{\log d}{n} \rightarrow 0,\] which allows the number of questions/variables to increase a lot faster than the sample size \(n\)!