Sample Space and Random Variables

Sample Space and Probability Measure

The sample space \(\Omega\) is the collection of all possible outcomes of a random experiment, e.g. toss of a coin, \(\Omega =\{ H,T\}\). Elements \(\omega\in \Omega\) are called outcomes, realizations or elements. Subsets \(A\subseteq\Omega\) are called events. You should able to express events of interest using the standard set operations. For instance:

• “Not \(A\)” corresponds to the complement \(A^c=\Omega\setminus A\);

• “\(A\) or \(B\)” corresponds to the union \(A\cup B\);

• “\(A\) and \(B\)” corresponds to the intersection \(A\cap B\).

We said that \(A_1,A_2,...\) are pairwise disjoint/mutually exclusive if \(A_i \cap A_j = \emptyset\) for all \(i\not= j\). A partition of \(\Omega\) is a sequence of pairwise disjoint sets \(A_1,A_2,...\) such that \(\cup_{i=1}^\infty A_i = \Omega\). We use \(|A|\) to denote the number of elements in \(A\).

The sample space defines basic elements and operations of events. But it is still too simple to be useful in describing our senses of ‘probability’. Now we introduce the concept of \(\sigma\)-algebra.

A \(\sigma\)-algebra \(\mathcal{F}\) is a collection of subsets of \(\Omega\) satisfying:

1. (full and null set) \(\Omega \in \mathcal{F}\), \(\emptyset \in \mathcal{F}\) (\(\emptyset\) = empty set).

2. (complement)\(A \in \mathcal{F} \Rightarrow A^c \in \mathcal{F}\).

3. (countably union) \(A_1,A_2,... \in \mathcal{F} \Rightarrow \bigcup_{i=1}^{\infty} A_i \in \mathcal{F}\).

The sets in \(\mathcal{F}\) are said to be measurable and \((\Omega,\mathcal{F})\) is a measurable space. The intuition of a set being measurable is that we can find a function that takes the elements of \(\mathcal{F}\) and output a real number; this number represents the ‘size’ of the input element.

Now we introduce the concept of probability. Intuitively, probability should be associated with an event – when we say a probability of something, this ‘something’ is an event. Using the fact that the \(\sigma\)-algebra \(\mathcal{F}\) is a collection of events and the property that \(\mathcal{F}\) is measurable, we then introduce a measure called probability measure \(P(\cdot)\) that assigns a number between \(0\) and \(1\) to every element of \(\mathcal{F}\). Namely, this function \(P\) maps an event to a number, describing the likelihood of the event.

Formally, a probability measure is a mapping \(P:\mathcal{F}\mapsto \mathbb{R}\) satisfying the following three axioms

1. \(P(\Omega) = 1\).

2. \(P(A) \geq 0\) for all \(A \in \mathcal{F}\).

3. (countably additivity) \(P\left(\bigcup_{i=1}^{\infty} A_i \right) = \sum_{i=1}^{\infty}P(A_i)\) for mutually exclusive events \(A_1,A_2,... \in \mathcal{F}\).

The triplet \((\Omega,\mathcal{F},P)\) is called a probability space.

The three axioms imply: \[\begin{align*} &P( \emptyset) = 0\\ & 0 \leq P(A) \leq 1\\ &A \subset B \Longrightarrow P ( A ) \leq P (B ), \\ &P( A^c ) = 1 - P(A)\\ &P( A \cup B) = P ( A ) + P(B) - P( A \cap B) \end{align*}\] The countable additivity (P3) also implies that if a sequence of sets \(A_1,A_2,\dots\) in \(\mathcal{F}\) satisfying \(A_n\subseteq A_{n+1}\) for all \(n\), then \[P\left( \bigcup_{n=1}^\infty A_n \right) \;=\; \lim_{n\to\infty} P(A_n).\] If \(A_n\supseteq A_{n+1}\) for all \(n\), then \[P\left( \bigcap_{n=1}^\infty A_n \right) \;=\; \lim_{n\to\infty} P(A_n).\]

How do we interpret the probability? There are two major views in statistics. The first view is called the frequentist view – the probability is interpreted as the limiting frequencies observed over repetitions in identical situations. The other view is called the Bayesian/subjective view where the probability quantifies personal belief. One way of assigning probabilities is the following. The probability of an event \(E\) is the price one is just willing to pay to enter a game in which one can win a unit amount of money if \(E\) is true. Example: If I believe a coin is fair and am to win 1 unit if a head arises, then I would pay \(\frac{1}{2}\) unit of money to enter the bet.

Now we have a basic mathematical model for probability. This model also defines an interesting quantity called conditional probability. For two events \(A,B\in\mathcal{F}\), the conditional probability of \(A\) given \(B\) is \[P(A|B) = \frac{P(A\cap B)}{P(B)}.\] Note that when \(B\) is fixed, the function \(P(\cdot|B):\mathcal{F}\mapsto \mathbb{R}\) is another probability measure.

In general, \(P(A|B)\neq P(B|A)\). This is sometimes called as the prosecutor’s fallacy: \[P(\mbox{evidence}|\mbox{guilty})\neq P(\mbox{guilty}|\mbox{evidence}).\]

The probability has a power feature called independence. This property is probably the key property that makes the ‘probability theory’ distinct from measure theory. Intuitively, when we say that two events are independent, we refers to the case that the two event will not interfere each other. Two events \(A\) and \(B\) are independent if \[P(A|B) = P(A)\qquad \mbox{(or equivalently, $P(A\cap B) = P(A)P(B)$)}.\] For three events \(A,B,C\), we say events \(A\) and \(B\) are conditional independent given \(C\) if \[P(A\cap B|C) = P(A|C) P(B|C)\]

Probability measure also has a useful property called law of total probability. If \(B_1,B_2,...,B_k\) forms a partition of \(\Omega\), then \[P(A) = \sum_{i=1}^k P(A|B_i) P(B_i).\] In particular, \(P(A) = P(A|B) P(B) + P(A|B^c)P(B^c)\). And this further implies the famous Bayes rule: Let \(A_1,...,A_k\) be a partition of \(\Omega\). If \(P(B)>0\) then, for \(i=1,...,k\): \[P(A_i|B) =\frac{P(B|A_i)P(A_i)}{\sum_{j=1}^k P(B|A_j)P(A_j)}.\]

Random Variable

So far, we have built a mathematical model describing the probability and events. However, in reality, we are dealing with numbers, which may not be directly link to events. We need another mathematical notion that bridges the events and numbers and this is why we need to introduce random variables.

Informally, a random variable is a mapping \(X:\Omega\mapsto \mathbb{R}\) that assigns a real number \(X(\omega)\) to each outcome \(\omega\in\Omega\). Fo example, we toss a coin 2 times and let \(X\) represents the number of heads. The sample space is \(\Omega=\{HH,HT,TH, TT\}\). Then for each \(\omega\in \Omega,\) \(X(\omega)\) outputs a real number: \(X(\{HH\})=2\), \(X(\{HT\})=X(\{TH\})=1,\) and \(X(\{TT\})=0\).

Rigorously, a function \(X(\omega): \Omega \rightarrow {\mathbb{R}}\) is called a random variable (R.V.) if \(X(\omega)\) is measurable with respect to \(\mathcal{F}\), i.e. \[X^{-1}((-\infty,c]):= \{ \omega \in \Omega : X(\omega) \leq c \} \in \mathcal{F}, \quad \text{for all} \ c\in\mathbb{R}.\] Note that the condition is also equivalent to saying that \(X^{-1}(B) \in \mathcal{F}\) for every Borel set \(B\)¹. This means that the set \(X^{-1}(B)\) is indeed an event so that it makes sense to talk about \(P(X \in B)\), the probability that \(X\) lies in \(B\), for any Borel set \(B\). The function \(B\mapsto P(X\in B)\) is a probability measure and is called the (probability) distribution of \(X\).

A very important characteristic of a random variable is its cumulative distribution function (CDF), which is defined as \[F(x) = P(X\leq x) = P(\{\omega\in\Omega: X(\omega)\leq x\}).\] Actually, the distribution of \(X\) is completely determined by the CDF \(F(x)\), regardless of \(X\) being a discrete random variable or a continuous random variable (or a mix of them).

When \(X\) takes discrete values, we may characterize its distribution using the probability mass function (PMF): \[p(x) = P(X=x) = F(x)-F(x^{-}),\] where \(F(x^-) = \lim_{\epsilon\rightarrow0}F(x-\epsilon)\). In this case, one can recover the CDF from PMF using \(F(x) = \sum_{x'\leq x} p(x')\).

If \(X\) is an absolutely continuous random variable, we may describe its distribution using the probability density function (PDF): \[p(x) = F'(x) = \frac{d}{dx}F(x).\] In this case, the CDF can be written as \[F(x) = P(X\leq x) = \int_{-\infty}^x p(x')dx'.\]

However, the PMF and PDF are not always well-defined. There are situations where \(X\) does not have a PMF or a PDF (see Example 1). The formal definition of PMF and PDF requires the notion of the Radon-Nikodym derivative, which is beyond the scope of this course.

Common Distributions

Discrete Random Variables

Bernoulli. If \(X\) is a Bernoulli random variable with parameter \(p\), then \(X=0\) or, \(1\) such that \[P(X=1) = p,\quad P(X=0)=1-p.\] In this case, we write \(X\sim {\sf Ber}(p)\).

Binomial. If \(X\) is a binomial random variable with parameter \((n,p)\), then \(X=0,1,\cdots, n\) such that \[P(X=k) = {n \choose k} p^k (1-p)^{n-k}.\] In this case, we write \(X\sim {\sf Bin}(n,p)\). Note that if \(X_1,\cdots, X_n\sim {\sf Ber}(p)\), then the sum \(S_n = X_1+X_2+\cdots+X_n\) is a binomial random variable with parameter \((n,p)\).

Geometric. If \(X\) is a geometric random variable with parameter \(p\), then \[P(X=n) = (1-p)^{n-1}p\] for \(n=1,2,\cdots\). Geometric random variable can be constructed using ‘the number of trials of the first success occurs’. Consider the case we are flipping coin with a probability \(p\) that we gets a head (this is a Bernoulli \((p)\) random variable). Then the number of trials we made to see the first head is a geometric random variable with parameter \(p\).

Poisson. If \(X\) is a Poisson random variable with parameter \(\lambda\), then \(X =0,1,2,3,\cdots\) and \[P(X=k)= \frac{\lambda^ke^{-\lambda}}{k!}.\] In this case, we write \(X\sim {\sf Poi}(\lambda)\). Poisson is often used to model a counting process. For instance, the intensity of an image is commonly modeled as a Poisson random variable.

Example: Wright-Fisher Model. Recall the Wright-Fisher model: \(X_n\) is the number of \(A\) alleles in the population at generation \(n\), with \(2m\) alleles in all. We have \(2m\) Bernoulli trials with \({P}(A) = j/2m\) where \(j\) is the number of \(A\) alleles in the previous generation (recall, assumed sampling with replacement). The probability of \(X_{n+1}=k\) given \(X_n=j\) is Binomial\((2m,j/2m)\): \[{P}(X_{n+1}=k | X_n=j ) = {2m \choose k}\left( \frac{j}{2m}\right)^k\left( 1- \frac{j}{2m}\right)^{2m-k},\] for \(j,k=0,1,....,2m\).

Continuous Random Variables

Uniform. If \(X\) is a uniform random variable over the interval \([a,b]\), then \[p(x) = \frac{1}{b-a}I(a\leq x \leq b),\] where \(I({\sf statement})\) is the indicator function such that if the \({\sf statement}\) is true, then it outputs \(1\) otherwise \(0\). Namely, \(p(x)\) takes value \(\frac{1}{b-a}\) when \(x\in[a,b]\) and \(p(x)=0\) in other regions. In this case, we write \(X\sim {\sf Uni}[a,b]\).

Normal. If \(X\) is a normal random variable with parameter \((\mu,\sigma^2)\), then \[p(x) =\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}.\] In this case, we write \(X\sim N(\mu,\sigma^2)\).

Exponential. If \(X\) is an exponential random variable with parameter \(\lambda\), then \(X\) takes values in \([0,\infty)\) and \[p(x) = \lambda e^{-\lambda x}.\] In this case, we write \(X\sim {\sf Exp}(\lambda)\). Note that we can also write \[p(x) = \lambda e^{-\lambda x}I(x\geq 0).\]

Distribution functions

While people often informally refer the term distribution function as the PMF or PDF, the distribution function formally refers to the CDF of a random variable. There are random variables who may not have a well-defined or meaningful PDF/PMF but can still have a well-defined CDF. Here is one example.

Properties of Random Variables

Conditional Probability and Independence

For two random variables \(X,Y\), the joint CDF is \[P_{XY}(x,y) = F(x,y) = P(X\leq x, Y\leq y).\] When both variables are absolute continuous, the corresponding joint PDF is \[p_{XY}(x,y) = \frac{\partial^2F(x,y)}{\partial x\partial y}.\] The conditional PDF of \(Y\) given \(X=x\) is \[p_{Y|X}(y|x) = \frac{p_{XY}(x,y)}{p_X(x)},\] where \(p_X(x) = \int_{-\infty}^\infty p_{XY}(x,y)dy\) is sometimes called the marginal density function.

When both \(X\) and \(Y\) are discrete, the joint PMF is \[p_{XY}(x,y) = P(X= x, Y= y)\] and the conditional PMF of \(Y\) given \(X=x\) is \[p_{Y|X}(y|x) = \frac{p_{XY}(x,y)}{p_X(x)},\] where \(p_X(x) = P(X=x) = \sum_{y}P(X=x,Y=y)\).

Random variables \(X\) and \(Y\) are independent if the joint CDF can be factorized as \[F(x,y) = P(X\leq x, Y\leq y) = P(X\leq x) P(Y\leq y).\]

For random variables, we also have the Bayes theorem: \[\begin{eqnarray*} p_{X|Y} (x|y) &=& \frac{p_{XY}( x,y)}{p_{Y}(y)}\\ &=& \frac{p_{Y|X}(y|x)p_X(x)}{p_Y(y)}\\ &=&\begin{cases} \frac{p_{Y|X}(y|x)p_X(x)}{\int p_{Y|X}(y|x')p_X(x')dx'}, &\quad\mbox{if $X,Y$ are absolutely continuous.}\\ \frac{p_{Y|X}(y|x)p_X(x)}{\sum_{x'} p_{Y|X}(y|x')p_X(x')}, &\quad\mbox{if $X,Y$ are discrete.} \end{cases} \end{eqnarray*}\]

Expectation

For a function \(g(x)\), the expectation of \(g(X)\) is \[\mathbb{E}(g(X)) = \int g(x) dF(x) = \begin{cases} \int_{-\infty}^\infty g(x) p(x)dx, \quad &\mbox{if $X$ is continuous}\\ \sum_{x} g(x) p(x), \quad &\mbox{if $X$ is discrete} \end{cases}.\]

Here are some useful properties and quantities related to the expected value:

• \(\mathbb{E}(\sum_{j=1}^k c_j g_j(X)) = \sum_{j=1}^k c_j \cdot\mathbb{E}(g_j(X_i)).\)

• We often write \(\mu=\mathbb{E}(X)\) as the mean (expectation) of \(X\).

• \({\sf Var}(X) = \mathbb{E}((X-\mu)^2)\) is the variance of \(X\).

• If \(X_1,\cdots,X_n\) are independent, then \[\mathbb{E}\left(X_1\cdot X_2\cdots X_n\right) = \mathbb{E}(X_1)\cdot \mathbb{E}(X_2)\cdots \mathbb{E}(X_n).\]

• If \(X_1,\cdots,X_n\) are independent, then \[{\sf Var}\left(\sum_{i=1}^n a_iX_i\right) = \sum_{i=1}^n a_i^2 \cdot {\sf Var}(X_i).\]

• For two random variables \(X\) and \(Y\) with their mean being \(\mu_X\) and \(\mu_Y\) and variance being \(\sigma^2_X\) and \(\sigma^2_Y\). The covariance \[{\sf Cov}(X,Y) = \mathbb{E}((X-\mu_x)(Y-\mu_y)) = \mathbb{E}(XY) - \mu_x\mu_y\] and the (Pearson’s) correlation \[\rho(X,Y) = \frac{{\sf Cov}(X,Y)}{\sigma_x\sigma_y}.\]

The conditional expectation of \(Y\) given \(X\) is the random variable \(\mathbb{E}(Y|X) = g(X)\) such that when \(X=x\), its value is \[\mathbb{E}(Y|X=x) = \int y p(y|x) dy,\] where \(p(y|x) = p(x,y)/p(x)\). Note that when \(X\) and \(Y\) are independent, \[\mathbb{E}(XY) = \mathbb{E}(X)\mathbb{E}(Y),\quad \mathbb{E}(X|Y=y) = \mathbb{E}(X).\]

Law of total expectation: \[\begin{eqnarray*} \mathbb{E}[ \mathbb{E}[Y|X]] &=& \int \mathbb{E}[Y|X=x] p_X(x) dx = \int \int y p_{Y|X}(y|x) p_X(x) dx dy\\ &=& \int \int y p_{XY}(x,y) dx dy = \mathbb{E}[Y]. \end{eqnarray*}\]

Law of total variance: \[\begin{eqnarray*} {\sf Var}(Y) &=& \mathbb{E}[Y^2] - \mathbb{E}[Y]^2 \\&=& \mathbb{E}[\mathbb{E}(Y^2|X)] - \mathbb{E}[ \mathbb{E}(Y|X)]^2\quad\mbox{(law of total expectation)}\\ &=&\mathbb{E}[{\sf Var}(Y|X) + \mathbb{E}(Y|X)^2] - \mathbb{E}[\mathbb{E}(Y|X)]^2\quad\mbox{(definition of variance)}\\ %&~&\mbox{definition of variance}\\ &=&\mathbb{E}[{\sf Var}(Y|X)] + \left\{ \mathbb{E}[\mathbb{E}(Y|X)^2] - \mathbb{E}[\mathbb{E}(Y|X)]^2 \right\}\\ &=& \mathbb{E}\left[{\sf Var}(Y | X)\right] + {\sf Var}\left(\mathbb{E}[Y \mid X]\right)\quad\mbox{(definition of variance)}. \end{eqnarray*}\]

Moment Generating Function and Characteristic Function

Moment generating function (MGF) and characteristic function are powerful functions that describe the underlying features of a random variable. The MGF of a RV \(X\) is \[M_X(t) = \mathbb{E}(e^{tX}).\] Note that \(M_X\) may not exist. When \(M_X\) exists in a neighborhood of \(0\), using the fact that \[e^{tX} = 1+tX+\frac{(tX)^2}{2!}+\frac{(tX)^3}{3!}+\cdots,\] we have \[M_X(t) = 1 + t\mu_1 + \frac{t^2\mu_2}{2!}+ \frac{t^3\mu_3}{3!}+\cdots,\] where \(\mu_j = \mathbb{E}(X^j)\) is the \(j\)-th moment of \(X\). Therefore, \[\mathbb{E}(X^j) = M^{(j)}(0) = \left. \frac{d^j M_X(t)}{dt^j}\right|_{t=0}\] Here you see how the moments of \(X\) is generated by the function \(M_X\).

For two random variables \(X,Y\), if their MGFs are the same, then the two random variables have the same CDF. Thus, MGFs can be used as a tool to determine if two random variables have the identical CDF. Note that the MGF is related to the Laplace transform (actually, they are the same) and this may give you more intuition why it is so powerful.

A more general function than MGF is the characteristic function. Let \(i\) be the imagination number. The characteristic function of a RV \(X\) is \[\phi_X(t) = \mathbb{E}(e^{itX}).\] When \(X\) is absolutely continuous, the characteristic function is the Fourier transform of the PDF. The characteristic function always exists and when two RVs have the same characteristic function, the two RVs have identical distribution.

Convergence

Let \(F_1,\cdots,F_n, \cdots\) be the corresponding CDFs of \(Z_1,\cdots, Z_n,\cdots\). For a random variable \(Z\) with CDF \(F\), we say that \(Z_n\) converges in distribution (a.k.a. converge weakly or converge in law) to \(Z\) if for every \(x\) that is continuous in \(F\), \[\lim_{n\rightarrow \infty}F_n(x) = F(x).\] In this case, we write \[Z_n\overset{D}{\rightarrow} Z.\] Namely, the CDF’s of the sequence of random variables converge to a the CDF of a fixed random variable.

For a sequence of random variables \(Z_1,\cdots,Z_n,\cdots\), we say \(Z_n\) converges in probability to another random variable \(Z\) if for any \(\epsilon>0\), \[\lim_{n\rightarrow \infty}P(|Z_n-Z|>\epsilon) = 0\] and we will write \[Z_n\overset{P}{\rightarrow} Z.\]

For a sequence of random variables \(Z_1,\cdots,Z_n,\cdots\), we say \(Z_n\) converges almost surely to a fixed number \(\mu\) if \[P(\lim_{n\rightarrow\infty} Z_n=Z) = 1\] or equivalently, \[P(\{\omega: \lim_{n\rightarrow\infty} Z_n(\omega)=Z(\omega)\}) = 1.\] We use the notation \[Z_n\overset{a.s.}{\rightarrow} Z\] to denote convergence almost surely.

Note that almost surely convergence implies convergence in probability. Convergence in probability implies convergence in distribution.

In many cases, convergence in probability or almost surely converge occurs when a sequence of RVs converging toward a fixed number. In this case, we will write (assuming that \(\mu\) is the target of convergence) \[Z_n\overset{P}{\rightarrow} \mu,\quad Z_n\overset{a.s.}{\rightarrow} \mu.\] Later we will see that the famous Law of Large Number is describing the convergence toward a fixed number.

Examples.

• Let \(\{X_1,X_2,\cdots, \}\) be a sequence of random variables such that \(X_n\sim N\left(0, 1+\frac{1}{n}\right)\). Then \(X_n\) converges in distribution to \(N(0,1)\).

Continuous mapping theorem: Let \(g\) be a continuous function.

• If a sequence of random variables \(X_n\overset{D}{\rightarrow}X\), then \(g(X_n)\overset{D}{\rightarrow}g(X)\).

• If a sequence of random variables \(X_n\overset{p}{\rightarrow}X\), then \(g(X_n)\overset{p}{\rightarrow}g(X)\).

Slutsky’s theorem: Let \(\{X_n:n=1,2,\cdots\}\) and \(\{Y_n:n=1,2,\cdots\}\) be two sequences of RVs such that \(X_n\overset{D}{\rightarrow} X\) and \(Y_n\overset{p}{\rightarrow}c\), where \(X\) is a RV \(c\) is a constant. Then \[\begin{align*} X_n+Y_n&\overset{D}{\rightarrow} X+c\\ X_nY_n&\overset{D}{\rightarrow} c X\\ X_n/Y_n&\overset{D}{\rightarrow} X/c \quad \mbox{(if $c\neq 0$)}. \end{align*}\]

We will these two theorems very frequently when we are talking about the maximum likelihood estimator.

Why do we need these notions of convergences? The convergence in probability is related to the concept of statistical consistency. An estimator is statistically consistent if it converges in probability toward its target population quantity. The convergence in distribution is often used to construct a confidence interval or perform a hypothesis test.

Convergence theory

We write \(X_1,\cdots,X_n\sim F\) when \(X_1,\cdots,X_n\) are IID (independently, identically distributed) from a CDF \(F\). In this case, \(X_1,\cdots,X_n\) is called a random sample.

The above theorem is also known as Kolmogorov’s Strong Law of Large Numbers.

Note that there are other versions of central limit theorem that allows dependent RVs or infinite variance using the idea of ‘triangular array’ (also known as the Lindeberg-Feller Theorem). However, the details are beyond the scope of this course so we will not pursue it here.

In addition to the above two theorems, we often use the concentration inequality to obtain convergence in probability. Let \(\{X_n:n=1,2,\cdots\}\) be a sequence of RVs. For a given \(\epsilon>0\), the concentration inequality aims at finding the function \(\phi_n(\epsilon)\) such that \[P(|X_n -\mathbb{E}(X_n)|>\epsilon)\leq \phi_n(\epsilon)\] and \(\phi_n(\epsilon)\rightarrow0\). This automatically gives us convergence in probability. Moreover, the convergence rate of \(\phi_n(\epsilon)\) with respect to \(n\) is a central quantity that describes how fast \(X_n\) converges toward its mean.

Let \(X_1,\cdots, X_n\sim F\) be a random sample such that \(\sigma^2 = {\sf Var}(X_1)\). Using the Chebyshev’s inequality, we know that the sample average \(\bar{X}_n\) has a concentration inequality: \[P(|\bar{X}_n-\mathbb{E}(\bar{X}_n)|\geq \epsilon)\leq \frac{\sigma^2}{n\epsilon^2}.\]

However, when the RVs are bounded, there is a stronger notion of convergence, as described in the following theorem.

Hoeffding’s inequality gives a concentration of the order of exponential (actually it is often called a Gaussian rate) so the convergence rate is much faster than the one given by the Chebyshev’s inequality. Obtaining such an exponential rate is useful for analyzing the property of an estimator. Many modern statistical topics, such as high-dimensional problem, nonparametric inference, semi-parametric inference, and empirical risk minimization all rely on a convergence rate of this form.

Note that the exponential rate may also be used to obtain an almost sure convergence via the Borel-Cantelli Lemma.