Reference: Casella and Berger Chapter 2.1.
In the previous lecture, we examined several distributions with useful properties. In data analysis, we often apply transformations to random variables. If the distribution of a random variable prior to transformation is known, how can we determine the distribution of the transformed variable?
Let \(X\) be a continuous random variable with a known probability density function (PDF) \(p_X(x)\). Consider a measurable function \(f\) and define a new random variable \(Y = f(X)\). We aim to derive the probability distribution of \(Y\).
When \(f\) is continuously differentiable, the following theorem is applicable.
Theorem 1. Assume that the support of \(X\) is \([a,b]\) and \(f\) is strictly increasing with \(f'(x)>0\) for all \(x\in(a,b)\). Then the PDF of \(Y\) is given by \[p_Y(y) = \begin{cases} \frac{p_X(f^{-1}(y))}{f'(f^{-1}(y))},\quad &f(a)\leq y\leq f(b)\\ 0,\quad&\mbox{otherwise}. \end{cases}\]
To start with, we consider the cumulative distribution function (CDF) of \(Y\): \[\begin{align*} P(Y\leq y) & = P(f(X)\leq y)\\ & = P(X\leq f^{-1}(y)). \end{align*}\] The PDF is obtained by differentiating the CDF with respect to \(y\): \[\begin{align*} p_Y(y) & = \frac{d}{dy} P(Y\leq y)\\ & = \frac{d}{dy}P(X\leq f^{-1}(y))\\ & = p_X(f^{-1}(y)) \frac{d}{dy} f^{-1}(y)\\ & = \frac{p_X(f^{-1}(y))}{f'(f^{-1}(y))}, \end{align*}\] which completes the proof.
Example. Suppose \(f(x) = x^2\) and \(X\sim {\sf Uniform}[0,1]\). And we are interested in the PDF of \(Y = f(X) = X^2\). Because \(f'(x) = 2x\) and \(X\geq 0\) so \(f^{-1}(y) = \sqrt{y}\), we have \[\begin{align*} p_Y(y) = \frac{1}{2\sqrt{y}}I(0\leq y\leq 1). \end{align*}\]
Example. Assume \(X\sim {\sf Uniform}[0,1]\) and consider \(f(x) = -2\log X\) and let \(Y = -2\log X\). In this case, \(f'(x) = -\frac{2}{X}\) and \(f^{-1}(y) = e^{-\frac{1}{2}y}\). However, \(f'(x)\) is strictly negative, so Theorem 1 does not apply directly. A straightforward modification establishes that an analogous formula holds provided we replace \(f'(f^{-1}(y))\) by \(|f'(f^{-1}(y))|\).
Then the PDF of \(Y\) will be \[\begin{align*} p_Y(y) = \frac{1}{2}e^{-\frac{1}{2}y}I(0\leq y) \end{align*}\] which is the Exponential distribution with parameter \(\lambda = \frac{1}{2}\).
Example. Suppose that \(Y\) is a continuous random variable with CDF \(F_Y\) and \(X\) is a uniform random variable within \([0,1]\). Then one can show that \(Z = F_Y^{-1}(X)\) has the CDF \(F_Z(z) = F_Y(z)\).
Example. Consider \(X\sim N(0,1)\) and \(Y = X^2\). What is the distribution of \(Y\)? Note that the transformation \(f(x) = x^2\) is not strictly monotone over \(\mathbb{R}\). In such cases, a general and robust strategy is to first derive the CDF: \[\begin{align*} F_Y(y) &= P(Y\leq y)\\ & = P(X^2\leq y) \\ & = P(-\sqrt{y}\leq X\leq \sqrt{y})\\ & = F_X(\sqrt{y})-F_X(-\sqrt{y}). \end{align*}\] Thus, \[\begin{align*} p_Y(y) &= \frac{d}{dy}[F_X(\sqrt{y})-F_X(-\sqrt{y})]\\ & = \frac{1}{2\sqrt{y}}(p_X(\sqrt{y}) + p_X(-\sqrt{y})). \end{align*}\] In this case, because \(X\sim N(0,1)\), it is symmetric so we further have \[p_Y(y) = \frac{1}{\sqrt{y}}p_X(\sqrt{y}).\] Putting \(p_X(x) = \frac{1}{\sqrt{2\pi}}e^{-x^2/2}\) into the above equation, we obtain \[p_Y(y) = \frac{1}{\sqrt{y}}\frac{1}{\sqrt{2\pi}}e^{-y/2} = \frac{1}{\sqrt{2\pi}}y^{-\frac{1}{2}}e^{-\frac{1}{2}y},\] which is Gamma \((\frac{1}{2},\frac{1}{2})\). Note: The Gamma \((\frac{1}{2},\frac{1}{2})\) distribution corresponds exactly to the \(\chi^2_1\) distribution, which is the chi-squared distribution with \(1\) degree of freedom.
In practice, one frequently encounters functions of multiple random variables. Let \(X\) and \(Y\) be two random variables with a known joint PDF \(p(x,y)\). We seek the distribution of a new random variable \(U = f(X,Y)\) for a given measurable function \(f\). A general strategy is to first derive the CDF of \(U\) and subsequently differentiate it to obtain the PDF. Here we will illustrate the idea via a few examples.
Example. Consider \((X,Y)\) to be a uniform distribution over \([0,1]\times [0,1]\). Note that in this case, they are from two independent uniform distributions.
Case 1: \(U= X+Y\). Note that the event \(\{U= X+Y\leq u\}\) will be the region of \([0,1]\times [0,1]\) intersecting with \(x+y\leq u\). So it will be \(0\) when \(u\leq 0\) and \(1\) when \(u\geq 2\). When \(u\in[0,2]\), we can easily work it out using the area of a triangle, which leads to \[\begin{align*} F_U(u) & = P(U\leq u)\\ & = \begin{cases} 0,\quad&u<0\\ u^2/2,\quad&0\leq u\leq 1\\ 1-(2-u)^2/2,\quad&1\leq u\leq 2\\ 1,\quad&u>2 \end{cases}. \end{align*}\] The PDF \(p_U(u)\) will be \[\begin{align*} p_U(u) & = \begin{cases} 0,\quad&u<0\\ u,\quad&0\leq u\leq 1\\ 2-u,\quad&1\leq u\leq 2\\ 0,\quad&u>2 \end{cases}. \end{align*}\]
Case 2: \(U= \max\{X,Y\}\). A standard technique to compute the distribution of the maximum of independent random variables relies on the following equivalence of events: \[\{\max\{X,Y\}\leq u\} \equiv \{X\leq u, Y\leq u\}.\] Therefore, \[F_U(u) = P(U\leq u) = P(\max\{X,Y\}\leq u) = P(X\leq u, Y\leq u) = P(X\leq u)P(U\leq u),\] which implies \(F_U(u) = u^2\) and \(p_U(u) = 2u\) when \(u\in[0,1]\).
Case 3: \(U= \min\{X,Y\}\). The case of the minimum is analogous to that of the maximum, but we consider the complementary inequalities: \[\{\min\{X,Y\}> u\} \equiv \{X> u, Y> u\}.\] Therefore. \[1-F_U(u) = P(U> u) = P(\min\{X,Y\}> u) = P(X> u, Y> u) = P(X> u)P(U> u) = (1-u)^2,\] Thus, \(F_U(u) = 1-(1-u)^2\) so \(p_U(u) = 2-2u\) for \(u\in[0,1]\).
Example (minimum of many uniforms). Now consider \(X_1,\cdots, X_n\) that are IID from a uniform distribution over \([0,1]\). Define \(U = n \min\{X_1,\cdots, X_n\}\). What is the asymptotic distribution of \(U\) as \(n \to \infty\)? Using the technique that we have discussed, \[\{\min\{X_1,\cdots,X_n\}> u\} \equiv \{X_1> u,\cdots, X_n> u\},\] so \[1-F_U(u) = P\left(\min\{X_1,\cdots,X_n\}> \frac{u}{n}\right) = \prod_{i=1}^n P\left(X_i>\frac{u}{n}\right) = \left(1-\frac{u}{n}\right)^n \rightarrow e^{-u}.\] As a result, \(F_U(u) \rightarrow 1-e^{-u}\) and \(p_U(u)\rightarrow e^{-u}\), so as \(n \to \infty\), \(U\) converges in distribution to a standard exponential random variable.
Example (exponential distributions). Consider \(X,Y\) are IID from exponential distribution with parameter \(1\).
Sum of two exponentials. What is the distribution of \(U = X+Y\)? A useful technique involves conditioning or performing multivariate integration over the defined region. Specifically, for a given \(u>0\), \[\begin{align*} F_U(u)&= P(U\leq u)\\ & = P(X+Y\leq u)\\ & = \int_{x+y\leq u} e^{-x-y}dxdy\\ & = \int_{x=0}^u \int_{y=0}^{u-x} e^{-x-y}dydx\\ & = \int_{x=0}^u e^{-x} (1-e^{x-u})dx\\ & = 1-e^{-u}-ue^{-u}. \end{align*}\] Thus, \(p_U(u) = ue^{-u}\).
Minimum of two exponentials. Now we consider \(V = \min\{X,Y\}\). Using the same trick as the minimum of many uniforms, i.e., \[\{\min\{X,Y\}>v\} \equiv \{X>v,Y>v\}\] so \[1-F_V(v) = P(X>v)P(Y>v) = e^{-2v},\] which implies that \(V\sim {\sf Exp}(2)\). In fact, you can easily generalize it to showing that if \(X_1,\cdots, X_n\sim {\sf Exp}(\lambda)\), then \(\min\{X_1,\cdots, X_n\}\sim {\sf Exp}(n\lambda)\).
Difference. Consider \[Z = \max\{X,Y\} - \min\{X,Y\} = |X-Y|.\] What will the distribution of \(Z\) be?
Using a direct computation, we see that \[\begin{align*} F_Z(z) & = P(Z\leq z)\\ & = P(|X-Y|\leq z)\\ & = P(-z\leq X-Y\leq z)\\ & = P(X- Y\leq z) - P(X-Y< -z)\\ & = P(X\leq Y+z) - 1+P(X-Y\geq -z)\\ & = -1 +P(X\leq Y+z) + P(Y\leq X+z)\\ &= -1 + 2P(X\leq Y+z) \qquad\mbox{$X$ and $Y$ are symmetric and exchangeable}. \end{align*}\] Moreover, \[\begin{align*} P(X\leq Y+z) & = \int_{y=0}^\infty \int_{x=0}^{y+z} e^{-x}dx e^{-y}dy\\ & = \int_{y=0}^\infty (1-e^{-y-z})e^{-y}dy\\ & = 1-e^{-z}\int_0^\infty e^{-2y}dy\\ & = 1-\frac{1}{2}e^{-z}. \end{align*}\] As a result, \[F_Z(z) = -1 + 2P(X\leq Y+z) = 1-e^{-z},\] which is the CDF of \({\sf Exp}(1)\)! This is another memoryless property.
Ratio. Finally, we consider \(W = \frac{X}{X+Y}\) and study its distribution. Since \(X, Y \geq 0\), it is clear that \(W \in [0,1]\); hence, we restrict our attention to this interval. \[\begin{align*} F_W(w) & = P\left(\frac{X}{X+Y}\leq w\right)\\ & = P(X\leq w(X+Y))\\ & = P((1-w)X\leq wY)\\ & = P\left(X\leq \frac{w}{1-w} Y\right)\\ & = \int_{y=0}^\infty \int_{x=0}^{\frac{w}{1-w}y}e^{-x}dx e^{-y}dy\\ & = \int_{y=0}^\infty (1-e^{\frac{-w}{1-w}y}) e^{-y}dy\\ & = 1 -\int_0^\infty e^{-\frac{1}{1-w}y}dy\\ & = 1- 1+w = w. \end{align*}\] Thus \(W\sim {\sf Unif}[0,1]\).
Useful properties of the normal distribution (left as an exercise).
Let \(X\sim N(\mu_1,\sigma^2)\) and \(Y\sim N(\mu_2,\sigma_2^2)\) be independent. Then \[X+Y \sim N(\mu_1+\mu_2, \sigma_1^2+\sigma_2^2)\] Also, for any real number \(a\), \[aX \sim N(a\mu_1, a^2 \sigma_1^2).\]
Let \(X_1,\cdots, X_n\) be IID normal random variables from \(N(\mu,\sigma^2)\). Then the sample mean \[\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i\sim N(\mu,\sigma^2/n).\]
Let \(X_1,\cdots, X_n\) be IID normal random variables from \(N(0,1)\). Then \(Z_1 = X_1^2\) follows a \(\chi^2\) distribution with \(1\) degree of freedom. And \(Z_n = \sum_{i=1}^n X_i^2\) follows a \(\chi^2\) distribution with \(n\) degrees of freedom.