Pearson’s correlation is a standard measure of the linear association between two random variables. Formally, for random variables \(X\) and \(Y\), their correlation coefficient is defined as \[\rho_{XY} = {\sf Cor}(X,Y) = \frac{{\sf Cov}(X,Y)}{\sqrt{{\sf Var}(X){\sf Var}(Y)}}.\] It has three nice properties:
\({\sf Cor}(X,Y) = {\sf Cor}(Y,X)\).
Location-scale: \({\sf Cor}(aX+b, cY+d) = {\sf sign}(ac){\sf Cor}(X,Y)\).
\(-1\leq {\sf Cor}(X,Y) \leq 1\). \({\sf Cor}(X,Y) = \pm 1\) if and only if they are perfectly linearly related, i.e., \(X=aY+b\) for some constants \(a,b\).
Correlation is strictly a measure of the linear relationship between two random variables. A large absolute correlation indicates a strong linear dependence. However, a low correlation does not imply that the two random variables are independent!
Example (0 correlation but perfectly related). Consider a random variable \(X\) taking three possible values \(-1,0,1\) with probabilities \(P(X=-1) = P(X=1) = 1/4\) and \(P(X=0) = 1/2\). Let \(Y = X^2\). You can see that \(X\) and \(Y\) are deterministically related. It is easy to see that \(\mathbb{E}[X] = 0\) and \(\mathbb{E}[XY] = 0\) and \({\sf Var}(X),{\sf Var}(Y)>0\). However, the covariance between them will be \[{\sf Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] = 0 - 0 = 0,\] which implies that \({\sf Cor}(X,Y) = 0\). So they are uncorrelated but perfectly dependent.
A classical problem in statistics is prediction. The prediction problem arises when we are given an observable random variable \(X\) and aim to predict the value of an unobserved random variable \(Y\). Thus, we can think of using a quantity \(g(X)\) as a prediction of \(Y\).
To measure how good the predictor \(g(X)\) is, we often use the mean-square error (MSE): \[R(g) = \mathbb{E}((Y-g(X))^2).\] Namely, the MSE is the expected squared deviation from our predictor \(g(X)\) to the target \(Y\).
Ideally, we want to choose \(g\) that minimizes \(R(g)\). Formally, we want to find \[g^* = {\sf argmin}_{g} R(g).\]
We now take a deeper look at the MSE \(R(g) = \mathbb{E}((Y-g(X))^2)\). Using the law of total expectation, \[\mathbb{E}((Y-g(X))^2) = \mathbb{E}[\mathbb{E}[(Y-g(X))^2|X]].\] Using the fact that for any fixed constant \(c\), \[\begin{align*} \mathbb{E}[(Y-c)^2] = \mathbb{E}[(Y-\mathbb{E}[Y]+E[Y]-c)^2] = \mathbb{E}[(Y-\mathbb{E}[Y])^2] + (\mathbb{E}[Y]-c)^2 = {\sf Var}(Y) + (\mathbb{E}[Y]-c)^2, \end{align*}\] we can rewrite the MSE as \[R(g) = \mathbb{E}[\mathbb{E}[(Y-g(X))^2|X]] = \mathbb{E}[{\sf Var}(Y|X) + (\mathbb{E}[Y|X] - g(X))^2]= \mathbb{E}[{\sf Var}(Y|X) ]+ \mathbb{E}[(\mathbb{E}[Y|X] - g(X))^2].\] The first term is independent of \(g\) and thus plays no role in the optimization over \(g\). The second quantity involves \[(\mathbb{E}[Y|X] - g(X))^2 \geq 0.\] Equality holds almost surely if and only if \(g(X) = \mathbb{E}[Y|X]\). As a result, to minimize the MSE, we should use the conditional expectation \(\mathbb{E}[Y|X]\) as our predictor. The conditional expectation \(\mathbb{E}[Y|X=x] = m(x)\) is also known as the regression function or the best predictor.
With the regression function, we can decompose \(Y\) as \[\begin{equation} Y = \underbrace{\mathbb{E}[Y|X]}_\text{best predictor} + \underbrace{(Y-\mathbb{E}[Y|X])}_\text{residuals}. \label{eq::decom} \end{equation}\] Here are some interesting properties of the decomposition in equation \(\eqref{eq::decom}\):
Unbiased. \(\mathbb{E}[\mbox{best predictor} ]= \mathbb{E}[\mathbb{E}[Y|X]] = \mathbb{E}[Y]\) and \(\mathbb{E}[\mbox{residual}] = 0.\)
Uncorrelated. \({\sf Cov}(\mathbb{E}[Y|X], Y-\mathbb{E}[Y|X]) = 0.\)
Residual variance. \({\sf Var}(Y-\mathbb{E}[Y|X]) = \mathbb{E}[{\sf Var}(Y|X)]\). To see this, \[\begin{align*} {\sf Var}(Y-\mathbb{E}[Y|X]) & = {\sf Var}(Y) - 2{\sf Cov}(Y,\mathbb{E}[Y|X]) + {\sf Var}(\mathbb{E}[Y|X])\\ & = \mathbb{E}[Y^2] - \mathbb{E}[Y]^2 - 2(\mathbb{E}[Y\mathbb{E}[Y|X]] - \mathbb{E}[Y]\mathbb{E}[\mathbb{E}[Y|X]]) + \mathbb{E}[\mathbb{E}[Y|X]^2] - \mathbb{E}[\mathbb{E}[Y|X]]^2\\ & = \mathbb{E}[Y^2] - \mathbb{E}[Y]^2 - 2\mathbb{E}[\mathbb{E}[Y|X]^2] +2 \mathbb{E}[Y]^2 + \mathbb{E}[\mathbb{E}[Y|X]^2] - \mathbb{E}[Y]^2\\ & = \mathbb{E}[Y^2] - \mathbb{E}[\mathbb{E}[Y|X]^2] \\ & = \mathbb{E}[\mathbb{E}[Y^2|X] -\mathbb{E}[Y|X]^2 ] \\ & = \mathbb{E}[{\sf Var}(Y|X)]. \end{align*}\]
Variance decomposition. With the above properties, we obtain \[{\sf Var}(Y) = {\sf Var}(\mathbb{E}[Y|X]) + \mathbb{E}[{\sf Var}(Y|X)].\] This equation represents the law of total variance and admits an intuitive interpretation: \[{\sf Var}(Y) = \underbrace{{\sf Var}(\mathbb{E}[Y|X]) }_\text{{\sf Var}(best predictor)}+ \underbrace{\mathbb{E}[{\sf Var}(Y|X)]}_\text{average {\sf Var}(residuals)}.\]
In the above analysis, we see that the best predictor is the conditional expectation. However, achieving this requires searching over the space of all measurable functions \(g(x)\). In practice, it is often necessary to restrict the predictor to a simpler class of functions. A canonical choice is the class of linear functions. Specifically, we seek parameters \(\alpha,\beta\) such that the MSE \[R(\alpha,\beta) = \mathbb{E}((Y- \alpha - \beta X)^2 )\] is minimized (this is also known as the least square approach). How will we choose \(\alpha\) and \(\beta\)?
To solve this problem, we first expand \(R(\alpha,\beta)\): \[\begin{align*} R(\alpha,\beta)& = \mathbb{E}((Y- \alpha - \beta X)^2 )\\ &= \mathbb{E}(Y^2 + \alpha^2 +\beta^2 X^2 -2 Y\alpha -2XY \beta +2\alpha\beta X)\\ &= \mathbb{E}[Y^2] + \alpha^2 +\beta^2 \mathbb{E}[X^2] -2\alpha\mathbb{E}[Y] -2\beta\mathbb{E}[XY] +2\alpha\beta \mathbb{E}[X], \end{align*}\] which is a quadratic function of \(\alpha,\beta\). Thus, \[\alpha^*,\beta^* = {\sf argmin}_{\alpha,\beta} R(\alpha,\beta)\] satisfies the first-order conditions: \[\begin{align*} 0 &= \frac{\partial}{\partial\alpha} R(\alpha^*,\beta^*)\\ &= 2\alpha^* - 2\mathbb{E}[Y] -2\beta^* \mathbb{E}[X]\\ 0&= \frac{\partial}{\partial\beta} R(\alpha^*,\beta^*)\\ & = 2\beta^* \mathbb{E}[X^2] - 2\mathbb{E}[XY] + 2\alpha^*\mathbb{E}[X]\\ \Rightarrow &\quad\beta^* {\sf Var}(X) = {\sf Cov}(X,Y)\\ \Rightarrow &\quad\beta^* = \frac{{\sf Cov}(X,Y)}{{\sf Var}(X)} \\ \Rightarrow &\quad\alpha^* = \mathbb{E}[Y] - \mathbb{E}[X] \beta^*. \end{align*}\]
With these, the best linear predictor (BLP) is \[\begin{align*} m^*(x) &= \alpha^* + \beta^* x \\ &= \mathbb{E}[Y] + \frac{{\sf Cov}(X,Y)}{{\sf Var}(X)} (x - \mathbb{E}[X])\\ & = \mu_Y + \rho_{XY}\frac{\sigma_Y}{\sigma_X} (x - \mu_X), \end{align*}\] where \(\mu_X = \mathbb{E}[X], \mu_{Y} = \mathbb{E}[Y],\sigma^2_X = {\sf Var}(X), \sigma^2_Y = {\sf Var}(Y)\) and \(\rho_{XY}\) is the Pearson’s correlation.
Interestingly, the MSE under the best linear predictor will be \[\begin{align*} R(\alpha^*,\beta^*) & = \mathbb{E}((Y- \alpha^* - \beta^* X)^2 )\\ & = \mathbb{E}[(Y-\mu_Y - \rho_{XY}\frac{\sigma_Y}{\sigma_X} (X - \mu_X))^2]\\ & = \sigma^2_Y - 2 \rho_{XY}\frac{\sigma_Y}{\sigma_X} \mathbb{E}[(Y-\mu_Y ) (X-\mu_X) ]+ \rho^2_{XY}\frac{\sigma^2_Y}{\sigma^2_X}\sigma^2_X\\ & = \sigma^2_Y(1 - \rho_{XY}^2). \end{align*}\]
A fundamental feature of the preceding analysis is that we did not assume the underlying conditional expectation is truly linear. The best linear predictor is well-defined regardless of the shape of the true regression function.
Suppose that the covariate \(X = (X_1,\cdots, X_p)\) is now multivariate. The linear prediction approach still works and the MSE will be \[R(\alpha, \beta) = \mathbb{E}((Y-\alpha - \beta_1X_1-\beta_2X_2-\cdots-\beta_p X_p)^2),\] where \(\beta = (\beta_1,\cdots, \beta_p)^T\in\mathbb{R}^p\). Let \(Z = (1,X_1,\cdots, X_p)^T\in\mathbb{R}^{p+1}\) be a data vector and \(\gamma = (\alpha, \beta_1,\cdots, \beta_p)^T\in\mathbb{R}^p\) be a coefficient vector. Then the MSE has an elegant form: \[R(\gamma) = R(\alpha,\beta) = \mathbb{E}((Y-\gamma^TZ)^2).\] A direct expansion shows that \[\begin{align*} R(\gamma) & = \mathbb{E}(Y^TY) - 2\mathbb{E}[YZ^T\gamma] +\mathbb{E}[\gamma^TZ Z^T\gamma]\\ &= \mathbb{E}(Y^TY) - 2\gamma^T\mathbb{E}[ZY] + \gamma^T \mathbb{E}[ZZ^T]\gamma, \end{align*}\] which is a quadratic function of \(\gamma\).
Differentiating with respect to the parameter vector \(\gamma\) yields \[0 = -2\mathbb{E}[ZY] + 2 \mathbb{E}[ZZ^T]\gamma.\] Setting the gradient to zero gives the least squares solution: \[\gamma^* = \mathbb{E}[ZZ^T]^{-1} \mathbb{E}[ZY].\] Note that \(\mathbb{E}[ZZ^T]\) is a matrix and \(\mathbb{E}[ZZ^T]^{-1}\) is the matrix inverse.
With \(\gamma^* = (\alpha^*, \beta^*)^T\), we can easily write down the BLP: \[m^*(x) = \gamma^{*T}z= \alpha^* + \beta^{*T}x = \alpha^* + \sum_{j=1}^p \beta^*_j x_j.\]
As previously noted, the best linear predictor does not rely on the assumption of linearity. However, when the true conditional expectation is linear, several desirable properties emerge. We adopt the multivariate notation for generality.
When the linear model is incorrect. The coefficient from the least square approach is \(\gamma^* = \mathbb{E}[ZZ^T]^{-1} \mathbb{E}[ZY].\) In general, this quantity will change if the distribution of the covariate \(X\) (\(Z\)) changes. Thus, the best linear predictor generally depends on the marginal distribution of the covariates.
When the linear model is correct. Suppose that the linear model is correct, i.e., \(Y =\bar \gamma^T Z+ \epsilon\) for some true parameter vector \(\bar \gamma\in\mathbb{R}^{p+1}\), where \(\epsilon\) is a mean-zero error term such that \(\epsilon\perp Z\) and \(\mathbb{E}[\epsilon|Z] = 0\).
Then the coefficient that minimizes the MSE will be \[\begin{align*} \gamma^* &= \mathbb{E}[ZZ^T]^{-1} \mathbb{E}[ZY]\\ & = \mathbb{E}[ZZ^T]^{-1}\mathbb{E}[Z (\bar \gamma^T Z+ \epsilon) ]\\ & = \mathbb{E}[ZZ^T]^{-1}\mathbb{E}[Z (Z^T \bar \gamma+ \mathbb{E}[\epsilon|Z]) ]\\ & = \mathbb{E}[ZZ^T]^{-1}\mathbb{E}[Z Z^T \bar \gamma ]\\ & = \bar \gamma. \end{align*}\] Thus, the least square coefficient is the same as the true coefficient. This also implies that the population least squares coefficient is invariant to the marginal distribution of the covariates when the linear model is correctly specified.
Classification is one of the most important data analysis problems. While early foundational work was developed in statistics, the fields of computer science and machine learning have significantly advanced the state-of-the-art over the past two decades.
Here are some classical applications of classification.
Email spam. Email service providers, such as Google, face the problem of classifying a new email. The problem they want to address is like: given an email, how do you decide if it is a spam, an ordinary email, or an important one?
Image classification. If you have used Facebook, you may notice that whenever your photo contains a picture of one of your friends, Facebook may ask you if you want to tag your friend, even if you did not manually tell the computer that this is your friend. How do they know that picture is a human and that guy is your friend?
We consider a simple scenario – binary classification. Namely, there are only two possible classes that we will consider. We will just denote the two classes as \(0\) and \(1\).
The classification problem can be formalized as follows. Given a feature vector \(x_0\), we want to create a classifier \(c\) that maps \(x_0\) into \(0\) or \(1\). That is, we seek a measurable function \(c(x_0)\) taking values in \(\{0,1\}\). Moreover, we want to make sure that our classification error is small. Let \(y_0\) be the actual label of \(x_0\). We measure the classification error using a loss function \(L\) such that \(L(c(x_0),y_0)\) denotes the penalty for predicting \(c(x_0)\) when the true label is \(y_0\). A ubiquitous choice is the 0-1 loss, defined as \(L(c(x_0),y_0) = I(c(x_0) \neq y_0)\), which penalizes all misclassifications equally.
How do we find the classifier \(c\)? Fortunately, we typically have access to a labeled training sample \((X_1,Y_1),\cdots, (X_n,Y_n)\) available. Then we will find \(c\) using this dataset.
In statistics, we often model the data as an IID random sample from a distribution. We now define several useful distribution functions: \[\begin{equation} \begin{aligned} p_0(x) &= p(X=x|Y=0):\quad \mbox{the density of $X$ when the actual label is $0$},\\ p_1(x) &= p(X=x|Y=1):\quad \mbox{the density of $X$ when the actual label is $1$},\\ P(y|x)& = P(Y=y|X=x):\quad \mbox{the probability of being in the class $y$ when the feature is $x$},\\ P_Y(y)& = P(Y=y):\quad \mbox{the probability of observing the class $y$, regardless of the feature value}. \end{aligned} \label{eq::cl4} \end{equation}\]
Using a probability model, we will define the risk function, which is the expected value of the loss function when the input is random. The risk of a classifier \(c\) is \[R(c) = \mathbb{E}(L(c(X), Y)).\] Ideally, we want to find a classifier that minimizes the risk because such a classifier will minimize our expected losses. In the linear prediction problem, the loss function is the square loss \(L(a,b) = (a-b)^2\) and the resulting risk function is the MSE.
Assume that we know the \(4\) quantities in equation \(\eqref{eq::cl4}\), what class label will you predict when seeing a feature \(X=x\)? An intuitive choice is that we should predict the value \(y\) that maximizes \(P(y|x)\). Namely, we predict the label using the one with highest probability. Such classifier can be written as \[\begin{equation} c_*(x) = {\sf argmax}_{y=0,1} P(y|x) = \begin{cases} 0,\quad \mbox{if $P(0|x)\geq P(1|x)$},\\ 1,\quad \mbox{if $P(1|x)>P(0|x)$}. \end{cases} \label{eq::bayes} \end{equation}\]
Is this classifier good in the sense of the classification error (risk)? The answer depends on the loss function. It turns out that this classifier is optimal under the \(0-1\) loss. Namely, \[R(c_*) = \min_{c} R(c)\] when using a \(0-1\) loss. However, under a different loss function, this classifier may no longer be risk-optimal.
Derivation of \(c_*\) is optimal under \(0-1\) loss. Given a classifier \(c\), the risk function \(R(c) = \mathbb{E}(L(c(X), Y))\). Using the property of expectation, we can further write it as \[R(c) = \mathbb{E}(L(c(X), Y)) = \mathbb{E}(\underbrace{\mathbb{E}(L(c(X),Y)|X)}_{(A)}).\] For the quantity (A), we have \[\begin{align*} \mathbb{E}(L(c(X),Y)|X) &= L(c(X),1)p(Y=1|X) +L(c(X),0)p(Y=0|X) \\ & = I(c(X)\neq 1)p(Y=1|X) + I(c(X)\neq 0)p(Y=0|X)\\ &=\begin{cases} p(Y=1|X) \quad \mbox{ if $c(X)=0$}\\ p(Y=0|X) \quad \mbox{ if $c(X)=1$}. \end{cases} \end{align*}\] Thus, seeing a feature \(X\), the expected loss we have when predicting \(c(X)=0\) is \(P(Y=1|X)\) whereas when prediction \(c(X)=1\) is \(P(Y=0|X)\). The optimal choice is predicting \(c(X)=0\) if \(P(Y=1|X)\leq P(Y=0|X)\) and \(c(X)=1\) if \(P(Y=1|X)>P(Y=0|X)\) (the equality does not matter), which is the classifier \(c_*\).
When a classifier attains the optimal risk (i.e., having a risk of \(\min_{c} R(c)\)), it is called a Bayes classifier. Thus, the classifier \(c_*\) is the Bayes classifier in \(0-1\) loss.
For a classifier \(c\), we define its excess risk (regret) as \[\mathcal{E}(c) = R(c) - \min_c R(c).\] The excess risk quantifies the suboptimality of the classifier \(c\) relative to the Bayes classifier. If we cannot find the Bayes classifier, we will at least try to find a classifier whose excess risk is small.