Classification is one of the most important data analysis problems. Much early work on this topic was done by statisticians but in the past 20 years, computer science and machine learning communities have made much more progress on this topic.
Here are some classical applications of classification.
Email spam. Email service providers such as Google often face the problem of classifying a new email. The problem they want to address is like: given an email, how do you decide if it is a spam, an ordinary email, or an important one?
Image classification. If you have used Facebook, you may notice that whenever your photo contains a picture of one of your friends, Facebook may ask you if you want to tag your friend, even if you did not manually tell the computer that this is your friend. How do they know that picture is a human and that guy is your friend?
We consider a simple scenario – binary classification. Namely, there are only two possible classes that we will consider. We will just denote the two classes as \(0\) and \(1\).
The classification problem can be formalized as follows. Given a feature vector \(x_0\), we want to create a classifier \(c\) that maps \(x_0\) into \(0\) or \(1\). Namely, we want to find a function \(c(x_0)\) that outputs only two possible numbers: \(0\) and \(1\). Moreover, we want to make sure that our classification error is small. Let \(y_0\) be the actual label of \(x_0\). We measure the classification error using a loss function \(L\) such that the loss of making a prediction \(c(x_0)\) when the actual class label is \(y_0\) is \(L(c(x_0),y_0)\). A common loss function is the 0-1 loss, which is \(L(c(x_0),y_0) = I(c(x_0) \neq y_0)\). Namely, when we make a wrong classification, we lose \(1\) point and we do not lose anything if we make the correct classification.
How do we find the classifier \(c\)? The good news is: often we have a labeled sample (data) \((X_1,Y_1),\cdots, (X_n,Y_n)\) available. Then we will find \(c\) using this dataset.
In statistics, we often model the data as an IID random sample from a distribution. We now define several useful distribution functions: \[\begin{equation} \begin{aligned} p_0(x) &= p(X=x|Y=0):\quad \mbox{the density of $X$ when the actual label is $0$},\\ p_1(x) &= p(X=x|Y=1):\quad \mbox{the density of $X$ when the actual label is $1$},\\ P(y|x)& = P(Y=y|X=x):\quad \mbox{the probability of being in the class $y$ when the feature is $x$},\\ P_Y(y)& = P(Y=y):\quad \mbox{the probability of observing the class $y$, regardless of the feature value}. \end{aligned} \label{eq::cl4} \end{equation}\]
Using a probability model, we will define the risk function, which is the expected value of the loss function when the input is random. The risk of a classifier \(c\) is \[R(c) = \mathbb{E}(L(c(X), Y)).\] Ideally, we want to find a classifier that minimizes the risk because such a classifier will minimize our expected losses.
Assume that we know the \(4\) quantities in equation [eq::cl4], what class label will you predict when seeing a feature \(X=x\)? An intuitive choice is that we should predict the value \(y\) that maximizes \(P(y|x)\). Namely, we predict the label using the one with highest probability. Such a classifier can be written as \[\begin{equation} c_*(x) = {\sf argmax}_{y=0,1} P(y|x) = \begin{cases} 0,\quad \mbox{if $P(0|x)\geq P(1|x)$},\\ 1,\quad \mbox{if $P(1|x)>P(0|x)$}. \end{cases} \label{eq::bayes} \end{equation}\]
Is this classifier good in the sense of the classification error (risk)? The answer depends on the loss function. The good news is: this classifier is the optimal classifier for the \(0-1\) loss. Namely, \[R(c_*) = \min_{c} R(c)\] when using a \(0-1\) loss. However, if we are using other loss functions, this classifier will not be the best one (with the smallest expected loss).
Derivation of \(c_*\) is optimal under \(0-1\) loss. Given a classifier \(c\), the risk function \(R(c) = \mathbb{E}(L(c(X), Y))\). Using the property of expectation, we can further write it as \[R(c) = \mathbb{E}(L(c(X), Y)) = \mathbb{E}(\underbrace{\mathbb{E}(L(c(X),Y)|X)}_{(A)}).\] For the quantity (A), we have \[\begin{align*} \mathbb{E}(L(c(X),Y)|X) &= L(c(X),1)p(Y=1|X) +L(c(X),0)p(Y=0|X) \\ & = I(c(X)\neq 1)p(Y=1|X) + I(c(X)\neq 0)p(Y=0|X)\\ &=\begin{cases} p(Y=1|X) \quad \mbox{ if $c(X)=0$}\\ p(Y=0|X) \quad \mbox{ if $c(X)=1$}. \end{cases} \end{align*}\] Thus, seeing a feature \(X\), the expected loss we have when predicting \(c(X)=0\) is \(P(Y=1|X)\) whereas when prediction \(c(X)=1\) is \(P(Y=0|X)\). The optimal choice is predicting \(c(X)=0\) if \(P(Y=1|X)\leq P(Y=0|X)\) and \(c(X)=1\) if \(P(Y=1|X)>P(Y=0|X)\) (the equality does not matter), which is the classifier \(c_*\).
When a classifier attains the optimal risk (i.e., having a risk of \(\min_{c} R(c)\)), it is called a Bayes classifier. Thus, the classifier \(c_*\) is the Bayes classifier in \(0-1\) loss.
For a classifier \(c\), we define its excess risk (regret) as \[\mathcal{E}(c) = R(c) - \min_c R(c).\] The excess risk is a quantity that measures how the quality of \(c\) is away from the optimal/Bayes classifier. If we cannot find the Bayes classifier, we will at least try to find a classifier whose excess risk is small.
If we know \(P(y|x)\), we can build the Bayes classifier and this classifier is the optimal one in terms of the risk function. However, \(P(y|x)\) is a population quantity, which is often unknown to us. All we have is a random sample \((X_1,Y_1),\cdots, (X_n,Y_n)\). So the question becomes: how do we estimate \(P(y|x)\) using the data?
It turns out that this is a problem we know how to solve. Here is just one hint: because the response variable \(Y\) only takes two possible values \(\{0,1\}\), it is actually a Bernoulli random variable! Thus, \(\mathbb{E}(Y) = P(Y = 1)\), which implies \[\begin{equation} \mathbb{E}(Y|X=x) = P(Y=1|X=x) = P(1|x). \label{eq::reg} \end{equation}\] Namely, \(P(1|x)\) is the regression function! Using the fact that \(P(0|x)+ P(1|x) =1\), an estimator of \(P(1|x)\) leads to an estimator of \(P(y|x)\) for both \(y=0\) and \(y=1\).
Thus, as long as we have a regression estimator, we can convert it into a classifier. Here is one example of using kernel regression. Let \[\widehat{m}_K(x) = \frac{\sum_{i=1}^n Y_i K\left(\frac{X_i-x}{h}\right)}{\sum_{i=1}^n K\left(\frac{X_i-x}{h}\right)}\] be the kernel regression. Then \[\widehat{P}_K(1|x) = \widehat{m}_K(x),\quad \widehat{P}_K(0|x) = 1-\widehat{m}_K(x).\] Thus, a classifier based on kernel regression is \[\begin{align*} \widehat{c}_K(x) &= \begin{cases} 0,\quad \mbox{if $\widehat P_K(0|x)\geq\widehat P_K(1|x)$},\\ 1,\quad \mbox{if $\widehat P_K(1|x)>\widehat P_K(0|x)$} \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $1-\widehat P_K(1|x)\geq\widehat P_K(1|x)$},\\ 1,\quad \mbox{if $\widehat P_K(1|x)>1-\widehat P_K(1|x)$}. \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $\widehat P_K(1|x)\leq\frac{1}{2}$},\\ 1,\quad \mbox{if $\widehat P_K(1|x)>\frac{1}{2}$}. \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $\widehat{m}_K(x)\leq\frac{1}{2}$},\\ 1,\quad \mbox{if $\widehat{m}_K(x)>\frac{1}{2}$}. \end{cases} \end{align*}\] Namely, the classifier will output \(1\) whenever the estimated regression function is greater than half and \(0\) otherwise.
Will this classifier be a good one? Intuitively, it should be true. If we have a good regression estimator, the corresponding classifier should also be good. In fact, we have the following powerful result linking the quality of a regression estimator and the excess risk.
Theorem 1. Assume we use the \(0-1\) loss. Let \(\widehat{m}\) be a regression estimator and \(\widehat{c}_m\) be the corresponding classifier. Then \[\mathcal{E}(\widehat{c}_m) \leq 2 \int |\widehat{m}(x)-m(x)|dP(x) \leq 2 \sqrt{\int |\widehat{m}(x)-m(x)|^2 dP(x)}.\]
Namely, if we have a regression estimator whose overall quality is good, the corresponding classifier will also have a small excess risk (i.e., perform comparably well compared to the optimal classifier).
In addition to using a regression function to construct a classifier, we can use a density estimator for classification. This approach is often known as the naive Bayes approach.
A key insight is from the Bayes’ rule: \[P(y|x) = P(Y=y|X=x) = \frac{p(x,y)}{p(x)} = \frac{p(x|y)P(Y=y)}{p(x)} = \frac{p_y(x)P_Y(y)}{p(x)},\] where \(p(x) = \sum_y p(x,y) = p(x,0)+p(x,1) = p_0(x)P_Y(0) + p_1(x)P_Y(1)\). Thus, the Bayes classifier can be written as \[\begin{align*} c_*(x) &= \begin{cases} 0,\quad \mbox{if $P(0|x)\geq P(1|x)$}\\ 1,\quad \mbox{if $P(1|x)>P(0|x)$} \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $\frac{p_0(x)P_Y(0)}{p(x)}\geq\frac{p_1(x)P_Y(1)}{p(x)}$}\\ 1,\quad \mbox{if $\frac{p_1(x)P_Y(1)}{p(x)}>\frac{p_0(x)P_Y(0)}{p(x)}$} \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $p_0(x)P_Y(0)\geq p_1(x)P_Y(1)$}\\ 1,\quad \mbox{if $p_1(x)P_Y(1)>p_0(x)P_Y(0)$} \end{cases}. \end{align*}\] Thus, if we can estimate \(p_0(x),p_1(x),\) and \(P_Y(y)\), we can construct a classifier.
\(P_Y(y)\) is very easy to estimate. It is the probability of seeing an observation with label \(y\). As a result, a simple estimator is to use the ratio of observations with this label. Namely, \[\widehat{P}_Y(y) = \frac{1}{n}\sum_{i=1}^n I(Y_i=y).\]
\(p_y(x)\) is just the conditional density of \(X\) given the label being \(y\). Thus, we can simply apply a density estimator to those observations with a class label \(y\).
Example: kernel density estimator. Using a kernel density estimator (KDE), we obtain \[\widehat{p}_{y,kde}(x) = \frac{1}{n_yh} \sum_{i=1}^n I(Y_i=y) K\left(\frac{X_i-x}{h}\right),\] where \(n_y = \sum_{i=1}^n I(Y_i=y)\) is the number of observations with label being \(y\). Note that \(\widehat{P}_Y(y) = \frac{n_y}{n}\). Thus, a classifier based on a KDE is \[\begin{align*} \widehat{c}_{\sf KDE}(x) &= \begin{cases} 0,\quad \mbox{if $\widehat{p}_{0,kde}(x)\widehat{P}_Y(0)\geq \widehat{p}_{1,kde}(x)\widehat{P}_Y(1)$}\\ 1,\quad \mbox{if $\widehat{p}_{1,kde}(x)\widehat{P}_Y(1)>\widehat{p}_{0,kde}(x)\widehat{P}_Y(0)$} \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $\sum_{i=1}^n I(Y_i=0) K\left(\frac{X_i-x}{h}\right)\geq \sum_{i=1}^n I(Y_i=1) K\left(\frac{X_i-x}{h}\right)$}\\ 1,\quad \mbox{if $\sum_{i=1}^n I(Y_i=1) K\left(\frac{X_i-x}{h}\right)>\sum_{i=1}^n I(Y_i=0) K\left(\frac{X_i-x}{h}\right)$} \end{cases}. \end{align*}\] The classifier \(\widehat{c}_{\sf KDE}(x)\) is also called the kernel classifier.
Example: density basis approach. We can use the basis approach as well. Assume that we consider \(M\) basis and we use the cosine basis \(\{\phi_1(x),\phi_2(x),\cdots\}\). The estimator of \(p_y(x)\) will be the density estimator using only those observations with a label \(y\). Let \[\widehat{\theta}_{y,\ell} = \frac{1}{n_y} \sum_{i=1}^n I(Y_i=y) \phi_\ell(X_i)\] be the estimator of the \(\ell\)-th coefficient using only those observations with a label \(y\). The corresponding density estimator is \[\begin{align*} \widehat{p}_{y,M}(x) = \sum_{\ell=1}^M \widehat{\theta}_{y,\ell} \phi_\ell(x) = \sum_{\ell=1}^M\frac{1}{n_y} \sum_{i=1}^n I(Y_i=y) \phi_\ell(X_i)\phi_\ell(x) = \frac{1}{n_y} \sum_{i=1}^n I(Y_i=y) \sum_{\ell=1}^M\phi_\ell(X_i)\phi_\ell(x). \end{align*}\] Thus, the corresponding classifier is \[\begin{align*} \widehat{c}_{\sf M}(x) &= \begin{cases} 0,\quad \mbox{if $\widehat{p}_{0,M}(x)\widehat{P}_Y(0)\geq \widehat{p}_{1,M}(x)\widehat{P}_Y(1)$}\\ 1,\quad \mbox{if $\widehat{p}_{1,M}(x)\widehat{P}_Y(1)>\widehat{p}_{0,M}(x)\widehat{P}_Y(0)$} \end{cases}\\ &= \begin{cases} 0,\quad \mbox{if $\sum_{i=1}^n I(Y_i=0) \sum_{\ell=1}^M\phi_\ell(X_i)\phi_\ell(x)\geq \sum_{i=1}^n I(Y_i=1) \sum_{\ell=1}^M\phi_\ell(X_i)\phi_\ell(x)$}\\ 1,\quad \mbox{if $\sum_{i=1}^n I(Y_i=1) \sum_{\ell=1}^M\phi_\ell(X_i)\phi_\ell(x)>\sum_{i=1}^n I(Y_i=0) \sum_{\ell=1}^M\phi_\ell(X_i)\phi_\ell(x)$} \end{cases}. \end{align*}\]
Given a classifier and a set of labeled data, we can illustrate the quality of classification using a confusion matrix. In binary classification, a confusion matrix is a \(2\times 2\) matrix (you can view it as a contingency table) as follows:
| Actual label: 0 | Actual label: 1 | |
|---|---|---|
| Predicted label: 0 | \(n_{00}\) | \(n_{01}\) |
| Predicted label: 1 | \(n_{10}\) | \(n_{11}\) |
\(n_{ij}\) is the number of instances/observations where the predicted label is \(i\) and actual label is \(j\).
The quantity \[\frac{n_{10}+{n_{01}}}{n_{00}+n_{01}+n_{10}+n_{11}},\] is called the misclassification rate and is an empirical estimate of the risk of the classifier.
If the class label \(0\) stands for ‘normal case’ while the label \(1\) stands for ‘anomaly’, then we can interpret the confusion matrix as
| Actual label: 0 | Actual label: 1 | |
|---|---|---|
| Predicted label: 0 | True negative | False negative |
| Predicted label: 1 | False positive | True positive |
This interpretation is commonly used in engineering problems and medical research for detecting abnormal situations.
The \(k\)-NN approach can be applied to classification as well. The idea is very simple – for a given point \(x_0\), we find its \(k\) nearest data points. Then we compare the labels of these \(k\) points and assign the label of \(x_0\) as the majority label in these \(k\) points.
Take the data in the following picture as an example. There are two classes: black dots and red crosses. We are interested in the class label at the two blue boxes (\(x_1\) and \(x_2\)).
Assume we use a \(3\)-NN classifier \(\widehat{c}_{3-NN}\). At point \(x_1\), its \(3\)-NN contains two black dots and one red cross, so \(\widehat{c}_{3-NN}(x_1) =\) black dot. At point \(x_2\), its \(3\)-NN has one black dot and two red crosses, so \(\widehat{c}_{3-NN}(x_2) =\) red cross. Note that if there are ties, we will randomly assign the class label attaining the tie.
The \(k\)-NN approach is simple and easy to operate. It can be easily generalized to multiple classes – the idea is the same: we assign the class label according to the majority in the neighborhood.
How do we choose \(k\)? The choice of \(k\) is often done by a technique called cross-validation. The basic principle is: we split the data into two parts, use one part to train our classifier and evaluate the performance on the other part. Repeating the above procedure multiple times and applying it to each \(k\), we can obtain an estimate of the performance. We then choose the \(k\) with the best performance. We will talk about this topic later.
The logistic regression is a regression model that is commonly applied to classification problems as well. Like the method of converting a regression estimator to a classifier, logistic regression uses a regression function as an intermediate step and then forms a classifier. We first talk about some interesting examples.
Example. In graduate school admission, we are wondering how a student’s GPA affects the chance that this applicant received the admission. In this case, each observations is a student and the response variable \(Y\) represents whether the student received admission (\(Y=1\)) or not (\(Y=0\)). GPA is the covariate \(X\). Thus, we can model the probability \[P({\sf admitted}|{\sf GPA}=x) = P(Y=1|X=x) = q(x).\]
Example. In medical research, people are often wondering if the heretability of the type-2 diabetes is related to some mutation from of a gene. Researchers record if the subject has the type-2 diabetes (response) and measure the mutation signature of genes (covariate \(X\)). Thus, the response variable \(Y=1\) if this subject has the type-2 diabetes. A statistical model to associate the covariate \(X\) and the response \(Y\) is through \[P(\mbox{subject has type-2 diabetes}|\mbox{mutation signature}=x) = P(Y=1|X=x) = q(x).\]
Thus, the function \(q(x)\) now plays a key role in determining how the response \(Y\) and the covariate \(X\) are associated. The logistic regression provides a simple and elegant way to characterize the function \(q(x)\) in a ‘linear’ way. Because \(q(x)\) represents a probability, it ranges within \([0,1]\) so naively using a linear regression will not work. However, consider the following quantity: \[O(x) = \frac{q(x)}{1-q(x)} = \frac{P(Y=1|X=x)}{P(Y=0|X=x)} \in [0,\infty).\] The quantity \(O(x)\) is called the odds that measures the contrast between the event \(Y=1\) versus \(Y=0\). When the odds is greater than \(1\), we have a higher change of getting \(Y=1\) than \(Y=0\). The odds has an interesting asymmetric form– if \(P(Y=1|X=x) = 2P(Y=0|X=x)\), then \(O(x) = 2\) but if \(P(Y=0|X=x) = 2P(Y=1|X=x)\), then \(O(x) = \frac{1}{2}\). To symmetrize the odds, a straight-forward approach is to take (natural) logarithm of it: \[\log O(x) = \log \frac{q(x)}{1-q(x)}.\] This quantity is called log odds. The log odds has several beautiful properties, for instance when the two probabilities are the same (\(P(Y=1|X=x) = P(Y=0|X=x)\)), \(\log O(x) = 0\), and \[\begin{align*} P(Y=1|X=x) = 2P(Y=0|X=x) &\Rightarrow \log O(x) = \log 2\\ P(Y=0|X=x) = 2P(Y=1|X=x) &\Rightarrow \log O(x) = -\log 2. \end{align*}\]
The logistic regression is to impose a linear model to the log odds. Namely, the logistic regression models \[\log O(x) = \log \frac{q(x)}{1-q(x)} = \beta_0 +\beta^Tx,\] which leads to \[P(Y=1|X=x) =q(x) = \frac{e^{\beta_0 +\beta^Tx}}{1+e^{\beta_0 +\beta^T x}} .\] Thus, the quantity \(q(x) = q(x; \beta_0,\beta)\) depends on the two parameters \(\beta_0,\beta\). Here \(\beta_0\) behaves like the intercept and \(\beta\) behaves like the slope vector (they are the intercept and slope in terms of the log odds).
When we observe data, how can we estimate these two parameters? In general, we will use the maximum likelihood approach to estimate them. You can view the (minus) likelihood function as the loss function in the classification (actually, we will use the log-likelihood function as the loss function). And the goal is to find the parameters via minimizing such a loss.
Recall that we observe a random sample: \[(X_1, Y_1),\cdots, (X_n, Y_n).\] Let \(p_X(x)\) denote the probability density of \(X\); note that we will not use it in estimating \(\beta_0,\beta\). For a given pair \(X_i,Y_i\), recall that the random variable \(Y_i\) given \(X_i\) is just a Bernoulli random variable with parameter \(q(x=X_i)\). Thus, the PMF of \(Y_i\) given \(X_i\) is \[\begin{align*} \mathcal{L}(\beta_0,\beta|X_i,Y_i) =P(Y=Y_i|X_i) &= q(X_i)^{Y_i}(1-q(X_i))^{1-Y_i} \\ &= \left(\frac{e^{\beta_0 +\beta^T X_i}}{1+e^{\beta_0 +\beta^TX_i}}\right)^{Y_i} \left(\frac{1}{1+e^{\beta_0 +\beta^T X_i}}\right)^{1-Y_i}\\ & = \frac{e^{\beta_0Y_i +\beta^T X_iY_i}}{1+e^{\beta_0 +\beta^T X_i}}. \end{align*}\] Note that here we construct the likelihood function using only the conditional PMF because similarly to linear regression, the distribution of the covariate \(X\) does not depend on the parameters \(\beta_0,\beta\). Thus, the log-likelihood function is \[\begin{align*} \ell (\beta_0,\beta|X_1,Y_1,\cdots,X_n,Y_n) &= \sum_{i=1}^n \log \mathcal{L}(\beta_0,\beta^T|X_i,Y_i)\\ &= \sum_{i=1}^n \log\left(\frac{e^{\beta_0Y_i +\beta^T X_iY_i}}{1+e^{\beta_0 +\beta^T X_i}}\right)\\ &= \sum_{i=1}^n \beta_0Y_i +\beta^T X_iY_i - \log\left(1+e^{\beta_0 +\beta^T X_i}\right). \end{align*}\] Our estimates are \[\begin{align*} \widehat{\beta}_0, \widehat{\beta} &= \underset{\beta_0,\beta}{\sf argmax}\,\,\ell (\beta_0,\beta|X_1,Y_1,\cdots,X_n,Y_n) \\ &= \underset{\beta_0,\beta}{\sf argmin}\,\, - \ell (\beta_0,\beta|X_1,Y_1,\cdots,X_n,Y_n)\\ & = \underset{\beta_0,\beta}{\sf argmin}\,\, \underbrace{\frac{1}{n}\sum_{i=1}^n \underbrace{- \ell (\beta_0,\beta|X_i,Y_i)}_\text{loss function}}_\text{empirical estimate of the loss function}, \end{align*}\] where the loss function is \[- \ell (\beta_0,\beta|X_i,Y_i) = \beta_0Y_i +\beta^T X_iY_i - \log\left(1+e^{\beta_0 +\beta^T X_i}\right). %= L_{\sf logistic}(c_{\beta_0,\beta}(X_i),Y_i).\]
\(\widehat{\beta}_0, \widehat{\beta}\) does not have a closed-form solution in general so we cannot write down a simple expression of the estimator. Despite this disadvantage, such a log-likelihood function can be optimized by a gradient ascent approach such as the Newton-Raphson1.
We can apply the basis approach to logistic regression. In fact, modern deep neural nets for classification tasks are mostly based on this idea. Let \(\phi(x)\in\mathbb{R}^p\) be a vector-valued basis functions such that \[\{\phi_1(x),\cdots ,\phi_p(x)\}\] is a basis. For simplicity, we assume that \(\phi_1(x) = 1\) so that the first parameter represents the intercept.
The basis logistic regression is very straightforward. We model the log-odds: \[\log O(x) = \beta^T \phi(x).\] This implies that \[P(Y=1|X=x) = \frac{e^{\beta^T\phi(x)}}{1+e^{\beta^T\phi(x)}}\] and the log-likelihood is \[\ell(\beta|x,y) = \beta^T \phi(x)y + \log\left(1+e^{\beta^T \phi(x)}\right).\] With the data \((X_i,Y_i)\), we then compute \[\widehat\beta = {\sf argmax}_\beta \quad \frac{1}{n}\sum_{i=1}^n \ell(\beta|X_i,Y_i).\]
While this basis logistic regression is very straightforward, it is a very flexible framework as many modern state-of-the-art classification methods are basis approach: orthornormal basis, spline basis, radial basis, autoencoder basis, neural nets, ... etc.
Decision tree is another common approach in classification. A decision tree is like a regression tree – we partition the space of covariate into rectangular regions and then assign each region a class label. If the tree is given, the class label at each region is determined by the majority vote (using the majority of labels in that region).
The following picture provides an example of a decision tree using the same data as the one in the previous section.
In the left panel, we display the scatterplot and regions separated by a decision tree. The background color denotes the estimated label. The right panel displays the tree structure.
In the case of binary classification (the class label \(Y=0\) or \(1\)), the decision tree can be written as follows. Let \((X_1,Y_1),\cdots, (X_n,Y_n)\) denotes the data and \(R_1,\cdots, R_k\) is a rectangular partition of the space of the covariates. Let \(R(x)\) denotes the rectangular regions where \(x\) falls within. The decision tree is \[\widehat{c}_{DT}(x) = I\left(\frac{\sum_{i=1}^n Y_i I(X_i\in R(x))}{\sum_{i=1}^n I(X_i\in R(x))}> \frac{1}{2}\right).\] Here, you can see that the decision tree is essentially a classifier converted from a regression tree.
A modification of the decision tree is called the Random Forests, which is a bagging approach. The idea of bagging is as follows. Suppose that we have one method of constructing a classifier. Given the original data set, we perform the bootstrap–sampling with replacement from the original data– to generate several bootstrap samples. Suppose that we have generated \(B\) bootstrap samples. If we train the classifier on each of the \(B\) bootstrap sample, we obtain a classifier. Thus, by training the classifier independently over each bootstrap sample, we obtain \(\widehat c_1,\cdots, \widehat c_B\), \(B\) classifiers. We then combine these \(B\) classifiers to form our final classifier using the majority vote, i.e,, \[\widetilde{c}(x) = \begin{cases} 1,\quad\mbox{if $\frac{1}{B}\sum_{b=1}^B\widehat c_b(x)\geq \frac{1}{2}$}\\ 0,\quad \mbox{otherwise} \end{cases}.\] The formal name of bagging is bootstrap aggregation.
The random forest is the bagging decision tree with one additional modification– when training each classifier, instead of using all features, we only use a randomly selected subset of features. Informally, you can say
Random Forest = Decision Tree + Bagging + Random features.
There will be three tuning parameters in training a random forest: the number of leaves in the decision tree \(k\), the bootstrap sample number \(B\), and the number of randomly selected features \(d_0\). Note that sometimes instead of using the bootstrap to generate a bootstrap sample, people would use subsampling method that randomly select only a fraction of the original sample in the bagging stage (each bagging sample has different subsets of the original data). If we use subsampling method, there will be one additional tuning parameter–the size of subsample.
The random forest can be applied to not only classification problem but also the regression problem. In the regression problem, the bagging stage will average the regression function from each bootstrap sample and the averaged value will be the final regression estimator. Namely, \[\widetilde{m}(x) = \frac{1}{B}\sum_{b=1}^B \widehat m_b(x),\] where \(\widehat m_1(x),\cdots, \widehat m_B(x)\) are the regression tree estimator of each bootstrap sample.
A useful tutorial summarizing recent advances in the theoretical behaviors of random forest can be found in
Biau, G., & Scornet, E. (2016). A random forest guided tour. Test, 25(2), 197-227.
One useful theoretical result on random forest is the following theorem:
Theorem 2 (Biau 2012). Suppose the regression function \(m\) is Lipschitz and only depends on a subset of features \(S\subset \{1,\cdots,d\}\) and the probability of selecting a feature \(j\in S\) is \(\frac{1}{\|S\|}(1+o(1))\), where \(\|S\|\) is the cardinality of \(S\). Then when the number of leaves \(k_n \asymp n^{4S\log 2/(4S\log 2+3)}\), \[\mathbb{E}(|\widetilde m(X)- m(X)|^2) = O\left(\frac{1}{n}\right)^{\frac{3}{4\|S\|\log 2+3}}.\]
This is from
Biau, G. (2012). Analysis of a random forests model. Journal of Machine Learning Research, 13(Apr), 1063-1095.
When using a subsampling method in training the random forest, here is an improved theoretical result that also regularizes the tuning parameters under the additive model.
Theorem 3 (Scornet, Biau and Vert (2015)). Suppose that \(Y= \sum_{j} m_k(X_j)+\epsilon\) where \(X\sim {\sf Uni}[0,1]^d, \epsilon \sim N(0,\sigma^2)\) and each \(m_j\) is continuous. Assume that the split is chosen using the maximum drop in sums of squares. Let \(t_n\) be the number of leaves on each tree and \(a_n\) be the size of each subsample. If \(t_n,a_n\rightarrow \infty\) and \(\frac{t_n \log^9 a_n}{a_n}\rightarrow0\) then \[\mathbb{E}(|\widetilde m(X)- m(X)|^2) \rightarrow 0.\]
The above result is from
Scornet, E., Biau, G., & Vert, J. P. (2015). Consistency of random forests. The Annals of Statistics, 43(4), 1716-1741.
Random forest is also related to the kNN approach; see the following paper
Lin, Y. and Jeon, Y. (2006). Random Forests and Adaptive Nearest Neighbors. Journal of the American Statistical Association, 101, p 578.
It is possible to use the random forest to perform statistical inference(e.g., hypothesis test and/or confidence intervals). See the following two papers:
Mentch, L., & Hooker, G. (2014). Ensemble trees and clts: Statistical inference for supervised learning. stat, 1050, 25.
Wager, S., & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228-1242.
Recently, there is a new approach that combines random forest and kernel regression. For a point \(x\), let \(A_{j}(x)\) be the cell such that \(x\) falls in \(j\)-th regression tree. The \(b\)-th regression tree estimator can be written as \[\widehat m_b(x) =\frac{\sum_{i=1}^n Y_i I(X_i \in A_b(x))}{\sum_{i=1}^n I(X_i \in A_b(x))}.\] Thus, the random forest regression function is \[\begin{align*} \widetilde m(x) &= \frac{1}{B}\sum_{b=1}^B\widehat m_b(x)\\ & = \frac{1}{B}\sum_{b=1}^B\sum_{i=1}^n\frac{ I(X_i \in A_b(x))}{\sum_{\ell=1}^n I(X_\ell \in A_b(x))}Y_i\\ & = \frac{1}{B}\sum_{i=1}^n\sum_{b=1}^B\frac{ I(X_i \in A_b(x))}{N_b(x)}Y_i\\ & =\frac{1}{B} \sum_{i=1}^n \sum_{b=1}^B W_{bi} Y_i\\ & = \sum_{i=1}^nW_i Y_i\\ \end{align*}\] where and \(N_b(x) = \sum_{i=1}^n I(X_i\in A_b(x))\) is the total number observations of \(b\)-th sample in the cell \(A_b(x)\) and \(W_{bi} = \frac{ I(X_i \in A_b(x))}{N_b(x)}\), \(W_{i} = \sum_{b=1}^B W_{bi}/B\) are both weights.
Notice that the weight \(W_{bi} \propto I(X_i\in A_b(x))\). This motivates us to construct a kernel function that is proportional to \(\sum_{b=1}^B I(X_i\in A_b(x))\), which leads to \[\widehat m_{\sf KeRF}(x) = \frac{\sum_{i=1}^n Y_i K(x,X_i)}{\sum_{\ell=1}^n K(x,X_\ell)},\] where \[K(x,y) = \frac{1}{B}\sum_{b=1}^B I(y\in A_b(x)).\] This method is called kernel based random forest (KeRF). Note that in KeRF, the kernel function is a random function. Its randomness can be described by the IID random regions \[A_1,\cdots, A_B\sim G,\] where \(G\) is a distribution of random regions. Although it looks complicated, the randomness of each \(A_b\) is determined by 1. the bootstrap sample and 2. the randomly chosen features. So \(G\) is not so difficult to analyze.
Warning! The KeRF is different from the random forest although the construction of KeRF is motivated by random forest.
More details about KeRF can be found in the following paper:
Scornet E. Random forests and kernel methods. (2016). IEEE Transactions on Information Theory. 62(3):1485-500.
Consider the binary classification problem that our observations are \[(X_1,Y_1),\cdots, (X_n,Y_n)\] such that \(X_i\in \mathbb{R}^d\) and \(Y_i\) are binary. Instead of using binary representation of the two classes, we use \(Y_i \in \{+1, -1\}\) to indicate the two classes.
SVM is one of the most popular classification tools that machine learning experts were using before the invention of deep neural nets. SVM is motivated by constructing the ‘best’ linear classifier that separates two classes when the two classes are indeed separable by a linear classifier (in this case, we call the problem linearly separable). Two classes are linearly separable if there exists one hyperplane \(H(x) = b^Tx +a\) such that the sign of \(H\) perfectly separate the two classes, i.e, \[Y_i \cdot H(X_i) = Y_i (b^TX_i+a)>0\] for all \(i=1,\cdots, n\).
Given a linear classifier \(H(x) = b^Tx+a\), the distance from observation \((X_i,Y_i)\) to the decision boundary \(H(x) = 0 = b^Tx+a\) is \[\frac{1}{\|b\|} Y_i(b^TX_i+a).\] The margin – minimal distance to the decision boundary – is \[\frac{1}{\|b\|} \min_{i}Y_i(b^TX_i+a).\] The SVM attempts to find the classifier that maximizes the margin, i.e., the SVM tries to solve \[\begin{equation} {\sf argmax}_{b,a}\quad\frac{1}{\|b\|} \min_{i}Y_i(b^TX_i+a). \label{eq::SVM1} \end{equation}\]
Because the classifier is unchanged when we rescale \((a,b)\) by \((\kappa a, \kappa b)\), we add a constraint \[\begin{equation} Y_i(b^TX_i+a) \geq 1\quad \text{for all $i=1,\cdots, n$} \label{eq::margin1} \end{equation}\] and there exists at least one observation such that the equality holds. Thus, the maximization of the margin becomes maximizing \(\frac{1}{\|b\|}\) subject to the constraint in equation [eq::margin1] so the problem in equation [eq::SVM1] is equivalent to solving \[\begin{equation*} {\sf argmax}_{b,a}\quad\frac{1}{\|b\|} \quad \text{subject to equation \ref{eq::margin1}} %\label{eq::SVM2} \end{equation*}\] or equivalently, \[\begin{equation} {\sf argmin}_{b,a}\quad\frac{1}{2}\|b\|^2\quad \text{subject to equation \ref{eq::margin1}}. \label{eq::SVM2} \end{equation}\] Note that we add the additional \(\frac{1}{2}\) to simplify later derivations.
Using Lagrangian multipliers, we can derive the dual form of the problem in equation [eq::SVM2] as minimizing the following Lagrangian: \[\begin{equation} L(a,b,\lambda) = \frac{1}{2}\|b\|^2 - \sum_{i=1}^n \lambda_i [Y_i(b^TX_i+a)-1], \label{eq::SVM2_2} \end{equation}\] where \(\lambda_1,\cdots, \lambda_n\) are the Lagrangian multiplier. To simplify the problem, we consider taking derivatives of \(L\) with respect to \(a\) and \(b\) first, which leads to \[\begin{equation} \begin{aligned} \frac{\partial L}{\partial b}= 0&\Rightarrow b = \sum_{i=1}^n \lambda_i Y_iX_i\\ \frac{\partial L}{\partial a}= 0&\Rightarrow 0 = \sum_{i=1}^n\lambda_iY_i \end{aligned}. \label{eq::SVM3} \end{equation}\] Plugging the above two equations into equation \(\ref{eq::SVM2_2}\), we obtain \[\begin{equation} \begin{aligned} L_1(\lambda) &= \frac{1}{2}\| \sum_{i=1}^n \lambda_i Y_iX_i\|^2 - \sum_{i=1}^n \lambda_j [Y_j(( \sum_{i=1}^n \lambda_i Y_iX_i)^TX_j+a)-1]\\ & = \frac{1}{2}\| \sum_{i=1}^n \lambda_i Y_iX_i\|^2 - \| \sum_{i=1}^n \lambda_i Y_iX_i\|^2 + \sum_{i=1}^n \lambda_i\\ & = \sum_{i=1}^n \lambda_i - \frac{1}{2}\sum_{i=1}^n \sum_{j=1}^n \lambda_i\lambda_jY_iY_jX_i^TX_j\\ & = \sum_{i=1}^n \lambda_i - \frac{1}{2}\sum_{i=1}^n \sum_{j=1}^n \lambda_i\lambda_jY_iY_j\langle X_i,X_j\rangle, \end{aligned} \label{eq::SVM4} \end{equation}\] where we use \(\langle \cdot,\cdot\rangle\) to denote inner product. Thus, the SVM problem becomes the above quadratic minimization problem– we will find \(\lambda_1,\cdots,\lambda_n\) such that \(L_1(\lambda)\) is minimized.
It turns out that many \(\lambda_i\)’s will have a value of \(0\). From Lagrangian multiplier theory, we know that when \(\lambda_i=0\), the constraint \(Y_i(b^TX_i+a)-1= 0\) does not hold, i.e, the pair \((X_i,Y_i)\) is not the observation that is closest to the decision boundary (so they are not on the margin). Those observation with \(\lambda_i>0\) are the ones that determines the margin so we will call them the support vectors.
The above analysis is under the assumption that the data is linearly separable. When the data is not linearly separable, the constraint in equation \(\ref{eq::margin1}\) \[Y_i(b^TX_i+a) \geq 1\] is too strict. We have to relax it.
One approach to relax such constraint is to introduce a set of slack variables \(\xi_1,\cdots, \xi_n\geq 0\) and replace the above constraint by \[\begin{equation} Y_i(b^TX_i+a) \geq 1-\xi_i,\quad \xi_i\geq 0, \quad \text{for all $i=1,\cdots, n$}. \label{eq::margin1c} \end{equation}\] If we attempt to optimize equation [eq::SVM2] with respect to the above constraint, we can always choose \(\xi_i\) to be as large as possible and it is bad for solving the problem. To avoid choosing a large value of \(\xi_i\)’s, we modify the optimization problem \[\begin{equation} {\sf argmin}_{b,a,\xi}\quad C\sum_{i=1}^n \xi_i+\frac{1}{2}\|b\|^2\quad \text{subject to equation \ref{eq::margin1c}}, \label{eq::SVM2_2c} \end{equation}\] where \(C>0\) behaves like a penalization parameter (later we will show this). After solving this problem, the value of \(\xi_i\) falls into three interesting regimes \[\begin{equation} \xi_i\begin{cases} = 0,\quad &\mbox{if $i$-th observation is correctly classified and away from the boundary},\\ \in (0,1],\quad &\mbox{if $i$-th observation is correctly classified but close to the boundary},\\ >1,\quad &\mbox{if $i$-th observation is mis-classified}. \end{cases} \end{equation}\]
Solving equation \(\ref{eq::SVM2_2c}\) is also a challenging task so we consider the dual problem of it. Using Lagrangian multipliers, we obtain the Lagrangian form of it as \[\begin{equation} L_3(a,b,\lambda,\xi,\mu) = C\sum_{i=1}^n \xi_i+ \frac{1}{2}\|b\|^2 - \sum_{i=1}^n \lambda_i [Y_i(b^TX_i+a)-1-\xi_i] - \sum_{i=1}^n \mu_i \xi_i, \label{eq::SVM2c} \end{equation}\] where \(\lambda\) and \(\mu\) are Lagrangian multipliers. Again, to simplify the problem we take derivatives, which leads to \[\begin{equation} \begin{aligned} \frac{\partial L}{\partial b}= 0&\Rightarrow b = \sum_{i=1}^n \lambda_i Y_iX_i\\ \frac{\partial L}{\partial a}= 0&\Rightarrow 0 = \sum_{i=1}^n\lambda_iY_i\\ \frac{\partial L}{\partial \xi_i}= 0&\Rightarrow C = \lambda_i + \mu_i \end{aligned}. \label{eq::SVM3c} \end{equation}\] Among all of them, the last constraint is very informative– because \(\mu_i\geq 0\), we obtain a constraint \(\lambda_i\leq C\).
Plugging equation \(\ref{eq::SVM3c}\) into equation \(\ref{eq::SVM2c}\), we obtain an interesting result: \[\begin{equation} L_4(\lambda,\mu) = \sum_{i=1}^n\lambda_i -\frac{1}{2}\sum_{i=1}^n\sum_{j=1}^n \lambda_i\lambda_j Y_iY_j \langle X_i, X_j\rangle. \label{eq::SVM4c} \end{equation}\] Namely, the objective function is the same as the linearly separable case. But the minimization of equation \(\ref{eq::SVM4c}\) has a slightly different constraints: \[\begin{equation} 0\leq \lambda_i\leq C,\quad \mbox{for all $i=1,\cdots n$}. \label{eq::SVM5c} \end{equation}\] The upper bound on \(\lambda_i\)’s is due to the penalty on the slack variables in the objective function.
Another way to view the SVM is to study the slack variable \(\xi_i\). We can rearrange equation \(\ref{eq::margin1c}\) to obtain \[\xi_i \geq 1- Y_i(b^TX_i+a),\quad \xi_i \geq 0,\] which can be re-expressed as \[\xi_i \geq [1- Y_i(b^TX_i+a)]_{+},\] where \([x]_{+} = \max\{x,0\}\). Since in equation \(\ref{eq::SVM2_2c}\), we are trying to minimize \(\xi_i\), it is easy to see that at the optima, we will always choose \[\xi_i = [1- Y_i(b^TX_i+a)]_{+}.\] Thus, equation \(\ref{eq::SVM2_2c}\) can be rewritten as \[\begin{equation} \begin{aligned} {\sf argmin}_{b,a,\xi}&\quad C\sum_{i=1}^n {[1- Y_i(b^TX_i+a)]_{+}}+\frac{1}{2}\|b\|^2\\ \Leftrightarrow{\sf argmin}_{b,a,\xi}&\quad \sum_{i=1}^n {[1- Y_i(b^TX_i+a)]_{+}}+\frac{1}{2C}\|b\|^2. \end{aligned} \label{eq::SVM6} \end{equation}\] The quantity \([1- Y_i(b^TX_i+a)]_{+}\) is called the hinge loss \(L(a,b) = (1-ab)_{+}\). So the SVM is to minimize a regularized hinge loss with squared penalty.
The hinge loss has a nice interpretation when we code the binary outcome \(Y\in\{-1,1\}\). Let \(m_\theta(x)\in\mathbb{R}\) be a model and our decision rule is \(c_\theta(x) = 1\) if \(m_\theta(x)>0\) and \(-1\) otherwise. One can see that \(m_\theta(x)\) can be viewed as a model on the expectation \(\mathbb{E}(Y|X=x)\) but it does not have to be. The value of \(|m_\theta(x)|\) can be interpreted as our ‘model’s confidence’ about the label.
The hinge loss \(L(m_\theta(X), Y) = (1- Y\cdot m_\theta(X))_{+}\) has two ranges. \(L(m_\theta(X), Y) =0\) when \(Y\cdot m_\theta(X)\geq 1\). Namely, when our confidence \(|m_\theta(x)|\) is high and the label (sign) is correct, the loss is \(0\), i.e., no loss. When we make a mistake in the sign, i.e., \({\sf Sign}(m_\theta(x))\neq Y\), the loss is always above \(1\) and increases when our confidence is higher.
The interesting part is the case when the loss is between \([0,1]\). This occurs only if \({\sf Sign}(m_\theta(x))= Y\), i.e., we get the class label correct, but \(Y\cdot m_\theta(X) <1\), namely, our confidence is low although we get the label correct. Therefore, the minimization of hinge loss means that we not only want to get the label correct, we also want to have high confidence on our prediction.
Remark (KKT conditions). In the above analysis, we are moving between a constrained form and its dual Lagrangian form. One sufficient condition that makes the optima from the two problems agree is that the Karush-Kuhn-Tucker (KKT) conditions hold. It is a very important topic in optimization and I would highly recommend you to look for related materials2.
Even without any probabilistic model, a classification problem can be viewed as an optimization problem. The key element is to replace the risk function \(\mathbb{E}(L(c(X), Y))\) by the empirical risk (training error) \[\widehat R_n(c) = \frac{1}{n}\sum_{i=1}^n L(c(X_i), Y_i).\] Consider a collection of classifiers \(\mathcal{C}\). Then the goal is to find the best classifier \(c^*\in \mathcal{C}\) such that the empirical risk \(\widehat R_n(c)\) is minimized. Namely, \[c^* = \underset{c\in\mathcal{C}}{\sf argmin}\,\, \widehat R_n(c).\] Ideally, this should work because \[\begin{equation} \mathbb{E}(\widehat R_n(c)) = \mathbb{E}\left(\frac{1}{n}\sum_{i=1}^n L(c(X_i), Y_i)\right) = \mathbb{E}(L(c(X_1),Y_1)) = R(c) \label{eq::empirical_risk} \end{equation}\] is the actual risk function. Namely, \(\widehat R_n(c)\) is an unbiased estimator of the risk function.
Here are some concrete examples.
Linear classifiers. Assume that we consider a linear classifier: \[c_{\beta_0,\beta}(x) = I(\beta_0 + \beta^T x > 0),\] where \(\beta_0\) is a number like the intercept and \(\beta\) is the vector like the slope of each covariate/feature. Namely, how we assign the class label purely depends on the value of \(\beta_0 + \beta^T x\). If this value is positive, then we give it a label \(1\). If the value is negative, then we give it a label \(0\). Then the set \[\mathcal{C}_{\sf lin} = \{c_{\beta_0, \beta}: \beta_0\in \mathbb{R}, \beta\in \mathbb{R}^d\}\] is a collection of all linear classifiers. The idea of empirical risk minimization is to find the classifier in \(\mathcal{C}_{\sf lin}\) such that the empirical risk \(\widehat R_n(\cdot)\) is minimized. Because every classifier is indexed by the two quantities (parameters) \(\beta_0,\beta\), finding the one that minimizes the empirical risk is equivalent to finding the best \(\beta_0,\beta\) minimizing \(\widehat R_n(\cdot)\).
Logistic regression. Similar to the linear classifier, the logistic regression can be viewed as a classifier of the form \[\widetilde{c}_{\beta_0,\beta}(x) = I\left(\frac{e^{\beta_0 + \beta^T x}}{1+e^{\beta_0 + \beta^T x}} > \frac{1}{2}\right).\] Then we can define the set \[\mathcal{C}_{\sf logistic} = \{\widetilde{c}_{\beta_0, \beta}: \beta_0\in \mathbb{R}, \beta\in \mathbb{R}^d\}\] as the collection of all classifiers from a logistic regression model. The MLE approach becomes an empirical risk minimization method using a particular loss function – the log-likelihood function.
Decision tree (fixed \(k\), fixed leaves). The decision tree classifier \(\widehat{c}_{\sf DT}\) can be viewed as a classifier from the empirical risk minimization as well. For simplicity, we assume that the regions/leaves of the decision trees \(R_1,\cdots, R_k\) are fixed. Then any decision tree classifier can be written as \[c_{\sf DT}(x) = \sum_{j=1}^k \alpha_j I(x\in R_j),\] where \(\alpha_1,\cdots, \alpha_k\) are quantities/parameters that determine how we will predict the class label of region \(R_1,\cdots, R_k\), respectively. And the collection of all possible classifiers will be \[\mathcal{C}_{\sf DT} = \{c_{\sf DT}(x): \alpha_j \in \{0,1\}, j =1,\cdots, k\}.\] Note that there are only \(2^k\) classifiers in the set \(\mathcal{C}_{\sf DT}\). The estimator \(\widehat{c}_{\sf DT}\) is just the one that minimizes the empirical loss \(\widehat R_n(\cdot)\) with a \(0-1\) loss.
Decision tree (fixed \(k\), non-fixed leaves). When the regions/leaves are not fixed, the collection of all possible decision tree classifiers is much more complex. Here is an abstract way of describing such a collection. Recall that at each split of the tree, we pick one region and one feature and a threshold (e.g., \(x_1 > 10\) versus \(x_1\leq 10\)), so every split can be represented by three indices: the region we are splitting, the feature index (where this split occurs), and the threshold index (what is the split level). Because at the \(\ell\)-th split, there are only \(\ell\) regions, the \(\ell\)-th split is characterized by a triplet \((\omega, m, \lambda)\), where \(\omega = \{1,2,\cdots, \ell\}\) and \(m \in \{1,2,\cdots, d\}\) and \(\lambda \in \mathbb{R}\). If a tree has \(k\) leaves, there will be \(k-1\) splits. So any decision tree classifier is indexed by \[\alpha_1,\cdots, \alpha_k, (\omega_1, m_1,\lambda_1),\cdots, (\omega_{k-1},m_{k-1},\lambda_{k-1}).\] Namely, given a set of these values, we can construct a unique decision tree classifier. Thus, the collection of all decision trees with \(k\) leaves can be written as \[\begin{align*} \mathcal{C}_{\sf DT}(k) = \{c_{\sf DT}(x) &= c_{\alpha,\omega, m,\lambda}(x): \alpha_j \in \{0,1\},\\ &(\omega_\ell, m_\ell,\lambda_\ell)\in\{1,\cdots,\ell\} \times\{1,2,\cdots, d\} \times \mathbb{R},\\ & j =1,\cdots, k, \ell=1,\cdots, k-1\}. \end{align*}\] In reality, when we train a decision tree classifier with a fixed number \(k\), the regions are also computed from the data. Thus, we are actually finding \(\widehat{c}_{\sf DT}\) such that \[\widehat{c}_{\sf DT} = \underset{c\in\mathcal{C}_{\sf DT}(k)}{\sf argmin}\,\, \widehat R_n(c).\]
Decision tree (both \(k\) and leaves are non-fixed). If we train the classifier with \(k\) being unspecified, then we are finding \(\widehat{c}_{\sf DT}\) from \(\bigcup_{k\in \mathbb{N}}\mathcal{C}_{\sf DT}(k)\) that minimizes the empirical risk. However, if we really consider all possible \(k\), such an optimal decision tree is not unique and may be problematic. When \(k>n\), we can make each leaf contain at most 1 observation and the predicted label is just the label of that observation. Such a classifier has \(0\) empirical risk but it may have very poor performance in future prediction because we are overfitting the data. Overfitting the data implies that \(\widehat{R}_n(c)\) and \(R(c)\) are very different, even if \(\widehat{R}_n(c)\) is an unbiased estimator of \(R(c)\)!
Why will this happen? \(\widehat{R}_n(c)\) is just the sample-average version of \(R(c)\), right? Does this contradict the Law of Large Numbersssss that \(\widehat{R}_n(c)\) converges to \(R(c)\)?
It is true that \(\widehat{R}_n(c)\) is an unbiased estimator of \(R(c)\) and yes indeed the Law of Large Numbersssss is applicable in this case. BUT a key requirement for using the Law of Large Numbersssss is that we assume \(c\) is fixed. Namely, if the classifier \(c\) is fixed, then the Law of Large Numbersssss guarantees that the empirical risk \(\widehat{R}_n(c)\) converges to the true risk function \(R(c)\).
However, when we are finding the best classifier, we consider many many many possible classifiers \(c\). Although for a given classifier \(c\) the Law of Large Numbersssss works, it may not work when we consider many classifiers. The empirical risk minimization works if \[\sup_{c\in\mathcal{C}}\left|\widehat{R}_n(c) - R(c)\right| \overset{P}{\rightarrow} 0\] Namely, the convergence is uniform for all classifiers in the collection that we are considering. In the next few lectures we will be talking about how the above uniform convergence may be established.
Remark.
Regression problems. The empirical risk minimization method can also be applied to regression problems. We just replace the classifier by the regression function and the loss function can be chosen as the \(L_2\) loss (squared distance). In this formulation, we obtain \[R(m) = \mathbb{E}\left(\|Y- m(X)\|^2\right)\] and \[\widehat{R}_n(m) = \frac{1}{n}\sum_{i=1}^n \|Y_i-m(X_i)\|^2.\] Estimating a regression function can thus be written as an empirical risk minimization – we minimize \(\widehat{R}_n(m)\) for \(m\in\mathcal{M}\) to obtain our regression estimator. The set \(\mathcal{M}\) is a collection of many regression functions. Similar to the classification problem, we need uniform convergence of the empirical risk to make sure we have a good regression estimator.
Penalty function. Another approach to handle the difference \(\sup_{c\in \mathcal{C}}|\widehat{R}_n(c)- R(c)|\) is to add an extra quantity to \(\widehat R_n(c)\) so that such a uniform difference is somewhat controlled. Instead of minimizing \(\widehat R_n(c)\), we minimize \[\widehat{R}_n(c) + \mathcal{P}_\lambda(c),\] where \(\mathcal{P}_\lambda\) is a penalty function. This is just the penalized regression approach but now applying it to a classification problem. When the penalty function is chosen in a good way, we can make sure the optimal classifier/regression estimator from the (penalized) empirical risk minimization indeed has a very small risk.
some references can be found: https://www.cs.princeton.edu/~bee/courses/lec/lec_jan24.pdf↩︎
A good reference: https://www.cs.cmu.edu/~ggordon/10725-F12/slides/16-kkt.pdf↩︎