Empirical Risk Minimization (ERM)

• The ERM principle for supervised learning requires:
  • A loss function $L(y,g(x))$ that measures the discrepancy between the true label $y$ and the predicted label $g(x)$.
  • A hypothesis class $\mathcal{G}$ which is a class of functions $g:\mathcal{X}\to \mathcal{Y}$.
  • A training dataset $(x_1,y_1),\dots ,(x_n,y_n)$.

Loss function

• A loss function $L:\mathcal{Y}\times {\mathbb{R}}^K\to \mathbb{R}$ quantifies how well $\hat{y}$ approximates $y$:
  • smaller values of $L(y,\hat{y}$) indicate that $\hat{y}$ is a good approximation of $y$
  • typically (but not always) $L(y,y)=0$ and $L(y,\hat{y})\ge 0$ for all $\hat{y}$, and $y$
• Examples:
  • Quadratic loss: $L(y,\hat{y})=(y-\hat{y}{)}^2$ or $L(y,\hat{y})=\parallel y-\hat{y}{\parallel }^2$
  • Absolute loss: $L(y,\hat{y})=|y-\hat{y}|$
  • Cross-Entropy loss: $L(y,\hat{y})=-y\log (\hat{y})-(1-y)\log (1-\hat{y})$ or $L(y,\hat{y})=-\sum _{i=1}^K{y}_i\log {\hat{y}}_i$

Risk Function

• Assume that $(X,Y)\sim F$ and $F$ is an unknown distribution.
• Given a loss function ,the risk function of a model $h$ is $$R(h)={\mathbb{E}}_{(X,Y)\sim F}[L(Y,h(X))].$$
• The optimal $h$ is the one that minimizes the risk function: $$h^{\star }=\underset{h:\mathcal{X}\to \mathcal{Y}}{\mathrm{argmin}}R(h).$$
• Denote the optimal risk as $R^{\star }=R(h^{\star })$.
• However, it is impossible to obtain either $h^{\star }$ or $R^{\star }$ because:
  1. the space of all possible functions $\{h:\mathcal{X}\to \mathcal{Y}\}$ is too large, and
  2. the data distribution $F$ is unknown.

Hypothesis Class

• To make the problem tractable, we restrict the space of functions to a hypothesis class $\mathcal{H}$.
• We denote the best model in $\mathcal{H}$ as $h_{\mathcal{H}}^{\star }={\mathrm{argmin}}_{h\in \mathcal{H}}R(h)$.
• Its associated risk is $R_{\mathcal{H}}^{\star }=R(h_{\mathcal{H}}^{\star })$.
• By definition, it is obvious that $R_{\mathcal{H}}^{\star }\ge R^{\star }$.
• Examples:
  • Linear regression: $\mathcal{H}=\{h:h(\mathit{x})={\mathit{x}}^T\mathit{\beta },\mathit{\beta }\in {\mathbb{R}}^p\}={\mathbb{R}}^p$
  • Logistic regression: $\mathcal{H}=\{h:h(\mathit{x})=\frac{1}{1+\exp (-{\mathit{x}}^T\mathit{\beta })},\mathit{\beta }\in {\mathbb{R}}^p\}={\mathbb{R}}^p$
• The difference between $R_{\mathcal{H}}^{\star }$ and $R^{\star }$ is called the approximation error.
• Intuitively, the larger the hypothesis class, the smaller the approximation error.

Empirical Risk

• Assuming that $(x_1,y_1),\dots ,(x_n,y_n)\stackrel{\mathrm{i}\mathrm{i}\mathrm{d}}{\sim }F$, the empirical risk is $$R_{\text{emp}}(h)={\mathbb{E}}_{(X,Y)\sim {\hat{F}}_n}[L(Y,h(X))]=\frac{1}{n}\sum _{i=1}^{n}L(y_i,h(x_i))$$ where ${\hat{F}}_n=\frac{1}{n}\sum _{i=1}^n{\delta }_{(x_i,y_i)}$ is the empirical distribution of the data.
• We choose the $h$ that minimizes the empirical risk function, i.e., the empirical risk minimizer: $${\hat{h}}_{n,\mathcal{H}}=\underset{h\in \mathcal{H}}{\mathrm{argmin}}{R}_{\text{emp}}(h)$$ where $\mathcal{H}$ is the hypothesis class.
• Denote the empirical risk associated with ${\hat{h}}_{n,\mathcal{H}}$ as ${\hat{R}}_n=R_{\text{emp}}({\hat{h}}_{n,\mathcal{H}})$ and this is what we obtain in practice.

Quick Summary

• Goal: find the best model $h^{\star }={\mathrm{argmin}}_{h}R(h)$, which is impossible since
  • the space of all possible functions is too large $\to$ restrict to hypothesis class
  • the data distribution is unknown $\to$ use empirical data
• We have three models:
  • $h^{\star }$: the best model (associated risk $R^{\star }=R(h^{\star })$)
  • $h_{\mathcal{H}}^{\star }$: the best model in the hypothesis class $\mathcal{H}$ (associated risk $R_{\mathcal{H}}^{\star }=R(h_{\mathcal{H}}^{\star })$)
  • ${\hat{h}}_{n,\mathcal{H}}$: the empirical risk minimizer, i.e., the trained model (empirical risk, or the training error, ${\hat{R}}_n=R_{\text{emp}}({\hat{h}}_{n,\mathcal{H}})$)
• We want ${\hat{h}}_{n,\mathcal{H}}$ to be as close as possible to $h^{\star }$ in terms of the risk function $R(h)$.

Error Decomposition

• Goal: $R({\hat{h}}_{n,\mathcal{H}})-R^{\star }=R({\hat{h}}_{n,\mathcal{H}})-R(h^{\star })=0$.
• Decomposition: $$\begin{array}{rl}R({\hat{h}}_{n,\mathcal{H}})-R^{\star }& =\underset{\text{estimation error}}{\underbrace{R({\hat{h}}_{n,\mathcal{H}})-R_{\mathcal{H}}^{\star }}}+\underset{\text{approximation error}}{\underbrace{R_{\mathcal{H}}^{\star }-R^{\star }}}\\ & =\underset{\text{estimation error}}{\underbrace{R({\hat{h}}_{n,\mathcal{H}})-R(h_{\mathcal{H}}^{\star })}}+\underset{\text{approximation error}}{\underbrace{R(h_{\mathcal{H}}^{\star })-R(h^{\star })}}\end{array}$$
• The approximation error comes from the use of a hypothesis class $\mathcal{H}$.
  • Larger $\mathcal{H}$ $\to$ smaller approximation error
• The estimation error comes from the use of empirical data.
  • More data $\to$ smaller estimation error

Example

• Linear Regression:
  • $\mathcal{H}=\{h:h(\mathit{x})={\mathit{x}}^T\mathit{\beta },\mathit{\beta }\in {\mathbb{R}}^p\}$
  • $L(y,h(\mathit{x}))=(y-h(\mathit{x}){)}^2$
• Logistic Regression:
  • $\mathcal{H}=\{h:h(\mathit{x})=\frac{1}{1+\exp (-{\mathit{x}}^T\mathit{\beta })},\mathit{\beta }\in {\mathbb{R}}^p\}$
  • $L(y,h(\mathit{x}))=-y\log (h(\mathit{x}))-(1-y)\log (1-h(\mathit{x}))$
• (Linear) Support Vector Machine:
  • $\mathcal{H}=\{h:h(\mathit{x})={\mathit{w}}^T\mathit{x}+b,\mathit{w}\in {\mathbb{R}}^p,b\in \mathbb{R}\}$
  • $L(y,h(\mathit{x}))=max(0,1-y\cdot h(\mathit{x}))$

Maximum Likelihood (ML) v.s. ERM

• In fact, the ML principle is a special case of the ERM principle.
• That is, specifying a likelihood function gives a loss function, i.e., use the negative log-likelihood as the loss function.
• ML:
  • Stronger assumptions
  • Stronger guarantees (consistency, asymptotic normality, etc.)
  • Allow us to do more things (e.g., hypothesis testing and confidence intervals)
  • Linear regression and logistic regression are ML and hence ERM
• ERM:
  • More flexible and practical, but weaker guarantees
  • Usually provide only a point estimate
  • SVM is ERM but not ML

Constructing Learning Algorithms

• Following the ERM principle, we need to specify a loss function and a hypothesis class in order to construct a learning algorithm.
• The choice of the loss function is based on the types of labels and the problem.
• The choice of the hypothesis class is more challenging:
  • Smaller $\mathcal{H}$ $\to$ larger approximation error, smaller estimation error, and less overfitting
  • Larger $\mathcal{H}$ $\to$ smaller approximation error, larger estimation error, more overfitting, and requires more data
• Next, we will discuss:
  • how to measure the “size” (capacity/complexity) of a hypothesis class
  • how to choose an appropriate hypothesis class

Complexity v.s. Dimension

• Let $\mathcal{H}$ be a parametric hypothesis class ,e.g., $\mathcal{H}=\{h:h(\mathit{x})={\mathit{x}}^T\mathit{\beta },\mathit{\beta }\in {\mathbb{R}}^p\}$.
• An intuitive way to measure the complexity of $\mathcal{H}$ is to count the number of unknown parameters, i.e., the dimension of $\mathcal{H}$.
• In this case, the dimension of $\dim (\mathcal{H})=p$.

Shattering

A hypothesis class $\mathcal{H}$ is said to shatter a set of points $S=\{x_1,\dots ,x_n\}$ if for all possible binary labelings (0/1) of these points, there exists a function $h\in \mathcal{H}$ that can perfectly separate the points.

Shattering

Definition (Restriction of $\mathcal{H}$ to $S$) Let $\mathcal{H}$ be a class of functions from $\mathcal{X}$ to $\{0,1\}$ and let $S=\{x_1,\dots ,x_n\}\subset \mathcal{X}$. The restriction of $\mathcal{H}$ to $S$ is the set of functions from $S$ to $\{0,1\}$ that can be derived from $\mathcal{H}$. That is, $${\mathcal{H}}_S=\{(h(x_1),\dots ,h(x_n)):h\in \mathcal{H}\}$$

Definition (Shattering) A hypothesis class $\mathcal{H}$ shatters a finite set $S\subset \mathcal{X}$ if the restriction of $\mathcal{H}$ to $S$ is the set of all functions from $S$ to $\{0,1\}$. That is, $|{\mathcal{H}}_S|=2^{|S|}$.

Vapnik-Chervonenkis (VC) Dimension

Definition (VC-dimension) The VC-dimension of a hypothesis class $\mathcal{H}$, denoted $\text{VC-dim}(\mathcal{H})$, is the maximal size of a set $S\subset \mathcal{X}$ that can be shattered by $\mathcal{H}$. If $\mathcal{H}$ can shatter sets of arbitrarily large size we say that $\mathcal{H}$ has infinite VC-dimension.

• One can show that for linear models, e.g., $\mathcal{H}=\{h:h(\mathit{x})={\beta }_0+{\mathit{x}}^T\mathit{\beta },\mathit{\beta }\in {\mathbb{R}}^p\}$, the VC-dimension is $p+1$ (the same as the number of parameters).
• However, for nonlinear models, the calculation of the VC-dimension is often challenging.

Example (Infinite VC-dimension)

• Let $\mathcal{H}=\{h:h(x)=\mathbb{I}(\sin (\alpha x)>0),\alpha >0\}$. Then $\text{VC-dim}(\mathcal{H})=\mathrm{\infty }$.
• Proof:
  • For any $n$, let $x_1=2\pi {10}^{-1},\dots ,x_n=2\pi {10}^{-n}$.
  • Then the parameter $\alpha =\frac{1}{2}(1+\sum _{i=1}^n(1-y_i){10}^i)$ can perfectly separate the points, where $y_i\in \{0,1\}$ is any labeling of the points.

Goodness-of-fit v.s. Generalization ability

• Goodness-of-fit: how well the model fits the data.
• Generalization ability: how well the model generalizes to unseen data.
• Recall that for an ERM ${\hat{h}}_{n,\mathcal{H}}$, we have
  • training error (error on training data) ${\hat{R}}_n=R_{\text{emp}}({\hat{h}}_{n,\mathcal{H}})$
  • generalization error (error on unseen data) $R({\hat{h}}_{n,\mathcal{H}})={\mathbb{E}}_{(X,Y)\sim F}[L(Y,{\hat{h}}_{n,\mathcal{H}}(X))\mid \mathcal{T}]$, where $\mathcal{T}$ denotes the training dataset.
• We can write $$\begin{array}{rl}R({\hat{h}}_{n,\mathcal{H}})& ={\hat{R}}_n+(R({\hat{h}}_{n,\mathcal{H}})-{\hat{R}}_n)\\ \text{generalization error}& =\text{training error}+\text{generalization gap}\end{array}$$

Overfitting and Underfitting

• To have low generalization error, we need to have both low training error and small generalization gap.
  • Large training error $\to$ underfitting
  • Low training error but large generalization gap $\to$ overfitting

VC Inequality

• The VC theory provides an upper bound for the generalization gap, known as the VC inequality: with probability at least $1-\delta$ $$R(h)\le R_{\text{emp}}(h)+\epsilon \sqrt{1+\frac{4R_{\text{emp}}(h)}{\epsilon }},\epsilon =O\left(\frac{d-\log \delta }{n}\right)$$ simultaneously for all $h\in \mathcal{H}$, where $\text{VC-dim}(\mathcal{H})=d<\mathrm{\infty }$.
• The generalization gap increases as
  • the VC-dimension increases
  • the samples size $n$ decreases
• This upper bound is only a loose bound and does not work for models with infinite VC-dimension.

Regularized ERM

• To prevent overfitting, we can add a regularization term to the empirical risk: $$R_{\text{reg}}(h)=R_{\text{emp}}(h)+\lambda \mathrm{\Omega }(h)$$ where $\mathrm{\Omega }(h)$ is a regularization term and $\lambda$ is the regularization parameter.
• Typically, $\mathrm{\Omega }(h)$ measures the smoothness or complexity of the model $h$.

Regularized ERM

• For example, $\mathrm{\Omega }(h)=\parallel h^{\prime }(x){\parallel }_2^2=\int {(h^{\prime }(x))}^{2}dx$. ($L_2$ Regularization)
• If $h(x)={\beta }_0+{\beta }_{1}x$, then $h^{\prime }(x)={\beta }_1$ and $\mathrm{\Omega }(h)={\beta }_1^2$.
• The $L_1$ regularization is $\mathrm{\Omega }(h)=\parallel h^{\prime }(x){\parallel }_1=\int |h^{\prime }(x)|dx$.
• If $h(x)={\beta }_0+{\beta }_{1}x$, then $h^{\prime }(x)={\beta }_1$ and $\mathrm{\Omega }(h)=|{\beta }_1|$.
• Using $L_1$ gives you sparsity; using $L_2$ gives you smoothness/insensitivity:
  • Consider a linear model $h(\mathit{x})={\mathit{x}}^T\mathit{\beta }$.
  • A good model should not be too sensitive to the input, i.e., small changes in the input should not lead to large changes in the output.
  • That is, if $\mathit{x}\approx \tilde{\mathit{x}}$ , then $|{\mathit{x}}^T\mathit{\beta }-{\tilde{\mathit{x}}}^T\mathit{\beta }|$ should be small.
  • Note that $$|{\mathit{x}}^T\mathit{\beta }-{\tilde{\mathit{x}}}^T\mathit{\beta }|=|(\mathit{x}-\tilde{\mathit{x}}{)}^T\mathit{\beta }|\le \parallel \mathit{x}-\tilde{\mathit{x}}{\parallel }_2\parallel \mathit{\beta }{\parallel }_2$$

Bias-Variance Tradeoff

• Adding a reguralization term often increases the bias but reduces the variance of the model.
• This tradeoff is known as the bias-variance tradeoff.

Quick Summary

• The VC dimension measures the complexity of a hypothesis class.
• The VC inequality provides an upper bound for the generalization gap, provided that the VC dimension is finite.
• The bound is often criticized for being too loose and does not work for models with infinite VC dimension.
• Example of infinite VC dimension:
  • Neural Networks
  • Kernel methods (e.g., kernel SVM, kernel regression)
  • $K$-nearest neighbors (with small $K$, say $K=1$)

Double Descent Curve

Estimating the Generalization Error

• Although the VC inequality provides an upper bound for the generalization gap, it is often too loose.
• In order to have a more accurate insight into the model’s generalization ability, we need to estimate the generalization error.
• To achieve this, we need to have an extra dataset, called the validation dataset $\mathcal{V}=\{({\tilde{x}}_i,{\tilde{y}}_i){\}}_{i=1}^m$.
• The generalization error is then estimated as $${\hat{R}}_{\text{gen}}=\frac{1}{m}\sum _{i=1}^{m}L({\tilde{y}}_i,{\hat{h}}_{n,\mathcal{H}}({\tilde{x}}_i)).$$
• Assuming the $\mathcal{V}$ and $\mathcal{T}$ (training dataset) are i.i.d, ${\hat{R}}_{\text{gen}}$ is an unbiased estimate of the generalization error, i.e., $\mathbb{E}[{\hat{R}}_{\text{gen}}\mid \mathcal{T}]=R({\hat{h}}_{n,\mathcal{H}})$.

$k$-fold Cross-Validation (CV)

• In practice, we often do not have an extra validation dataset and hence we need to use the training dataset to estimate the generalization error.
• One common method is the $k$-fold cross-validation:
  1. Split the training dataset $\mathcal{T}$ into $k$ equal-sized folds.
  2. For each fold $i=1,\dots ,k$, train the model on the remaining $k-1$ folds and evaluate the model on the $i$th fold.
  3. Average the $k$ validation errors to obtain the estimated generalization error.

$k$-fold Cross-Validation

• When $k=n$, it is called the leave-one-out cross-validation (LOOCV), i.e., train the model on $n-1$ samples and evaluate on the remaining one.
• Choice of $k$?
  • Larger $k$ $\to$ low bias, high variance (the model is trained on a larger dataset and validated on a smaller dataset)
  • Smaller $k$ $\to$ high bias, low variance (the model is trained on a smaller dataset and validated on a larger dataset)
  • $k=5$ or $k=10$ are common choices.

CV for Hyperparameter Tuning

• In practice, the models often have hyperparameters that need to be tuned, e.g., the regularization parameter $\lambda$.
• We can use CV to choose the best hyperparameters:
  1. For each hyperparameter value, perform $k$-fold CV to estimate the generalization error.
  2. Choose the hyperparameter value that minimizes the CV error.
• However, the CV error after the selection will overestimate the generalization error.
• Such bias is known as the selection bias.

CV for Hyperparameter Tuning

• To avoid the selection bias, we first split the dataset into two parts: the training dataset and the test dataset.
• The test dataset should not be used in the neither the traing process nor hyperparameter tuning process.
• The training dataset is further split into $k$-folds for CV.
• After all the processes, including training, hyperparameter tuning, model selection, etc., we evaluate the final model on the test dataset to estimate the generalization error.

Example: Using CV to Choose the Regularization Parameter

from sklearn.linear_model import LassoCV
from sklearn import datasets

X, y = datasets.load_diabetes(return_X_y=True)

X /= X.std(axis=0)

alpha_seq = np.logspace(-2, 2, 100)
reg = LassoCV(alphas = alpha_seq, cv = 5, random_state = 42)
reg.fit(X, y)

print("best alpha:", np.round(reg.alpha_, 4))

best alpha: 0.0774

Example: Using CV to Choose the Regularization Parameter

Proof of Example (Infinite VC-dimension)

• We need to show that the model $h(x)=\mathbb{I}(\sin (\alpha x)>0)$ with $\alpha =\frac{1}{2}(1+\sum _{i=1}^n(1-y_i){10}^i)$ can perfectly separate the $n$ points.
• Consider the $j$th sample $x_j=2\pi {10}^{-j}$.
• If $y_j=0$, then $$\begin{array}{rl}\alpha x_j& =\pi {10}^{-j}(1+\sum _{i=1}^n(1-y_i){10}^i)=\pi {10}^{-j}(1+\sum _{\{i:y_i=0\}}{10}^i)\\ & =\pi {10}^{-j}(1+{10}^j+\sum _{\{i:y_i=0,i\ne j\}}{10}^i)\\ & =\pi ({10}^{-j}+1+\sum _{\{i:y_i=0,i>j\}}{10}^{i-j}+\sum _{\{i:y_i=0,i<j\}}{10}^{i-j})\end{array}$$

Proof of Example (Infinite VC-dimension)

• For $i>j$, ${10}^{i-j}$ is even and so is $\sum _{\{i:y_i=0,i>j\}}{10}^{i-j}$, say $$\sum _{\{i:y_i=0,i>j\}}{10}^{i-j}=2m,m\in \mathbb{N}.$$
• Note that $$\sum _{\{i:y_i=0,i<j\}}{10}^{i-j}<\sum _{i=1}^{\mathrm{\infty }}{10}^{-i}=\sum _{i=0}^{\mathrm{\infty }}{10}^{-i}-1=\frac{1}{1-0.1}-1=\frac{1}{9}.$$
• Therefore, $\alpha x_j=\pi (1+2m+ϵ)$, where $$0<ϵ={10}^{-j}+\sum _{\{i:y_i=0,i<j\}}{10}^{i-j}<\frac{1}{10}+\frac{1}{9}<1.$$
• Hence $\sin (\alpha x_j)<0$ and $h(x_j)=0$.

Proof of Example (Infinite VC-dimension)

• If $y_j=1$, then $$\begin{array}{rl}\alpha x_j& =\pi {10}^{-j}(1+\sum _{i=1}^n(1-y_i){10}^i)=\pi {10}^{-j}(1+\sum _{\{i:y_i=0\}}{10}^i)\\ & =\pi ({10}^{-j}+\sum _{\{i:y_i=0,i>j\}}{10}^{i-j}+\sum _{\{i:y_i=0,i<j\}}{10}^{i-j})\end{array}$$
• Similarly, we have $\alpha x_j=\pi (2m+ϵ)$, where $$0<ϵ={10}^{-j}+\sum _{\{i:y_i=0,i<j\}}{10}^{i-j}<\frac{1}{10}+\frac{1}{9}<1.$$
• Hence $\sin (\alpha x_j)>0$ and $h(x_j)=1$.