Midterm Proposal presentation

• A group of 3-4 students.
• Let me know your group members next Tuesday.
• The presentation will be on 10/22.
  • Problem statement
  • Data description
  • 1-2 references
• Each group has 15 minutes to present.
• Homework 1 is due on 10/15.

Recap of the Last Lecture

A general procedure for supervised learning:

1. Gather a dataset \(\mathcal{D} = \{(\boldsymbol{x}_1, y_1), \ldots, (\boldsymbol{x}_n, y_n)\}\).
2. Choose an appropriate loss function \(L\) for your dataset and problem.
3. Choose a hypothesis class \(\mathcal{H}\).
   • Not too simple (underfitting), nor too complicated (overfitting).
4. Add a regularization term to the loss function to better control the complexity of the model.
5. Find the model that minimizes the (regularized) empirical risk function.
6. Use cross-validation to select the hyperparameters (if any).
7. Estimate the generalization error of the final model using the test dataset.

Nonlinear Hypothesis Class

• A simple and useful hypothesis class is the linear hypothesis class, for example:
  • Linear regression: \(\mathcal{H} = \{h(\boldsymbol{x}) = \boldsymbol{x}^T\boldsymbol{\beta}, \boldsymbol{\beta} \in \R^p\}\)
  • Logistic regression: \(\mathcal{H} = \left\{h: h(\boldsymbol{x}) = \frac{1}{1 + \exp(-\boldsymbol{x}^T \boldsymbol{\beta})}, \boldsymbol{\beta} \in \R^p\right\} = \R^p\)
• To extend linear models to represent nonlinear functions of \(\boldsymbol{x}\), we can apply the linear model not to \(\boldsymbol{x}\) itself but to a transformed input \(\phi(\boldsymbol{x})\), where \(\phi\) is a nonlinear transformation.
• That is, \[ \mathcal{H} = \{ h(\boldsymbol{x}) = \phi(\boldsymbol{x})^T \boldsymbol{\beta}, \phi: \R^p \to \R^d, \boldsymbol{\beta}\in \R^d\}. \]
• The function \(\phi\) is usually called a feature map and usually we choose \(d \gg p\).

Choice of Feature Map

There are three common ways to choose the feature map \(\phi\):

1. Kernel method
   • Instead of explicitly specifying \(\phi\), we can define a symmetric function \(k(\boldsymbol{x}, \boldsymbol{x}')\) called a kernel, which corresponds to the dot product of some feature map \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = \phi(\boldsymbol{x})^T \phi(\boldsymbol{x}^{\prime})\).
2. Manual engineering (hand-crafted features)
   • Choose a feature map \(\phi\) manually, e.g., \(\phi(x) = [x, x^2, \exp(x), \sin(x)]\).
   • A good feature map requires human effort for each separate task, with practitioners specializing in different domains.
3. Deep Learning
   • Parametrize the feature map \(\phi\) with a deep neural network, \(\phi(\boldsymbol{x}) = \phi(\boldsymbol{x}; \boldsymbol{\theta})\).
   • The parameter \(\boldsymbol{\theta}\) is learned from the data, i.e., we learn the feature map from the data.

Outline

• Kernel Methods
  • Kernel Ridge Regression
• Multilayer Perceptron (MLP)
  • Basic Structure
  • Activation Function
• Training an MLP
  • Gradient-based Learning
  • Back-propagation
  • Computational Graph
• Example

Regression with a Feature Map

• Consider the regression model with a given feature map \(\phi: \R^p \to \R^d\), i.e., \[ y = f(\boldsymbol{x}) = \phi(\boldsymbol{x})^T \boldsymbol{\beta}. \]
• Given a training dataset \(\{(\boldsymbol{x}_i, y_i)\}_{i=1}^n\), let \(\boldsymbol{z_i} = \phi(\boldsymbol{x}_i)\) and \(\boldsymbol{Z} = [\boldsymbol{z}_1, \boldsymbol{z}_2, \ldots, \boldsymbol{z}_n]^T \in \R^{n \times d}\).
• Then the model parameter \(\boldsymbol{\beta}\) can be estimated by minimizing the regularized empirical risk function: \[\begin{align*} \hat{\boldsymbol{\beta}} & = \argmin_{\boldsymbol{\beta} \in \R^d} \|\boldsymbol{y} - \boldsymbol{Z}\boldsymbol{\beta}\|^2 + \alpha \|\boldsymbol{\beta}\|^2_2 = (\boldsymbol{Z}^T \boldsymbol{Z} + \alpha I_d)^{-1} \boldsymbol{Z}^T \boldsymbol{y}\\ & \stackrel{\textcolor{red}{\text{(*)}}}{=} \boldsymbol{Z}^T (\boldsymbol{Z} \boldsymbol{Z}^T + \alpha I_n)^{-1} \boldsymbol{y}. \end{align*}\]
• The prediction for a new input \(\boldsymbol{x}^{\star}\) is \[ \widehat{y^{\star}} = \phi(\boldsymbol{x}^{\star})^T \hat{\boldsymbol{\beta}} = \phi(\boldsymbol{x}^{\star})^T\boldsymbol{Z}^T (\boldsymbol{Z} \boldsymbol{Z}^T + \alpha I_n)^{-1} \boldsymbol{y}. \]

Regression with a Feature Map

• Note that \[ \boldsymbol{Z} \boldsymbol{Z}^T = \begin{bmatrix} \boldsymbol{z}_1^T \\ \boldsymbol{z}_2^T \\ \vdots \\ \boldsymbol{z}_n^T \end{bmatrix} \begin{bmatrix} \boldsymbol{z}_1 & \boldsymbol{z}_2 & \cdots & \boldsymbol{z}_n \end{bmatrix} = \begin{bmatrix} \boldsymbol{z}_1^T \boldsymbol{z}_1 & \boldsymbol{z}_1^T \boldsymbol{z}_2 & \cdots & \boldsymbol{z}_1^T \boldsymbol{z}_n \\ \boldsymbol{z}_2^T \boldsymbol{z}_1 & \boldsymbol{z}_2^T \boldsymbol{z}_2 & \cdots & \boldsymbol{z}_2^T \boldsymbol{z}_n \\ \vdots & \vdots & \ddots & \vdots \\ \boldsymbol{z}_n^T \boldsymbol{z}_1 & \boldsymbol{z}_n^T \boldsymbol{z}_2 & \cdots & \boldsymbol{z}_n^T \boldsymbol{z}_n \end{bmatrix} = \left[\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j)\right]_{i,j=1}^n \] and \[ \phi(\boldsymbol{x}^{\star})^T\boldsymbol{Z}^T = \begin{bmatrix} \phi(\boldsymbol{x}^{\star})^T \boldsymbol{z}_1 & \phi(\boldsymbol{x}^{\star})^T \boldsymbol{z}_2 & \cdots & \phi(\boldsymbol{x}^{\star})^T \boldsymbol{z}_n \end{bmatrix} = \left[\phi(\boldsymbol{x}^{\star})^T\phi(\boldsymbol{x}_i)\right]_{i=1}^n. \]
• BIG NEWS: we don’t need to know the feature map \(\phi\) explicitly, but only its inner product \(\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j)\).
• That is, if we can find a function \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime})\) such that \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = \phi(\boldsymbol{x})^T\phi(\boldsymbol{x}^{\prime})\) for some \(\phi\), then we don’t need to know \(\phi\) explicitly.
• The function \(k\) is called a kernel function and the model is called a kernel ridge regression.
• Mercer Condition: If the function \(k(\cdot, \cdot)\) is symmetric and positive definite, then there exist a \(\phi\) such that \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = \phi(\boldsymbol{x})^T\phi(\boldsymbol{x}^{\prime})\).

Kernel Method vs Deep Learning

• Both kernel method and deep learning utilize nonlinear feature map to obtain a nonlinear hypothesis class.
• Kernel method use a kernel function (the inner product of the feature map) to implicitly define the feature map. Some common kernel functions include:
  • Linear kernel: \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = \boldsymbol{x}^T\boldsymbol{x}^{\prime} \Rightarrow \phi(\boldsymbol{x}) = \boldsymbol{x}\).
  • Polynomial kernel: \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = (\boldsymbol{x}^T\boldsymbol{x}^{\prime} + c)^d\).
  • Radial basis function (RBF) kernel: \(k(\boldsymbol{x}, \boldsymbol{x}^{\prime}) = \exp\left(-\gamma\|\boldsymbol{x} - \boldsymbol{x}^{\prime}\|^2\right)\).¹
• Deep learning uses a deep neural network to parametrize the feature map.

Example: Kernel Regression

Generate a dataset from the model \(y = \sin(2\pi x) + x + \epsilon\), where \(\epsilon \sim N(0, 0.5^2)\).

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(100)
n = 100
X = rng.uniform(low=0, high=3, size=n)
y = np.sin(2 * np.pi * X) + X + rng.normal(loc=0, scale=0.5, size=n)
plt.scatter(X, y, c="black", s=20)
plt.show()

Compare four different models:

• Linear kernel (linear ridge regression, \(\alpha = 1\))
• RBF kernel with \(\gamma = 1\) and \(\gamma = 100\) (both with \(\alpha = 1\))
• RBF kernel with hyperparameters \(\alpha\) and \(\gamma\) selected by cross-validation.

from sklearn.kernel_ridge import KernelRidge
from sklearn.model_selection import GridSearchCV

# Ridge regression
mod1 = KernelRidge(alpha = 1, kernel='linear').fit(X.reshape(-1, 1), y)

# Kernel ridge regression with RBF kernel and different gamma (bandwidth)
mod2 = KernelRidge(alpha = 1, kernel='rbf', gamma=1).fit(X.reshape(-1, 1), y)
mod3 = KernelRidge(alpha = 1, kernel='rbf', gamma=100).fit(X.reshape(-1, 1), y)

# Use cross-validation to select the hyperparameters
mod4 = GridSearchCV(
    KernelRidge(kernel="rbf", gamma=0.1),
    param_grid={"alpha": [1, 0.1, 0.01, 0.001], "gamma": np.logspace(-2, 2, 5)},
).fit(X.reshape(-1, 1), y)

• Small value of \(\gamma\) leads to underfitting and large value of \(\gamma\) leads to overfitting.

Multilayer Perceptron (MLP)

• It is also called a feedforward neural network because information flows through the function being evaluated from \(\boldsymbol{x}\), through the intermediate computations used to define \(f\) , and finally to the output \(y\).
• When feedback connections are included, they are called recurrent neural networks.

Multilayer Perceptron (MLP)

Feature Map in MLP

• The relationship between the input \(\boldsymbol{x}\) and the hidden unit \(\boldsymbol{h}\) is \[ \boldsymbol{h} = \sigma\left(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}\right), \] where \[ \boldsymbol{W} = \begin{bmatrix} \boldsymbol{w}_1^T \\ \boldsymbol{w}_2^T \\ \vdots \\ \boldsymbol{w}_m^T \end{bmatrix} \in \R^{m \times p}, \quad \boldsymbol{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_m \end{bmatrix} \in \R^m. \]

• The function \(\sigma: \R \to \R\) is called an activation function and is applied element-wise to the vector \(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}\).

• Tha map \(\boldsymbol{x} \mapsto \sigma\left(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}\right)\) can be viewed as a feature map parametrized by \(\boldsymbol{W}\) and \(\boldsymbol{b}\).

• That is, we replace the linear predictor \(\beta_0 + \boldsymbol{x}^T \boldsymbol{\beta}\) with \(\beta_0 + \boldsymbol{\beta}^T \boldsymbol{h} = \beta_0 + \boldsymbol{\beta}^T \sigma\left(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}\right)\).

• Finally, we can link the predictor \(\beta_0 + \boldsymbol{\beta}^T \sigma\left(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}\right)\) to the output \(y\) (or more specifically \(\E(Y \mid \boldsymbol{x})\)) using a link function, e.g., identity, logit, or softmax.

Activation function

• The main role of the activation function is to introduce nonlinearity into the model.
• If \(\sigma(x) = x\), then the MLP is equivalent to a linear model since \[\begin{align*} \beta_0 + \boldsymbol{\beta}^T \boldsymbol{h} & = \beta_0 + \boldsymbol{\beta}^T \sigma(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{b}) = \beta_0 + \boldsymbol{\beta}^T \boldsymbol{W} \boldsymbol{x} + \boldsymbol{\beta}^T \boldsymbol{b}\\ & = \left(\beta_0 + \boldsymbol{\beta}^T \boldsymbol{b}\right) + \left(\boldsymbol{\beta}^T \boldsymbol{W}\right) \boldsymbol{x} = \tilde{\beta}_0 + \tilde{\boldsymbol{\beta}}^T \boldsymbol{x}. \end{align*}\]
• From this perspective, \(\sigma\) can be any nonlinear function.
• However, the choice of activation function has a crucial impact on training, especially when the neural network is deep, i.e., it has many hidden layers.
• Common choices of activation functions include:
  • Rectified Linear Unit (ReLU): \(\sigma(x) = \max(0, x)\)
  • Logistic: \(\sigma(x) = \frac{1}{1 + \exp(-x)}\)
  • Hyperbolic Tangent (tanh): \(\sigma(x) = \tanh(x)\)

Activation Functions

More Hidden Layers

• Increasing the number of hidden layers is straightforward and it allows the model to learn more complex functions.
• The MLP is also called a fully connected neural network (FCNN) because we have connections between all the units in adjacent layers.

The general form of an FCNN

• An \(L\)-layer FCNN (the \(L\)th layer is the output layer) can be written recursively as \[ f^{(L)}(\boldsymbol{x}) = \boldsymbol{W}^{(L)}\boldsymbol{h}^{(L-1)} + \boldsymbol{b}^{(L)} \in \R^k, \] where \[ \boldsymbol{h}^{(l)} = \sigma^{(l)}\left(\boldsymbol{W}^{(l)} \boldsymbol{h}^{(l-1)} + \boldsymbol{b}^{(l)}\right), \quad l = 1, \ldots, L-1, \] and \(\boldsymbol{h}^{(0)} = \boldsymbol{x} \in \R^p\).
• The function \(\sigma^{(l)}\) is the activation function for the \(l\)-th layer. Typically, we use the same activation function for all layers.
• The parameter of the model is \(\theta = \{\boldsymbol{W}^{(1)}, \ldots, \boldsymbol{W}^{(L)}, \boldsymbol{b}^{(1)}, \ldots, \boldsymbol{b}^{(L)}\}\).
• Denote the number of nodes in the \(l\)th layer by \(m_l\) (\(m_0 = p\) and \(m_{L} = k\)), i.e., \(\boldsymbol{h}^{(l)} \in \R^{m_l}\).
• Then \(\boldsymbol{W}^{(l)} \in \R^{m_l \times m_{l-1}}\) and \(\boldsymbol{b}^{(l)} \in \R^{m_l}\) and the total number of parameters is \[ \sum_{i=1}^{L} m_l\cdot m_{l-1} + m_l = \sum_{i=1}^{L} m_l(m_{l-1} + 1). \]