Introduction to Machine Learning

Maximum Likelihood
and Generalized Linear Models

One recipe behind every model we have built

Probabilistic Interpretation of Linear Regression

Assume the noise is normally distributed around the model:

Noise model \( y^{(i)} = \theta^\top x^{(i)} + \varepsilon^{(i)}, \qquad \varepsilon^{(i)} \sim \mathcal{N}(0, \sigma^2) \)

the model is the line;
every step wobbles around it

\( p(\varepsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left( -\frac{(\varepsilon^{(i)})^2}{2\sigma^2} \right) \)

\( p(y^{(i)} \mid x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left( -\frac{(y^{(i)} - \theta^\top x^{(i)})^2}{2\sigma^2} \right) \)

Likelihood of Output Given Input

The examples are independent and identically distributed, so the dataset probability is a product:

\( \displaystyle L(\theta) = \prod_{i=1}^{m} p(y^{(i)} \mid x^{(i)}; \theta) = \prod_{i=1}^{m} \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left( -\frac{(y^{(i)} - \theta^\top x^{(i)})^2}{2\sigma^2} \right) \)

Take the log, and the product becomes a sum:

\( \displaystyle \ell(\theta) = \log L(\theta) = m \log \frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{\sigma^2} \cdot \frac{1}{2}\sum_{i=1}^{m} \left( y^{(i)} - \theta^\top x^{(i)} \right)^2 \)

Maximize Log-Likelihood

The punchline Maximizing \( \ell(\theta) \) is minimizing \( \tfrac12 \sum_i (y^{(i)} - \theta^\top x^{(i)})^2 \)

Least squares was never an arbitrary choice. It is maximum likelihood under Gaussian noise.

The question that opens today What if the noise is not Gaussian?

Why Not Least Squares in Logistic Regression?

\( h_\theta(x) = \dfrac{1}{1 + e^{-\theta^\top x}} = \sigma(\theta^\top x) \)

The output is a probability The label is 0 or 1, never in between. Squared error treats a probability like a measurement, and its noise is not Gaussian.

\( P(y = 1 \mid x; \theta) = h_\theta(x), \qquad P(y = 0 \mid x; \theta) = 1 - h_\theta(x) \)

\( p(y \mid x; \theta) = \big(h_\theta(x)\big)^{y} \big(1 - h_\theta(x)\big)^{1-y} \) Bernoulli noise

The Bernoulli Distribution

If \(X\) is a random variable with a Bernoulli distribution, then:

\( \Pr(X = 1) = p = 1 - \Pr(X = 0) = 1 - q \)

Its probability mass function \(f\), over possible outcomes \(k\), is

\( f(k; p) = \begin{cases} p & k = 1 \\ q = 1 - p & k = 0 \end{cases} \)

\( f(k; p) = p^k (1-p)^{1-k} \quad \text{for } k \in \{0, 1\} \)

The Bernoulli distribution is the binomial distribution with \(n = 1\).

Define the Log-Likelihood

\( p(y \mid x; \theta) = \big(h_\theta(x)\big)^{y}\big(1 - h_\theta(x)\big)^{1-y} \quad \text{for all } (x, y) \)

\( \displaystyle L(\theta) = \prod_{i=1}^{m} \big(h_\theta(x^{(i)})\big)^{y^{(i)}} \big(1 - h_\theta(x^{(i)})\big)^{1 - y^{(i)}} \)

\( \displaystyle \ell(\theta) = \sum_{i=1}^{m} \left[ y^{(i)} \log h_\theta(x^{(i)}) + (1 - y^{(i)}) \log\big(1 - h_\theta(x^{(i)})\big) \right] \)

Maximize It, and Meet an Old Friend

Update rule

\( \theta_j := \theta_j + \alpha \dfrac{\partial \ell(\theta)}{\partial \theta_j} \)

gradient ascent: we are maximizing

Derive

\( \theta := \theta + \alpha \sum_{i=1}^{m} \big( y^{(i)} - h_\theta(x^{(i)}) \big) x^{(i)} \)

Same as linear regression Different model, different noise, identical update. That is not a coincidence.

Generalized Linear Models

Gaussian \( \longrightarrow \) Linear regression

Bernoulli \( \longrightarrow \) Logistic regression

\( \theta := \theta - \alpha \sum_{i=1}^{m} \big( h_\theta(x^{(i)}) - y^{(i)} \big) x^{(i)} \)

Are there other distributions that lead to the same update rule?

The Exponential Family

The family of distributions for which we can derive the same update rule.

Exponential family \( p(y; \eta) = b(y) \exp\big\{ \eta^\top y - a(\eta) \big\} \)

\( b(y) \) is the base measure, and does not depend on \(\eta\)
\( a(\eta) \) is the log partition function, and does not depend on \(y\)
\( \eta \) is the natural parameter; \(\eta\) and \(y\) have the same dimensions, and \(a(\eta)\), \(y\) and \(b(y)\) are scalars here

Example 1: Bernoulli Becomes Logistic Regression

\( p(y; \phi) = \phi^{y} (1 - \phi)^{1-y} \)

Rewrite it in exponential-family form:

\( \displaystyle p(y; \phi) = \exp\left\{ y \underbrace{\log \frac{\phi}{1 - \phi}}_{\eta} + \underbrace{\log(1 - \phi)}_{-a(\eta)} \right\} \)

Read the natural parameter \( \eta = \log \dfrac{\phi}{1-\phi} \), so inverting gives \( \phi = \dfrac{1}{1 + e^{-\eta}} \): the sigmoid appears on its own.

Example 2: Gaussian Becomes Linear Regression

\( p(y; \mu) = \dfrac{1}{\sqrt{2\pi}} \exp\left\{ -\tfrac12 (y - \mu)^2 \right\} \)

Expanding the square and collecting terms:

\( \displaystyle p(y; \mu) = \underbrace{\frac{1}{\sqrt{2\pi}} e^{-y^2/2}}_{b(y)} \exp\left\{ \underbrace{\mu}_{\eta} y - \underbrace{\tfrac{\mu^2}{2}}_{a(\eta)} \right\} \)

The natural parameter is the mean, so the canonical response is the identity: a straight line.

Why Do We Care?

\( p(y; \eta) = b(y) \exp\big\{ \eta^\top y - a(\eta) \big\} \)

\( \eta = \theta^\top x \)

Inference is easy

\( \mathbb{E}[y; \eta] = \dfrac{d\,a(\eta)}{d\eta} \)

\( \mathrm{Var}[y; \eta] = \dfrac{d^2 a(\eta)}{d\eta^2} \)

Mean and variance come from derivatives of one function.

Learning is easy Maximum likelihood leads to a convex problem in \(\eta\), so gradient descent finds the global optimum.

Data Type Chooses the Distribution

Assumption: \( p(y \mid x; \theta) \) is an exponential family.

Binary Bernoulli → logistic regression

Real Gaussian → linear regression

Counts Poisson

Positive real Gamma, exponential

Distributions Dirichlet

The rule Pick the distribution that matches what \(y\) is, and the model follows.

Building a GLM

The natural parameter is linear in the inputs: \( \eta = \theta^\top x \). The predictor is a natural consequence, not a separate choice.
The prediction is the conditional mean: \( h_\theta(x) = \mathbb{E}[y \mid x; \theta] \)
Learning maximizes \( \log p(y \mid x; \theta) \); inference reads \( \mathbb{E}[y \mid x; \theta] \)

Update rule, always \( \theta := \theta - \alpha \sum_{i=1}^{m} \big( h_\theta(x^{(i)}) - y^{(i)} \big) x^{(i)} \)

Terminology

\(x\) \( \longrightarrow \) \( \theta^\top x \) \( \longrightarrow \) \( \eta \) \( \xrightarrow{\;g\;} \) \( h_\theta(x) \)

Names \(\theta\) model parameter · \(\eta\) natural parameter · \(\phi\) canonical parameter · \(g\) canonical response function · \(g^{-1}\) link function

Logistic regression, named \( h_\theta(x) = \mathbb{E}[y \mid x; \theta] \) and \( \phi = \dfrac{1}{1 + e^{-\eta}} = \dfrac{1}{1 + e^{-\theta^\top x}} \)

Evaluation Metrics

Precision \( \dfrac{TP}{TP + FP} \): of what we flagged, how much was real

Recall \( \dfrac{TP}{TP + FN} \): of what was real, how much we caught

F1 \( \dfrac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = \dfrac{2 TP}{2 TP + FP + FN} \)

Accuracy and Precision Are Not the Same Word

Accuracy How close the estimates sit to the reference value: the distance from the truth.

Precision How tightly the estimates cluster: the spread, whether or not they are centred correctly.

Summary

The recipe

\( p(y; \eta) = b(y)\exp\{\eta^\top y - a(\eta)\} \)

choose the distribution that matches \(y\)

The link