Three questions, three kinds of answer

Least squares answered the first, and logistic regression the second, with the same update rule. Today that stops being a coincidence: one recipe answers all three, and the only thing that changes between them is one function.

Six questions for today

1. 1The likelihoodWhat does it mean for a model to explain the data?
2. 2Yes or noWhat is the likelihood of an event that either happens or does not?
3. 3One familyWhy do linear and logistic regression share one update rule?
4. 4CountsWhat if the answer counts something?
5. 5The recipeWhat do all three models share, and what are the parts called?
6. 6In codeWhat does one GLM look like in ten lines of Python?

Every figure today is real data you can download: the Ames houses, the Challenger O-rings, and the US Geological Survey earthquake catalogue.

Probabilistic Interpretation of Linear Regression

Assume the noise is normally distributed around the model:

the model is the line;
every step wobbles around it

\( p(\varepsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left( -\frac{(\varepsilon^{(i)})^2}{2\sigma^2} \right) \)

\( p(y^{(i)} \mid x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left( -\frac{(y^{(i)} - \theta^\top x^{(i)})^2}{2\sigma^2} \right) \)

Likelihood of Output Given Input

The examples are independent and identically distributed, so the dataset probability is a product:

Take the log, and the product becomes a sum:

Likelihood, feltThe twenty houses from linear regression, with the noise model laid over them

Each house glows by how probable the model finds its price. The intercept follows the slope, so the line always passes through the average house.

Maximize Log-Likelihood

Least squares was never an arbitrary choice. It is maximum likelihood under Gaussian noise.

The night before Challenger27 January 1986: will the cold hurt the seals?

Data: Dalal, Fowlkes and Hoadley, Risk analysis of the space shuttle: pre-Challenger prediction of failure, JASA 84 (1989), via the UCI repository. Damage means at least one of the six O-rings came back eroded or scorched; on two flights, two of the six did. Flights launched at the same temperature are drawn side by side.

Why Not Least Squares in Logistic Regression?

The Bernoulli Distribution

If \(X\) is a random variable with a Bernoulli distribution, then:

Its probability mass function \(f\), over possible outcomes \(k\), is

The Bernoulli distribution is the binomial distribution with \(n = 1\).

Define the Log-Likelihood

The Bernoulli likelihood, flight by flight

Maximize It, and Meet an Old Friend

Generalized Linear Models

Are there other distributions that lead to the same update rule?

The Exponential Family

The family of distributions for which we can derive the same update rule.

• \( b(y) \) is the base measure, and does not depend on \(\eta\)
• \( a(\eta) \) is the log partition function, and does not depend on \(y\)
• \( \eta \) is the natural parameter; \(\eta\) and \(y\) have the same dimensions, and \(a(\eta)\), \(y\) and \(b(y)\) are scalars here

Example 1: Bernoulli Becomes Logistic Regression

Rewrite it in exponential-family form:

Example 2: Gaussian Becomes Linear Regression

Expanding the square and collecting terms:

The natural parameter is the mean, so the canonical response is the identity: a straight line.

Example 3: Poisson Becomes Log-Linear Regression

The Poisson distribution counts events in a fixed window when each happens independently of the others: earthquakes in a decade, clicks in an hour, typos on a page. \(\lambda\) is the expected count. Rewrite it in exponential-family form:

Why Do We Care?

One function holds the mean and the varianceTurn \(\eta\), and read both off the curve \(a(\eta)\)

Data Type Chooses the Distribution

Assumption: \( p(y \mid x; \theta) \) is an exponential family.

How many earthquakes of each size?Every magnitude 5 and above on Earth, 2015 to 2024: 16,945 earthquakes

Data: US Geological Survey, ANSS Comprehensive Catalog, earthquake.usgs.gov/fdsnws/event/1/query, fetched 2026-09-28. Below M 5.5 half the magnitudes are body-wave estimates, which run low for large events; from M 6.0 almost all are moment magnitudes.

Reading a Poisson coefficient

A linear model adds \(\theta_1\) for every unit of \(x\). A Poisson model multiplies the expected count by \(e^{\theta_1}\): on the log scale the line is straight, on the count scale it is a curve that never goes below zero.

Poisson says the variance equals the mean. Year by year, earthquakes of M 7.0 to 7.4 average 9.0 with variance 9.1. Smaller ones come in aftershock swarms and vary more: M 6.0 to 6.4 average 91.6, variance 203.8.

Same loop, three modelsOnly \(h\) changes

Building a GLM

• The natural parameter is linear in the inputs: \( \eta = \theta^\top x \). The predictor is a natural consequence, not a separate choice.
• The prediction is the conditional mean: \( h_\theta(x) = \mathbb{E}[y \mid x; \theta] \)
• Learning maximizes \( \log p(y \mid x; \theta) \); inference reads \( \mathbb{E}[y \mid x; \theta] \)

Terminology

One function, two datasetsPress Run: this is real Python, in your browser

import numpy as np, pandas as pd

def fit_glm(X, y, h, alpha, steps=20000):
    theta = np.zeros(X.shape[1])
    for _ in range(steps):                       # the same update for every GLM
        theta += alpha * X.T @ (y - h(X @ theta)) / len(y)
    return theta

sigmoid = lambda z: 1 / (1 + np.exp(-z))

c = pd.read_csv('challenger_orings.csv')         # yes or no: Bernoulli, h = sigmoid
X = np.column_stack([np.ones(len(c)), c.temp_c - 20])
th = fit_glm(X, c.damage.values, sigmoid, alpha=0.05)
print('Challenger theta:', th.round(3), ' P(damage at -0.6 C):', sigmoid(th[0] + th[1] * (-0.6 - 20)).round(4))

q = pd.read_csv('usgs_quakes_m5_2015_2024.csv')  # a count: Poisson, h = exp
M = np.round(np.arange(6.0, 8.95, 0.1), 1)
y = np.array([(q.mag.round(1) == m).sum() for m in M])
X = np.column_stack([np.ones(len(M)), M - 6])
th = fit_glm(X, y, np.exp, alpha=0.01)
print('Earthquake theta:', th.round(3), ' ten times fewer? e^theta1 =', np.exp(th[1]).round(4))
    

You should get \(\theta = (-0.749, -0.417)\) and a 99.96% chance of damage, then \(\theta = (5.616, -2.306)\) and \(e^{\theta_1} = 0.0997\): the numbers on the slides. Try h = lambda z: z on a real-valued \(y\), and the same function is least squares.

Summary

What to take away

1. The likelihood asks how probable the data is under the model. Least squares is its answer under Gaussian noise, cross-entropy its answer under Bernoulli noise.
2. Zeros are data. The Challenger flights that went fine carry the pattern; leave them out and it disappears.
3. Choose the distribution from what \(y\) is: a real number, a yes or no, a count. Its natural parameter \(\eta = \theta^\top x\) gives the model: the identity, the sigmoid, the exponential.
4. One function \(a(\eta)\) holds the mean and the variance, and the update \(\theta := \theta + \alpha \sum_i (y^{(i)} - h_\theta(x^{(i)}))\,x^{(i)}\) is the same for every member of the family.
5. A Poisson coefficient multiplies: \(e^{\theta_1} = 0.0997\) says ten times fewer earthquakes per unit of magnitude.