Input (size)	Output (price)
\( x^{(1)} \)	\( y^{(1)} \)
\( x^{(2)} \)	\( y^{(2)} \)
\( x^{(3)} \)	\( y^{(3)} \)
\( x^{(4)} \)	\( y^{(4)} \)
\( \vdots \)	\( \vdots \)

Query \( x_q \) goes in, prediction \( y_p \) comes out.

Given the data, find a function \( h \), a.k.a. a hypothesis, that predicts outputs given inputs. \( h : \mathbb{R} \to \mathbb{R} \)

Given the data, find a function \( h \), a.k.a. a hypothesis, that predicts outputs given inputs. \( h : \mathbb{R} \to \mathbb{R} \)

Given the data, find a function \( h \), a.k.a. a hypothesis, that predicts outputs given inputs. \( h : \mathbb{R} \to \mathbb{R} \)

Two parents, two numbers, one axis. The 1.08 scales mothers up before averaging, so that a tall mother and a tall father count the same. It is a modelling decision, made before any fitting, and it is baked into every number that follows.

The fitted line is flatter than the dashed line where a child simply matches the parent. That gap is the whole story.

Galton called this regression towards mediocrity. The field is named after this slope, in this dataset, being less than one.

Tall parents do have tall children. Just less tall. Short parents likewise, in the other direction.

Correlation \( r = 0.459 \). Its square is the share of the variance in the child's height that the fit accounts for: residual over total, subtracted from one.

They fitted world population with one curve:

Fitted on two thousand years of data, they got \( t_0 = 2026.87 \) and \( k = 0.99 \). The curve goes to infinity at \( t_0 \), and that date is a Friday.

To be fair to them: the paper was partly tongue in cheek. They were not predicting infinite people. They were arguing that the growth pattern of the previous two millennia was about to break. It did.

Drag the cutoff and doomsday wanders from 2063 to 2039. It never admits it is wrong: \( R^2 \) stays above 0.99 throughout, highest at 2020, while the 2023 forecast drifts from 8.4 billion past 14.

Growth peaked at 2.2 per cent a year in the early 1960s and has fallen ever since: 1.36 in 2000, 0.88 in 2023. Births per woman fell as child mortality fell, incomes rose and cities grew.

Read the original: Doomsday, Science 132 (1960). A modern critique: arXiv 0910.3056, and the end of hyperbolic growth.

Example: construct a polynomial model

Pick two features and nothing else:

No constant term and no \( x^2 \): those are modelling choices, and they are choices you make before any fitting happens.

The orange houses are the twenty from the linear regression lecture. The faint blue ones are the rest of Ames: data the fit never sees.

Each added power helps a little on the twenty and hurts everywhere else. Ten parameters still cannot pass through twenty scattered houses, so the curve spends its freedom where there are no houses, and past degree 4 its error on real houses runs into millions of dollars. The training error never warned you.

Split data:

• Training error falls as you add features, and keeps falling. It is not evidence of anything.
• Test error turns around. Everything to the left of the turn is bias, everything to the right is variance.
• You cannot see the turn from the training set alone. That is what the held-out data is for.

Seven columns of the Ames file give 127 possible feature sets. Draw 40 houses at random to fit each one, 30 more to test it, and keep the winner. Then ask the houses nobody used.

Choosing on the test set is fitting to the test set. Its score stops being an estimate of the future and becomes one more training error. The fix is a third set.

Test set: evaluate the model at the end of hyperparameter optimization, once.

Validation set: evaluate the model during hyperparameter optimization. The degree and \(\lambda\) are chosen here. Every look spends a little of it too, which is why the test set is touched only once.

$$ J_{train}(\theta) = \frac{1}{2d_{tr}} \sum_{i=1}^{d_{tr}} \left(\theta^\top\phi(x^{(i)}) - y^{(i)}\right)^2 $$ $$ \color{#c678dd}{J_{val}(\theta) = \frac{1}{2d_{val}} \sum_{i=1}^{d_{val}} \left(\theta^\top\phi(x^{(i)}) - y^{(i)}\right)^2} $$ $$ \color{#98c379}{J_{test}(\theta) = \frac{1}{2d_{tt}} \sum_{i=1}^{d_{tt}} \left(\theta^\top\phi(x^{(i)}) - y^{(i)}\right)^2} $$

A degree 8 polynomial on the twenty houses: the kind that failed on the other houses. Raise \(\lambda\), the weight of the penalty.

L2 gives smaller coefficients; L1 gives fewer, setting some exactly to zero.

Two real coefficients: price on size and bedrooms, both standardized (minus their mean, divided by their spread, so the two are comparable). The rings are the cost \(J(\theta)\), lowest at the least-squares point. Drag \(\lambda\), the weight of the penalty in \(J(\theta) + \lambda \lVert \theta \rVert_2^2\). The shaded disc has the radius of the answer it gives: that answer is also the best point anywhere inside the disc, so a bigger \(\lambda\) is a smaller disc.

The answer, \(\hat\theta\), is where the smallest ring touches the disc. A circle has no corners, so the touching point sits on an axis only by coincidence.

Two real coefficients: price on size and bedrooms, both standardized (minus their mean, divided by their spread, so the two are comparable). The rings are the cost \(J(\theta)\), lowest at the least-squares point. Drag \(\lambda\), the weight of the penalty in \(J(\theta) + \lambda \lVert \theta \rVert_1\). The shaded diamond has the size of the answer it gives: that answer is also the best point anywhere inside the diamond, so a bigger \(\lambda\) is a smaller diamond.

The vertical axis is the bedrooms coefficient, not a number of bedrooms. It is negative: at a fixed size, one more bedroom means smaller rooms and a lower price, as in the linear regression lecture.

The answer, \(\hat\theta\), is where the smallest ring touches the diamond. A diamond has corners on the axes, and a ring meets a corner first: a coefficient becomes exactly zero. The tangency is the argument.

Cost function

def cost_function(X, y, theta):
    m = len(y)
    return 1/(2*m) * np.sum((X @ theta - y)**2)

Run it

theta = np.random.randn(degree+1, 1)
learning_rate = 0.01
num_iters = 10000
theta, J_history = gradient_descent(
    X_train, y_train, theta,
    learning_rate, num_iters)

Gradient descent

def gradient_descent(X, y, theta,
                     learning_rate, num_iters):
    m = len(y)
    J_history = np.zeros(num_iters)
    for i in range(num_iters):
        theta = theta - (learning_rate/m) \
            * X.T @ (X @ theta - y)
        J_history[i] = cost_function(X, y, theta)
    return theta, J_history