A smooth function that returns probability of occurrence
midpoint \( x_0 = -\theta_0/\theta_1 = \)
The curve crosses \( \tfrac12 \) where \( \theta_0 + \theta_1 x = 0 \), at \( x_0 = -\theta_0/\theta_1 \).
So \( \theta_1 \) sets the steepness but also moves the midpoint; to steepen in place, scale \( \theta_0 \) with it.
Read \( h_\theta(x) \) as \( P(y = 1 \mid x;\, \theta) \). Still a linear model at heart: the
linearity lives inside \( \theta^\top x \); the sigmoid only turns that score into a probability.
A smooth function that returns probability of occurrence
✏ exercise · do it now
What goes wrong if you fit the labels \( 0/1 \) by least squares?
theta = np.linalg.lstsq(X, y)[0] # why not?
Squared Error Meets the Sigmoid
The loss as a function of \( \theta_1 \) on the tumor data. Squared error saturates into flat
plateaus where the gradient dies; cross-entropy stays a clean bowl.
What Are Those Two Curves?
Write \( z = \theta^\top x \). Each example pays a per-example loss \( \ell(z) \), and \( J(\theta) = \tfrac{1}{m}\sum_i \ell\big(\theta^\top x^{(i)}\big) \) is their average: the curves on the last slide, swept over \( \theta_1 \).
One example with \( y = 1 \), loss against the score \( z \).
The probability the model gives to the labels we actually observed, read as a function of
\( \theta \) with the data fixed. Maximum likelihood picks the \( \theta \) that makes it largest.
Log, Then Average
Take \( \log \): the product becomes a sum and the exponents come down.
Flip the sign to get something to minimize, and average. That is the loss
\( J \); each example contributes its own negative log-likelihood \( \ell^{(i)} \):
Notation for the rest of the course: \( L \) is the likelihood of the whole data set,
\( \log L \) the log-likelihood, \( \ell^{(i)} \) one example's loss, \( J \) the loss we minimize.
Two measured features (tumor size, cell irregularity). The decision boundary is \( \theta^\top x = 0 \) and
the shading is \( P(y=1) \). Click inside the plot to add a malignant point and watch the boundary respond.
The Geometry of the Boundary
The boundary
Predict 1 when \( P \ge \tfrac12 \), that is when the
score \( \theta_0 + \theta_1 x_1 + \theta_2 x_2 \) is positive.
The boundary is where the score is \( 0 \).
The normal
\( (\theta_1, \theta_2) \) is perpendicular to the line and points to the positive side.
\( \theta_0 \) has no direction: it only slides the line.
\( \theta_0 \) is not in the norm because it is not part of the direction.
Further out, more confident: \( P \to 0 \) or \( 1 \).
The Margin
Labels \( y \in \{-1, +1\} \)
Relabel the classes \( \pm 1 \). Then a point is on the correct side exactly when
\( y\,(\theta^\top x) > 0 \): the sign of the score agrees with the label.
The geometric margin is the signed distance of the last slide: positive for correct points, negative for mistakes.
Margin of the boundary
\( \gamma = \min_i \gamma^{(i)} \): the distance to the closest point. Wider is safer; the boundary that maximises it is the support vector machine.
Step 4: Test. But Accuracy of What?
Every prediction lands in one of four cells:
TP flagged and real,
FP flagged and not,
FN missed,
TN correctly ignored.
The trap
Flag nobody: accuracy 0.95, better than the classifier, and recall 0. Accuracy rewards agreeing with the majority.
Which number, whenRecall when a miss is the expensive error: cancer screening, fraud, a security scan. Precision when a false alarm is the expensive error: spam filters (a real email lost), an alert a human must investigate. Accuracy only when classes are balanced and both errors cost the same. F1 when you must rank models by one number and care about both.
When Accuracy Lies: Imbalanced Data
pred 1
pred 0
true 1
0
0
true 0
0
0
accuracyprecisionrecallF1
Practical advice
With 1% positives, predicting "no" for everyone scores 99% accuracy and catches nobody.
Report precision and recall, not accuracy. Then move the threshold:
lower it to buy recall when a miss is expensive (disease, fraud), raise it to buy precision
when a false alarm is expensive.
Reading the Metrics Together
Tumour data with 300 more healthy patients (6% ill): one trained model, every threshold from 0 to 1.
One model, many classifiers
Training fixes \( \theta \); the threshold is chosen afterwards. Lower it: recall up, precision down. Raise it: the reverse.
What accuracy is doing
Past a threshold of 0.3 it sits above 0.95 and barely moves, while precision and recall swing between 0.5 and 1. It cannot see the trade, so it is the wrong compass here.
What to report
Confusion matrix, precision and recall at the threshold you deploy. To compare models regardless of threshold: the precision-recall curve, or the ROC curve, and the area under it.
Logistic Regression in Twelve Lines
import numpy as np
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
X = np.c_[np.ones(len(x1)), x1, x2] # n x 3: bias + two features
theta = np.zeros(3)
alpha = 0.5
for t in range(2000): # batch gradient descent
p = sigmoid(X @ theta) # predicted probabilities
theta -= alpha / len(y) * (X.T @ (p - y))
print(theta) # boundary: theta . x = 0
Line 12 is the whole lecture: (prediction error) times (input), averaged. Swap the sigmoid
for the identity and this is last week's linear regression code.
In scikit-learn the same fit is LogisticRegression().fit(X, y).
✏ exercise · do it now
What is the loop doing: batch gradient descent, stochastic gradient descent, something else?
What would you change to make it the other one?
How to turn scores into probabilities?Note: this slide explains the intuition. Softmax regression is not exactly equivalent to a combination of one-vs-all classifiers.
What’s the probability it’s a car?
WARNING!!! Notation Alert \( \vec{\theta}_i \) is a vector
They open inside the slide; press Esc to come back.
Look closely
The Playground link has zero hidden layers and a sigmoid output. That configuration
is logistic regression: one neuron. Hold that thought for two lectures.
What if the boundary needs to curve?
No straight line separates these classes.
Two answers exist: invent better features by hand, or let the model learn them.
That second answer is called a neural network.
one-hot labels, one score per class, cross-entropy loss
Caveats: boundaries are straight; accuracy misleads when classes are imbalanced; pick the threshold for the cost you care about.
Next: maximum likelihood as a worldview, and the family that unites linear and logistic regression.
