Introduction to Machine Learning

Math and Python:
Your Computational Toolkit

The same idea, written twice: once in notation, once in code

Every Idea, Written Twice

Notation
\( \theta^\top x \)
Compact, exact, and the way papers and textbooks will speak to you.
Code
theta @ x
Runnable, checkable, and the way you will actually test an idea.

This session is a translation exercise. Every symbol you meet has a line of code, and every line of code has a symbol.

Bring your laptop The five notebooks are the lab. These slides are the map.

Vectors

drag either arrow head

Definition An ordered list of numbers, \( x \in \mathbb{R}^n \).
\( x = [x_1, x_2, \dots, x_n]^\top \)
Geometry A direction with a length. Addition is placing them head to tail; the sum is the diagonal.
In machine learning One example is a vector. One row of your data table.

Vector Operations

import numpy as np

x = np.array([2, 3, 1])
y = np.array([1, -2, 4])
print(x.shape, len(x))

# addition and scaling
print(x + y)
print(3 * x)

# the inner product
print(np.dot(x, y))
print(np.linalg.norm(x))

The inner product \( x^\top y = \sum_i x_i y_i \) is the single most common operation in this course.

Matrices

Definition A rectangular array, \( A \in \mathbb{R}^{m \times n} \): \(m\) rows, \(n\) columns.
In machine learning Your whole dataset: one row per example, one column per feature.
\( X \in \mathbb{R}^{m \times d} \)
Also A linear transformation: \(A x\) sends a vector to another vector.

\( C = AB \): hover a cell of C to see which row and column produce it

Matrices in NumPy

A = np.array([[1, 2, 3],
              [4, 5, 6]])
print(A.shape)   # (2, 3)
print(A.size)    # 6

B = np.zeros((3, 3))
C = np.ones((2, 4))
D = np.eye(3)
E = np.random.rand(2, 3)

# the two products people confuse
print(A * A)        # elementwise
print(A @ A.T)      # matrix product
The classic bug A * B multiplies elementwise; A @ B is the matrix product. They are different operations, and NumPy will happily run the wrong one.

Inverse, Determinant, and When Things Break

Inverse \( A A^{-1} = A^{-1} A = I \). Solving \( Ax = b \) means \( x = A^{-1} b \).
Determinant \( \det(A) \) is the volume scale factor. If it is zero, the map flattens space and cannot be undone.
When does an inverse exist? Square, and \( \det(A) \ne 0 \), that is, full rank: no column is a combination of the others.
Why you will care The normal equations need \( (X^\top X)^{-1} \). Duplicate a feature and that inverse stops existing.

In practice use np.linalg.solve(A, b), never np.linalg.inv(A) @ b: it is faster and numerically safer.

The Calculus You Actually Need

Derivative The slope: how much the output moves when the input nudges.
Gradient The vector of partial derivatives, pointing uphill.
\( \nabla_\theta J = \left[ \frac{\partial J}{\partial \theta_1}, \dots, \frac{\partial J}{\partial \theta_d} \right]^\top \)

drag the point to see the slope

Three rules cover almost everything \( \nabla_\theta (\theta^\top x) = x \),   \( \nabla_\theta \lVert \theta \rVert^2 = 2\theta \),   and the chain rule.

Reading and Exploring Data

import pandas as pd

df = pd.read_csv("students.csv")
print(df.shape)

print(df.head())
print(df.describe())
print(df.isnull().sum())

# one column, as a NumPy vector
hours = df["study_hours"].values
print(hours.mean(), hours.std())

Always describe() and check for missing values before modeling anything.

Look at It Before You Model It

Same statistics These four datasets share their means, variances, and correlation to two decimal places.
Different worlds A line fitted to all four is identical, and wrong for three of them. Plot your data.

Anscombe's quartet, 1973

The Lab: Five Notebooks

  • 00 Python intro: syntax, collections, functions
  • 01 Vectors: creation, arithmetic, inner products, norms
  • 02 Matrices: indexing, slicing, the two products
  • 03 Matrix properties and calculus: inverse, determinant, gradients
  • 04 Data reading and plotting: pandas, summary statistics, figures
How to use them Run every cell. Change a number and predict what happens before you press shift-enter. That prediction is the learning.

Summary

Vectors
\( x^\top y = \sum_i x_i y_i \)

one example, one row

Matrices
\( X \in \mathbb{R}^{m \times d} \)

the dataset, and a transformation

Gradients
\( \nabla_\theta J \)

the direction learning walks against

And always

look at the data before you model it

Next: supervised learning and linear regression, where all of this becomes one line of code.