From Data to Learning

Data-Driven Modeling in Science and Engineering

Announcements and Logistics

• Course site: ml4science.com. Maths and Python reference: intro2ml.com
• Assignment 0 is posted: Problem Set 0
• The purpose of this class is to learn how to learn. In fields that move this fast, papers are the medium, not textbooks.
• scholar-inbox.com, plus YouTube, blogs, and whatever else keeps you current
• The simple formula: read a paper, clone the repository, reproduce the result, come back with a question
• Main deliverable: a presentation and a paper

Finding constants of Nature That Generalize Across Space and Time

Galileo

Kepler

Descartes

Galileo's cartoon says the quiet part out loud: "It shouldn't happen this way. I'll have to alter the data."

The Three Pillars of Artificial Intelligence

The world is messy. The question is simple: what is the shortest way home?

The Data Science Hierarchy of Needs

As illustrated by Monica Rogati. Most of the pyramid is not machine learning: it is collecting, moving and cleaning the data first.

Stocks

Audio

• Speech recognition is a function \( F: \text{array} \rightarrow \text{text} \)

From a Wave to a Picture

The short-time Fourier transform: the first thing anyone does to a sound before modelling it

• The waveform is one number per instant. It hides which frequencies are present.
• Cut the signal into short windows, take the Fourier transform of each, stack them side by side. Now pitch is visible.
• Speech recognisers and birdsong classifiers almost never see the wave. They see this picture.

Images

move your mouse over the image

• A color image is \( h \times w \times 3 \): height, width, and three channels (red, green, blue)
• Your phone: \( 3000 \times 4000 \times 3 = 36 \) million numbers per photo

Colour Is Three Matrices

The grey image was one number per pixel. Colour is three, stacked

• A colour image of height \(h\) and width \(w\) is an array of shape \( h \times w \times 3 \).
• Nothing about it is a picture to the machine. It is three grids of numbers between 0 and 255.

Image Recognition

• 2010: deep learning passes traditional methods at speech recognition
• 2011: IBM Watson beats the best human players at Jeopardy
• 2012: Google Brain finds cats in YouTube, unsupervised
• 2014: face recognition reaches 97%
• 2019: deep models match radiologists on lung scans
• 2021: AlphaFold predicts protein structure from sequence
• 2022: text becomes images, and images become text
• 2024: one model reads an image, a page and a recording together
• 2026: two lines of prompt produce fifteen seconds of film

The same training recipe each time, on new data.

2019: lung cancer detection from chest scans
Ardila et al., Nature Medicine 2019

Generative Adversarial Networks (2014)

Karras et al., Progressive Growing of GANs for Improved Quality, Stability, and Variation. Video

Coupled Ocean Atmosphere Simulation

youtube.com/watch?v=4f0iVY2nd2M

Video

Eadweard Muybridge, 1878: the first time motion was recorded as an array of stills

• One frame is \( h \times w \times 3 \); a clip is \( T \times h \times w \times 3 \)
• One minute of 1080p at 30 fps: about 11 billion numbers
• Muybridge settled a bet: are all four hooves ever off the ground at once? You cannot see it, so he turned it into data.

Frames from The Horse in Motion, Wikimedia Commons, public domain.

Fluid Dynamics

Machine Learning in Materials Research

Text

One-hot word representation

• One-hot is honest but lonely: every word equally far from every other

Text

Word embedding representation

Text Generation

Language Models That Do Science

arxiv.org/abs/2503.08979

Games

IBM creates a checker-playing program · 1959

IBM Deep Blue beats Kasparov · 1997

AlphaGo beats the world's best Go player · 2016

Solving Puzzles

A Rubik's cube has 10¹²⁰ possibilities. You cannot search that. You have to learn something about its structure instead.

Graph Representation

• Maps, molecules, social networks: all matrices, \( A_{ij} = 1 \) when \(i\) connects to \(j\)

Robotics and Games: Reinforcement Learning

Robotics and Manufacturing

Robotics

Boston Dynamics · youtube.com/watch?v=tF4DML7FIWk

Classification, Regression, Structured Prediction

Same skeleton every time. Only the shape of \(y\) changes.

Now you: think of one input-output pair from your own life. What is \( x \), what is \( y \), and which of the three is it?

Data Analysis

Before a model, the summaries: what one variable does, and what two of them do together.

Correlation Coefficient (2 Variables)

Correlation Matrix (Multiple Variables)

\( r_{ij} = \dfrac{\mathrm{Cov}(X_i, X_j)}{\sigma_{X_i} \sigma_{X_j}} \), assembled into \( R \): every pair at once, the diagonal all ones.

statology.org

Histogram and Probability Density Function

Box and Whisker Plot

Five numbers instead of a histogram: minimum, lower quartile, median, upper quartile, maximum. Boston University SPH

Where to Look for Data?

• Curated: Kaggle, UCI, Hugging Face, data.gov
• Collected: sensors, experiments, your phone
• Scraped: the web, politely
• Asked: a language model will name datasets you would not have found, and invent ones that do not exist. Check every link it gives you.

import pandas as pd

url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
df = pd.read_csv(url)

print(df.shape)        # (344, 7): 344 penguins, 7 measurements
print(df.describe())   # your first look at any dataset
    

How Do You Come Up With Project Ideas?

• Start from something you already care about, then go looking for whether anyone has measured it
• Search properly: Google, Google Scholar, and ask a model to argue with you about it
• Read a paper, clone the repository, reproduce the result. The gap between what the paper claims and what the code does is where the project ideas live.
• A good idea names its data before it names its method.