Lecture Zero

What is a model?

a fashion model

a model home

a model student

a climate model

a language model

The same word, every time. That is not an accident.

A model is a simplified representation
of some reality we choose to study.

Hold on to every word in that sentence. Especially "simplified". Especially "choose".

cup

then hand · grab · drink

"I drank water."

Words are models.
Language is models, composed.

Everything you can think of is a model, and language is full of them. The word "cup" is a model. It represents a set of experiences: objects in the world and how you have handled them. And you do not only model "cup". You model "hand", you model "grab", you model "drink". All of these are real experiences of the world that you compress into words. Then you put them together and say: I drank water. A model of a reality that generalizes. Many people can say it; someone who was not there gets to know what happened because of the words. That is language: small representations, composed into a complex representation of reality. What is a cup, really? An averaging out of the differences between all the cups you have experienced: shapes, sizes, materials, colors, collapsed into essential properties we decided make something a cup. It is simplification. It is an approximation. This is something quite deep to ponder.

"For the mind, the only difference between
'I am a table' and 'I am a human being'
is the distance of clustering."

FROM MY NOTEBOOKS, 2014

The nature of the thing

every detail one idea

A model is a choice of what to ignore. The choice is the assumption.

Here is a world: too many details to hold. Slide toward "one idea" and watch what we do to it. We lump. When you cluster, you lose information: you decide that everything within this blob IS the blob. You blur out the details so you can treat the thing as one thing. That is not a flaw of modeling. That IS modeling. A model is a compression, and a compression is a choice: get rid of some information, keep some other. That choice of what to keep and what to ignore is the modeling assumption, and it serves you down the line, because you build models on top of models on top of models, and the filtering is part of every one of them. And here is what you buy with what you lose: you blur out the parts so you can treat the whole as one lumped thing, and act on it. You trade small-scale information for large-scale handles. Now you know what the car is, and if you know what the car is, you can manipulate the car.

the sun rose. the sun rose. the sun rose.
…every morning you have ever had…

so: the sun will rise.

Every model is an extrapolation.
It compresses the past into a pattern, and rides the pattern into the future.

"All models are wrong,
but some are useful."

GEORGE BOX

Models are true until they're not.

You don't get to prove a model right.

Testing models is adversarial, and it is tiring.
So sometimes we stop trying, and live with models that grow more wrong every year.

Freeze your models, and you get dumber.

You are constantly trying to prove your models wrong, whether you notice or not. It is adversarial, and sometimes it is tiring, so you stop testing. Then you live with models that become more wrong with time. A model has to stay alive: it lives among your other models, in a world made of models, and it has to keep interacting and updating. If you freeze some of your internal models, you become, somehow, dumber. Look closely at anything you think you know well: you will be baffled by how much your vague idea was hiding. We are pretty bad at seeing the thing itself. We choose the surface. That is a modeling strategy too. And for the biggest systems, history does not repeat its initial conditions: we are stuck modeling everything at the same time, without the luxury of checking whether the model is correct at all.

temperature

a thing in the air that makes mercury expand.

then we zoomed in: particles, jiggling, bouncing.

The model was replaced by a deeper model.
It will happen again.

Now the mathematics

A network of nonlinear functions has a name. A neural network.

It is mappings, all the way down.

There are two stories you can tell about the world. One: the world is made of things, the things interact, and the laws of their interaction are sometimes known, and we use those laws to predict. That is a good story. But there is another story I like to tell: everything is a function. It turns out you can map between language and images, because that is basically what our brains do. It turns out you can map from previous words to next words. There are a lot of things in the world you can measure, and some relationship between them, and that is just a function, and you fit that function. The mappings are lossy or they are accurate, but it is mappings all the way down. Mathematically, the cleanest model of "this relates to that" is a function: an input-output map. Words are functions. The mappings between words are functions. Functions within functions. Some are simple, and we love those; most are not, and we cannot unpack them, we only know pieces. So draw it honestly: a network of functions, all of them nonlinear, feeding each other. You have seen this drawing before in this course's future: it is a neural network. Machines, it turns out, can learn these functions. That is the whole subject.

\( x \rightarrow y \) two things, and what connects them.

\( x \rightarrow x' \) one thing, and what it means.

Supervised, and unsupervised.
The grounding has to be unsupervised.

Two shapes of learning. Sometimes two things exist and we look for what connects them: x to y. That is supervised learning. Sometimes one thing exists and we try to make sense of it: x becomes a new x the machine discovers on its own. There is no answer sheet. People in this field have always suspected the second one is the real problem, and I have been on that side since before I learned the words for it. To me the grounding has to be unsupervised. Curiously, if you see enough of the world, the unsupervised becomes supervised: enough examples, and reality starts labeling itself. I wrote it this way in 2021: understanding is a combination of the two. The compression part is unsupervised; its use, reconstructing unseen observations, is supervised. The dichotomy, taken too seriously, is misleading.

A confession about how I know things

I learned rope flow with my body,
not with labels.

The rope is an extension of the body. Its weight puts you in resonance with frequencies you discover, not memorize.

It starts at the rope, and it ends other places.

Let me tell you how I know some of what I know. I practice rope flow: moving a heavy rope around the body. Nobody labeled it for me. The purpose of the rope is to highlight how the body moves: it is an extension of your body, and depending on how heavy it is, it puts you in resonance with different frequencies. You learn the frequencies by becoming them. Whatever model of learning we build, at its core it cannot be supervised: it starts at the rope and then it ends other places. And notice what our machines are missing: large language models are trained on one type of intelligence. There is no kinetic intelligence in there, nothing of movement, nothing of the body. That embodied intelligence is, in fact, our intelligence. Keep your body in the loop this semester: walk, draw, build. It is part of how you model.

Thinking is obsessing.

If I ask you to think about X and you come back with X, you haven't thought yet.
The Y and the Z nobody asked for, that is the thinking.

This course will ask you for X. Bring Y and Z.

One more thing before any mathematics. You cannot use a machine to think for you. To create value you have to produce it, out of your own creative will, and that requires thinking. Thinking is not reading a lot, and it is not solving the assigned problem. Thinking is more like obsessing. If I tell you go think about X, and you think about X and stop, you did not think. X was a prompt, a warm-up. The Y and the Z you were never asked for, the ones that might waste your time, might lead nowhere, might even be a little dangerous: those are the thinking. We are usually taught not to go there. In this course, go there. And here is the connection to the mathematics you will meet soon: humans overfit all the time. You overfit on exams; exams are your test set, and you drill them because you think your future is in peril otherwise. Creativity is the way out of overfitting: question everything, rethink from scratch, do something that makes no sense, test it, fail, try again, fail again. That is how you become creative.

This whole course is one long story about models.

Models of data.
Models of learning.
Models of you.

A model turns ceaseless change into dependable constants.

Next: we fit our first one.

Everything that follows in this course, the regressions, the networks, the losses and gradients, is this lecture wearing mathematics. Every method is a way of choosing what to ignore. Every loss is a way of saying what "wrong" means. Every model we build will be true until it is not, and your job is to notice when. I have been circling one sentence for years, since before this course existed: intelligence comes down to finding the smallest program that can reconstruct observations. The more the observation is compressed, the more powerful the tool it creates; and the tool is judged by acting on the world and comparing again. That is the whole course in one breath. Next time we fit our first model, and now you know what that word is carrying.