A Brief History of Empirical Modeling

From Galileo to Machine Learning

Data-Driven Modeling in Science & Engineering

"The most incomprehensible thing about the world is that it is comprehensible. The fact that it is comprehensible is a miracle."
-- Albert Einstein

The Renaissance Revolution

  • Language opened our eyes to abstract representations
  • The Renaissance: space and time can be quantified
  • Simple mathematical forms predict behavior with surprising accuracy

Galileo Galilei

  • Dropped objects from the Leaning Tower of Pisa
  • Deduced relationships between time and displacement
  • Newton later formalized: $F = ma$
Key insight Controllable input measurements linearly affect output behavior

20th Century Paradigm Shifts

  • Relativity of time
  • Quantum nature of matter
  • The advent of computers (most revolutionary!)

Before Computers

Scientists discovered mathematical rules using:

  • Pens and paper
  • Vivid imagination
  • Analytical techniques

Strong bias towards:

  • Well-posed linear equations
  • Few variables as possible
  • Simple building blocks (atoms, cells, agents)
  • Systems where superposition applies

The Great Limitation

These assumptions cannot capture the true complexity of:

  • Living things
  • Ecosystems
  • Self-organizing systems
  • Turbulent flows

Almost everything that matters!

Computers Change Everything

  • Crunch numbers at very high speeds
  • Solve complicated nonlinear systems
  • Numerical "experiments" / simulations
  • Study complex, nonlinear, high-dimensional, multi-scale systems

The Modern Scientific Process

  1. Derive equations
  2. Design numerical solvers
  3. Simulate
  4. Analyze predictions
  5. Refine model until observations match

The ML Revolution: Three Factors

1. Moore's Law

Transistors double every 2 years

2. Data Explosion

Data generated online doubles every 2 years

3. Algorithmic Advances

ML algorithms became efficient to train and scale

The Big Question

Can we automate the process of
modeling complex systems?

Empirical Laws

Discovering physical laws =

  • Well-designed experiment
  • Controlled input causes output to change
  • Simple linear fit
  • Proportionality constant gets physical meaning

Classical Empirical Laws

  • Pascal's Law (1653): $\Delta p = \rho g \Delta h$
  • Hooke's Law (1678): $F = -kx$
  • Newton's Viscosity (1701): $\tau = \mu \frac{du}{dy}$
  • Ohm's Law (1781): $I = V/R$
  • Fourier's Law (1822): $q = -k \frac{dT}{dx}$
  • Fick's Law (1855): $J = -D \frac{dC}{dx}$

The Ideal Gas Law

$PV = k_B NT$

Combining 2 centuries of linear laws:

  • Boyle (1662): $P \propto V$
  • Charles (1787): $T \propto V$
  • Amonton (1808): $P \propto T$
  • Avogadro (1811): $N \propto V$

The Birth of Curve Fitting

  • Linear relationships inferred geometrically (with a ruler!)
  • Early 19th century: Legendre & Gauss use least-squares
  • 1821: Gauss develops full theory of least-squares optimization
The foundation of modern machine learning!

Summary

  • From Galileo's experiments to automated discovery
  • Computers unlocked nonlinear, complex systems
  • ML + Data + Compute = New paradigm

Next: Introduction to Machine Learning