ML for Science - Lecture 5
From differential equations to predictions on a computer
We've seen how to extract models from data (learning/fitting).
Now: How do we use models to make predictions?
The most basic form of computing: apply a rule repeatedly
Peano Axioms: Generating Natural Numbers
Start with 0, apply successor function S repeatedly → all natural numbers
This is just a for loop:
i = 0
for _ in range(n):
i = i + 1 # successorSimple rule → Complex, interesting sequence
def fib(n):
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return aProperties emerge:
Planetary distances follow a pattern:
The process:
Von Neumann's idea (1940s) + Conway's rules (1970): simple local rules → complex emergent behavior
Click cells to toggle | Generation: 0
Recall Galileo's projectile: $\displaystyle\frac{d^2x}{dt^2} = -g$
We integrated analytically: $x(t) = -\frac{1}{2}gt^2 + v_0 t + x_0$
Solution: Solve on a computer by going back to discrete
Turn derivatives back into differences:
For $\frac{d^2x}{dt^2} = -g$, introduce velocity $v = \frac{dx}{dt}$:
Blue dots = Numerical (Euler) | Green dashed = Analytical solution
$\Delta t$ controls the accuracy of the approximation
The Forward Euler method is the simplest numerical scheme for solving ordinary differential equations (ODEs).
Given an initial value problem:
$$\frac{dy}{dt} = f(t, y), \quad y(t_0) = y_0$$
The Euler update rule is:
$$y_{n+1} = y_n + h \cdot f(t_n, y_n)$$
where $h$ is the step size.
The simplest ODE to test our solver:
$$\frac{dy}{dt} = -y, \quad y(0) = 1$$
The exact solution is $y(t) = e^{-t}$.
The Euler method's accuracy depends on the step size $h$. Smaller steps = better accuracy but more computation.
A more interesting system — two coupled ODEs modeling predator-prey dynamics:
$$\frac{dx}{dt} = \alpha x - \beta x y$$ $$\frac{dy}{dt} = \delta x y - \gamma y$$
where $x$ = prey population, $y$ = predator population.
Exercises:
1. Change the step size h and observe how the Lotka-Volterra solution changes. At what step size does Euler blow up?
2. Implement the 4th-order Runge-Kutta method and compare it to Euler for the same step size.
3. Add a third species to the model (e.g., a super-predator that eats the predator).
The Fur Trade Business:
Hudson Bay Company records (1900-1920)
■ Hare | ■ Lynx (thousands)
Lotka (1925) and Volterra (1926) independently proposed:
The logic:
$x$ = prey (hares), $y$ = predators (lynx)
$\color{#98c379}{\alpha}$ = prey growth |
$\color{#e5c07b}{\beta}$ = predation rate
$\color{#c678dd}{\delta}$ = predator efficiency |
$\color{#56b6c2}{\gamma}$ = predator death
Apply Euler's method to each equation:
Dots = Real data | Lines = Model simulation | Hare | Lynx
The equation of motion:
For small angles: $\sin\theta \approx \theta$ → simple harmonic motion
For large angles: nonlinear! Must solve numerically.
Introduce angular velocity $\omega = \frac{d\theta}{dt}$, then apply Euler:
Left: Pendulum animation | Right: Phase space (θ, ω)
Multiple trajectories in phase space for different initial conditions
Small θ₀: circles | Large θ₀: distorted "eye" shapes
Unlike Fibonacci (exact), numerical solutions have error. Use Taylor expansion:
Rearranging for the derivative:
Key insight Euler method has error $O(\Delta t)$ — halving step size halves the error
Central difference is more accurate for the same $\Delta t$!
Error: $O((\Delta t)^2)$
Now $u(x, t)$ depends on both space and time
Examples where diffusion appears:
Define a grid with spacing $\Delta x$ in space and $\Delta t$ in time:
Apply finite differences to both derivatives:
u value (0 to 1)
"Step Stencil" moves one cell in space | "Next Row" completes a time step
Each new value depends on neighboring values:
FTCS
(Forward Time,
Central Space)
$u_i^{n+1} = u_i^n + r(\Delta u)$
BTCS
(Backward Time,
Central Space)
Solve: $Au^{n+1} = u^n$
Crank-Nicolson
(Average
FTCS + BTCS)
$O((\Delta t)^2, (\Delta x)^2)$ accuracy
Try r > 0.5 to see instability!
How do we build numerical methods that respect physical laws?
The workhorse of scientific computing
Sample the slope at 4 points, take a weighted average → 4th-order accuracy!
ode45, SciPy's solve_ivp)When equations have conservative form:
Finite volume methods ensure conservation is preserved in the discretization.
Theme in Scientific ML: Encode physical knowledge into numerical schemes
Conservation: What enters one cell exits the neighbor. No numerical mass loss.
The dye enters at the left. The mouse is an obstacle that blocks the flow. This is the real incompressible Navier-Stokes, solved in your browser.
Each step solves part of Navier-Stokes numerically:
Reference: Jos Stam, "Stable Fluids" (SIGGRAPH 1999)
After diffusion and advection, velocity may not satisfy $\nabla \cdot \mathbf{u} = 0$
Solve Poisson equation for pressure:
The physics: Fluid heated from below, cooled from above.
Two-dimensional and very turbulent. Simulation by the turbulenceteam group. The cells on the previous slide are what this looks like when the heating is gentle.
Barry Saltzman (Yale, 1961) developed a 7-equation model for convection.
He showed it to Edward Lorenz at MIT — one solution "refused to settle down."
Lorenz noticed: 4 variables quickly became tiny. Only 3 were "keeping each other going."
Saltzman's 7 equations → Lorenz's 3 equations:
Just 3 coupled ODEs, yet the dynamics are incredibly complex
The trajectory never repeats but stays on this strange "butterfly" shape
Lorenz wanted to extend a simulation. Instead of starting over, he typed in the values from a printout:
A difference of about 0.0001 — surely that can't matter?
He goes for a coffee, comes back, and...
The two simulations start nearly identical, then completely diverge.
■ Original vs ■ Perturbed by ε
No matter how small ε is, the trajectories eventually diverge completely.
"Does the flap of a butterfly's wings in Brazil set off a tornado in Texas?"
Lorenz's discovery:
If systems are chaotic and sensitive...
How do we account for uncertainty?
See you next time!