Intro to Neural Networks

🎯 Two ideas you need before you touch a neural network

A neural network is just a pile of multiplications and additions that we adjust to make its outputs less wrong. To do the adjusting, you need exactly two tools from calculus:

• Derivatives — how much does an output change when you nudge an input?
• The chain rule — how do you chain those nudges together across a pipeline of operations?

Everything in this deck — forward pass, loss, backprop, PyTorch's loss.backward() — is these two ideas applied relentlessly. So let's spend five minutes on them first.

📐 Derivatives — the slope at a point

A derivative answers one question: if I move the input by a tiny amount, how much does the output move? That ratio — output change over input change, in the limit — is the slope of the function at that point.

At x = 3, the slope is 2·3 = 6. That means: nudging x up by 0.001 makes x² go up by roughly 0.006. Not exact, but very close — and the smaller the nudge, the better the approximation. (3.001)² − 3² = 9.006001 − 9 = 0.006001. The 0.000001 extra is what vanishes in the limit.

🔗 The chain rule — nudges flow through pipelines

Real functions aren't single terms. They're pipelines: you transform x to get an intermediate value, then transform that to get another, then finally get the output. The chain rule says: to get the end-to-end derivative, multiply the derivatives of each step along the way.

In words: how x affects y = (how g affects y) × (how x affects g). Each arrow in the pipeline contributes one factor.

🏍️ The chain rule, ATV version — Monteverde

You're on an ATV descending from a ridge in Monteverde. You want to know: how fast is my altitude dropping per minute? Two things feed into that:

• Speed : how much distance you cover per unit time — dx/dt = 30 km/h
• Grade : how much altitude you lose per unit distance — dh/dx = −32 m/km

The km units cancel. You drop 960 meters per hour — faster if you speed up, faster if the grade steepens. Neither factor alone tells you the answer. The chain rule chains them.

🍂 The chain rule of composting

Another version — same math. You're composting. Three things speed up decomposition, and they multiply :

Now the backward pass — which factor matters most right now?

Heat has the biggest gradient → improving heat gives you the most bang for your buck. That's exactly how a neural network decides which weight to adjust most: compute each parameter's gradient, update most where the gradient is largest.

🧠 The first neuron — McCulloch & Pitts, 1943

Before there were CNNs, transformers, or GPUs, there was a paper in the Bulletin of Mathematical Biophysics , Volume 5, 1943. Warren McCulloch (a neurophysiologist) and Walter Pitts (an 18-year-old homeless mathematical prodigy) proposed the first mathematical model of a neuron.

Their claim was audacious: cognition could be explained as computation performed by interconnected binary threshold units. That model is the root of everything in modern deep learning.

🍜 Neural network alphabet soup

Before we build one from scratch, here's what's on the menu. Every "X-NN" you've heard of is a different way to wire neurons together for a different shape of data.

📈 Learning curve

A few honest warnings before we start coding:

• Transformers are not intuitive. Neither was the first time you saw a for-loop.
• A neural network can be implemented in a few lines of code — even a transformer . The math is harder than the code.
• A single line change inside a research codebase often represents a full published paper worth of thought. Read slowly.
• Patience is key. The concepts compound. What feels impossible at slide 30 is obvious by slide 80.

🧮 The simplest neural network

Strip a neural network to its bones and you get one line of math:

That's it. A weight w , an input x , a bias b , and an output y . A real neural network stacks millions or billions of these and pipes them through nonlinearities — but the fundamental unit is this single line. Understand it, and you understand what every weight in GPT-4 is doing.

🎛️ Why bias matters — real examples

Bias sounds mundane until you see where it hides in every modern network:

🎯 A concrete example

Let's pick specific numbers. Suppose the true relationship we want to model is:

The weight is 3, the bias is 1. If we feed it inputs 1 through 5, it should produce:

A story to hang this on: an object started 1 m from the origin and is moving at 3 m/s. y = 3·x + 1 is its position after x seconds. The network only sees the pairs — and has to rediscover the 3 and the 1.

Now here's the setup for learning: we have the inputs x and the true outputs y, but we don't know w or b. The job of the neural network is to figure them out from examples alone.

💻 Forward pass — in real Python

Let's put real numbers through real code. The entire forward pass — from raw inputs to a single loss number — is a handful of numpy calls.

# Initial state
w = 0.0
b = 0.0

X = np.array([1, 2, 3, 4, 5])
y = np.array([4, 7, 10, 13, 16])  # true values

Five training examples. We know what the right answer looks like. Now the forward pass:

# Forward pass with w=0, b=0
y_pred = w * X + b
# y_pred = 0 * [1, 2, 3, 4, 5] + 0
# y_pred = [0, 0, 0, 0, 0]

Every prediction is zero. Predictably bad — we haven't trained anything yet. But the structure is there: five inputs, five predictions, all coming out of the same one-line formula w·X + b .

📉 Measuring how wrong we are — step by step

We need a single number that captures the total badness. Three stages:

# Error (prediction - truth)
error = y_pred - y
# error = [0-4, 0-7, 0-10, 0-13, 0-16]
# error = [-4, -7, -10, -13, -16]

# Square it (makes everything positive, penalizes big errors more)
squared_error = error ** 2
# squared_error = [16, 49, 100, 169, 256]

# Mean (average of all squared errors)
loss = np.mean(squared_error)
# loss = (16 + 49 + 100 + 169 + 256) / 5
# loss = 590 / 5
# loss = 118.0

🎚️ From loss number to loss signal

We have loss = 118. But which direction should we push w? Which way for b? The loss alone doesn't say — it just reports the current damage.

To make a decision, we need to know: if I nudge w a tiny bit, how does the loss change? That's the derivative ∂L/∂w . Same for the bias: ∂L/∂b .

These two numbers — a pair of gradients — are the entire steering system of the neural network. We're going to derive them next, using the chain rule you met in Tab 1.

🔗 Why we need the chain rule here

We want ∂(squared_error) / ∂w . But w doesn't appear in squared_error directly . Trace the pipeline:

w affects y_pred, which affects error, which affects squared_error. So we chain the nudges, one stage at a time:

Three nudges, multiplied together. Exactly the same pattern as dh/dt = dh/dx · dx/dt from the ATV example — just with more stages.

🧮 Deriving each stage

Now chain them:

🔬 Sanity-check by hand

Let's verify for the first training point, where x = 1, y_true = 4, y_pred = 0, error = −4. The chain-rule formula says:

Does that match what we'd get by literally nudging w? Let's check. Nudge w by 0.001 and recompute the squared error for the same point:

y_pred = 0.001 * 1 + 0 = 0.001
error = 0.001 - 4 = -3.999
squared_error = (-3.999)² = 15.992
# went from 16 to 15.992 → change of -0.008
# Δ(squared_error) / Δw = -0.008 / 0.001 = -8

Same answer — −8 . The chain rule gets it in one shot without needing to poke around with nudges.

🔗 The bias gradient — same pattern

For b, the chain is the same except for the final stage: ∂(y_pred)/∂b = 1 instead of x .

So for the first training point: 2 · (−4) = −8 . Nudging b up decreases the error for that example — just like nudging w, because at x=1 they happen to have the same effect.

📋 The gradient table — all five points

MSE averages over the whole dataset, so the final gradient is the mean of each point's gradient. Compute for every training example, then average:

Both negative → both should increase. The magnitude tells you which one to push harder: w's gradient is ~3.6× larger, so w moves proportionally more. That's the steering signal — and it came straight out of the chain rule.

⬇️ Using gradients to adjust weights

We have ∂L/∂w = −72 and ∂L/∂b = −20 . Two rules decide the direction:

• If the gradient is positive → increasing the parameter increases loss → decrease it.
• If the gradient is negative → increasing the parameter decreases loss → increase it.

Both of our gradients are negative. So both parameters should go up. But by how much ?

📐 The update rule

For every parameter, one step of gradient descent is:

Plug in:

# Starting point
w = 0.0, b = 0.0
lr = 0.01

# Gradients (computed via chain rule)
dw = -72
db = -20

# Update
w_new = 0 - 0.01 * (-72) = 0 + 0.72 = 0.72
b_new = 0 - 0.01 * (-20) = 0 + 0.20 = 0.20

After one training step, our model went from y = 0·x + 0 to y = 0.72·x + 0.20 . Still wrong (target is 3·x + 1), but closer. Each step nudges it a little closer.

🔁 Gradient Descent — step-by-step breakdown

The full recipe of one training step — exactly what PyTorch's optimizer.step() hides under a friendly name:

y_pred = 0.00 × X + 0.00 = [0.0, 0.0, 0.0, 0.0, 0.0]

error = y_pred − y = [-4, -7, -10, -13, -16]

∂L/∂w = mean(2 × error × X) = -72.0000
∂L/∂b = mean(2 × error)     = -20.0000

w = 0.0000 − 0.0100 × -72.0000 = 0.7200
b = 0.0000 − 0.0100 × -20.0000 = 0.2000

118.0000 → 68.7664 (↓ 49.2336)

📉 Loss history — 10 epochs

An epoch = one full pass over the training data. Do step 1–5 above, ten times. Watch what happens.

🚀 Beyond vanilla SGD — Adam & AdamW

What we just implemented is SGD (stochastic gradient descent). It works, but it has a weakness: it only uses the current gradient and treats every parameter the same. Real networks are trained with smarter optimizers:

💻 The whole thing in numpy — 25 lines

Everything we derived — forward pass, loss, gradients via chain rule, update — in one file:

#!/usr/bin/python3
import numpy as np

# Single linear neuron: y = w·x + b
# Training data: y = 3x + 1
X = np.array([1, 2, 3, 4, 5], dtype=np.float32)
y = np.array([4, 7, 10, 13, 16], dtype=np.float32)

# Initialize parameters
w = 0.0
b = 0.0
lr = 0.01

# Training loop
for epoch in range(100):
    # Forward pass
    y_pred = w * X + b

    # Loss (MSE)
    loss = np.mean((y_pred - y) ** 2)

    # Gradients (by hand)
    dw = np.mean(2 * (y_pred - y) * X)
    db = np.mean(2 * (y_pred - y))

    # Update
    w -= lr * dw
    b -= lr * db

    if epoch % 20 == 0:
        print(f"epoch {epoch}: loss={loss:.4f}, w={w:.4f}, b={b:.4f}")

print(f"\nLearned: y = {w:.2f}x + {b:.2f}")
print(f"Target:  y = 3.00x + 1.00")

After 100 epochs this script prints Learned: y = 3.00x + 1.00 . No magic, no black box — the whole thing is ~25 lines of arithmetic.

🔥 Same thing in PyTorch — let the framework chain-rule for us

Once you have one neuron working, you want to scale. PyTorch (originally led by Meta) is the deep-learning library that makes that scaling tolerable. Its core trick: you write the forward pass, it remembers the chain, and when you call loss.backward() it runs automatic differentiation — applies the chain rule for you across the entire graph.

#!/usr/bin/python3
import torch

# Same data
X = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float32)
y = torch.tensor([4, 7, 10, 13, 16], dtype=torch.float32)

# requires_grad=True tells PyTorch to track gradients
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
lr = 0.01

for epoch in range(100):
    # Forward pass
    y_pred = w * X + b
    loss = torch.mean((y_pred - y) ** 2)

    # Backward pass — PyTorch runs the chain rule for us
    loss.backward()

    # Update (no_grad: don't track this as part of the graph)
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad

    # Zero the gradients (they accumulate by default)
    w.grad.zero_()
    b.grad.zero_()

Notice what disappeared: the hand-derived dw = 2·error·X line. PyTorch's autograd walks the chain for us. That's the entire reason you can train a 175-billion-parameter transformer without going insane — no human is computing ∂L/∂w₁₂₃,₄₅₆,₇₈₉ by hand.

🧩 Terminology sync: "Backpropagation"

Everything we've been doing manually has a name:

• Forward pass — compute the prediction from the inputs.
• Backward pass (backpropagation) — compute the gradient of the loss with respect to every parameter, by applying the chain rule from the loss back to the inputs.
• Dynamic programming — the backward pass reuses intermediate results efficiently instead of recomputing them. Roots in Richard Bellman's work at RAND in the 1950s, originally for aerospace and engineering.

How "backprop" became the default training algorithm

The episode the textbooks skip — Werbos & Minsky

Werbos got the math first, but the field didn't pick it up for another twelve years. The reason is closer to a soap opera than to a research narrative. Werbos walked the algorithm in to Marvin Minsky — whose 1969 Perceptrons book had argued that multi-layer learning was a dead end and helped trigger the first AI winter:

On where the math actually came from, Werbos is unusually direct. He read Freud as a teenager — long before he ever read von Neumann — and decided the shape of the missing algorithm was the one Freud had described informally: the mind allocates and re-allocates "psychic energy" ( cathexis ) to optimize outcomes (the pleasure principle ). Translating that picture into a quantifiable framework — a backwards flow of partial derivatives that assigns blame for the loss back to every weight — is the path that produced the chain rule for ordered derivatives. As Werbos puts it in the same interview:

Sources used in this section: Mind Matters #137 transcript · Werbos AD2004 · Backpropagation and Freud's psychic energy · Werbos on noosphere species theory .

🕸️ Scaling & the computation DAG

When you have one neuron, you can draw the chain on a napkin. When you have millions, you need a Direct Acyclic Graph (DAG) . Every operation is a node; every dependency is an edge. PyTorch builds this graph as your forward pass runs, then walks it backward to compute gradients.

This is a tiny 2-input, 2-hidden-unit, 1-output toy network. A modern transformer's DAG has billions of nodes — but it's the same structure: multiply, sum, activate, repeat. Every node's gradient is computed by looking at its outgoing edges and applying the chain rule.

📐 Scale-up: LeNet-5 (1998) — the first real CNN

Yann LeCun 's LeNet-5 (1998) was the first broadly successful convolutional neural network, trained on the MNIST handwritten-digit dataset. It applied a modified version of backpropagation to a multi-layer convnet and actually worked at a useful scale.

🧮 The memory math — why params become a wall

Parameter count is the easy number. The hard number is memory — how much RAM you need just to hold the model in GPU.

Why is training so much worse? Adam keeps a running mean and variance per parameter (Tab 6). That's 2× the weights, doubled again for FP32 optimizer state, plus activations for the backward pass, plus gradient buffers. Rule of thumb: ~2 GB per 1B parameters at FP16 for inference, ~16 GB per 1B parameters for training.

🕰️ 1998 – 2015 · No memory wall

For the first 17 years of deep learning, models stayed within one consumer GPU's reach. A single card trained what you needed.

AlexNet (2012) is the shot heard round the world — it won ImageNet by a huge margin and convinced the field that GPUs + deep nets were the future. ResNet-50 (2015) proved deeper networks could work with residual connections. Both fit in a gaming GPU.

🤖 2017 – 2019 · Early Transformers — mild signs

Then came the 2017 paper "Attention Is All You Need." Parameters start climbing fast; training memory starts eating real cards.

BERT-Large needs a V100 for training. GPT-2 XL needs the biggest consumer card you can buy. Still one-GPU territory — but only just.

🚀 2020 – 2022 · GPT-3 era — the wall arrives

GPT-3 at 175 billion parameters is the moment single-GPU training dies. You can't hold the weights of GPT-3 on a single GPU, let alone the Adam state.

🌟 2023 – 2025 · Modern LLMs — strong wall

Llama 2 democratized the 7B – 70B range. Frontier models (GPT-4, Llama-3 405B) pushed well into territory where fleets of H100s are the only option.

🛸 2025+ · Frontier & beyond — insane signs

Rumored and theoretical frontier sizes. The numbers start to look unhinged because they are.

🏁 Where this leaves us

Start to finish: a single weight, a single bias, one derivative rule, and the chain rule. That's the whole toolkit. Everything that makes a modern neural network "hard" comes from scaling that toolkit — not from new math.

• The forward pass is w·x + b , stacked and nonlinearly composed.
• The backward pass is the chain rule, run automatically by PyTorch's autograd engine across a DAG.
• The optimizer (SGD, Adam) turns gradients into parameter updates — the same new = old − lr · grad you saw in Tab 6.
• The GPU memory wall is the physical limit on how many parameters and how much optimizer state you can fit per accelerator.

🧮 Memory Wall Explorer — pick any model, any GPU

Slide through parameter counts from a laptop-sized MLP to the theoretical 10T model. Choose a precision. Choose a GPU. See whether it fits — and if not, how many you need.