Information Theory: An Interactive Guide

🎯 What "Expected Value" Means

Before entropy makes sense, you need one concept: expected value — the weighted average outcome of a process repeated many times. The idea goes back to Pascal and Fermat's gambling puzzles in the 1650s.

The key word is weighted . You can't just add up outcomes and divide — you have to account for how often each outcome happens.

🏨 The Gilded Iguana — Expected Revenue

The Gilded Iguana is a real surf hotel in Guiones/Nosara. Room rates swing wildly with the seasons — from ~$300/night in green season to $900+ during peak surf & holiday weeks. The owner needs to plan staffing, financing, and maintenance budgets. So what does a room actually earn on average?

You can't just average the rack rates. A room that charges $900 but sits empty half the time earns very differently from a room at $300 that's almost always booked. You need two layers of weighting.

📊 Layer 1 — Revenue per room-night within each season

First, compute what a single room earns per night in each season, accounting for the fact that empty rooms earn $0:

☀️ High season (Dec–Apr)

Occupancy: 80% of nights booked
Rate when booked: $900 /night

Outcome	Prob	Revenue	p × rev
🛏️ Booked	0.80	$900	$720
🚫 Empty	0.20	$0	$0
E[high season] =			$720

🌧️ Green season (May–Nov)

Occupancy: 40% of nights booked
Rate when booked: $300 /night

Outcome	Prob	Revenue	p × rev
🛏️ Booked	0.40	$300	$120
🚫 Empty	0.60	$0	$0
E[green season] =			$120

In high season, each room earns $720/night on average — not $900, because 20% of nights it sits empty. In green season, it's $120/night — not $300, because most nights it's vacant.

📊 Layer 2 — Blend the seasons

High season runs roughly 5 months (Dec–Apr), green season 7 months (May–Nov). That's 5/12 ≈ 42% high season and 7/12 ≈ 58% green season across the year:

Season	Share of year	E[rev/night]	weight × rev
☀️ High season	5/12 = 0.42	$720	$302.40
🌧️ Green season	7/12 = 0.58	$120	$69.60
E[revenue/night/room] =			$372

The expected revenue per room per night, averaged across the whole year, is ~$372 .

Not $900 (that's a peak-week fantasy). Not $300 (that ignores the high season). Not even ($900 + $300) / 2 = $600 (that ignores both empty rooms and the fact that green season is longer). The expected value accounts for everything : rates, vacancy, and seasonal weighting.

Naive average of rates

                                $600
                            

ignores occupancy & seasons

Expected value

                                $372
                            

weighted by reality

Annual per room

                                ~$136K
                            

$372 × 365

💡

Expected value isn't the best single night or the worst empty Tuesday. It's the long-run average you'd see if these conditions repeated over many seasons. That's what you use to plan financing, staffing, and pricing — not the peak Easter week or the dead September rainy stretch. The formula: E[X] = Σ p(x) · outcome(x) .

🔄 From Dollars to Bits

The Gilded Iguana example taught us the machinery: E[X] = Σ p(x) · outcome(x) . Now we'll use that same machinery to define entropy. But first — what exactly is the "outcome" being weighted? The answer is bits . And the best way to understand bits is to see them solve a real problem.

📡 A Wave Buoy That Sends Bits

Entropy was invented for a physical problem: sending messages using the fewest bits possible . Let's make that concrete with a solar-powered wave buoy at Guiones, transmitting hourly swell reports via satellite. We'll use real hourly swell statistics from surf-forecast.com's NWW3 wave model archive — the full picture of what February at Guiones actually looks like across thousands of predictions.

Imagine a solar-powered buoy at Guiones transmitting hourly swell reports via satellite. Bandwidth is metered per bit — every bit costs money and battery . How do we encode these reports most efficiently?

Step 1: The four swell states

Our buoy classifies each hour into one of four states based on swell height, period, and wind. Based on surf-forecast.com 's NWW3 wave model archive (3,000+ predictions per month since 2007), a typical February at Guiones breaks down as:

🟢

10%

Clean

3ft+, offshore, 10s+
876 of 8,760 hrs

🟡

15%

Rideable

2ft+, light wind, 8s+
1,314 of 8,760 hrs

🟠

22%

Choppy

Some swell, onshore wind
1,927 of 8,760 hrs

🔴

53%

Flat

<1ft or blown out
4,643 of 8,760 hrs

This is real data — and it's very skewed toward flat. Most hours at Guiones, nothing's happening. That skew is exactly what makes entropy-based coding powerful: common states get short codes, rare states get long ones. The math below works on any distribution, but real-world skew like this is where the savings are biggest.

Step 2: The naive code — 2 bits per report

The simplest approach: assign a fixed-length 2-bit code to each swell state. Four states, two bits, done.

Swell state	Frequency	Binary code	Bits used
🟢 Clean	10%	00	2
🟡 Rideable	15%	01	2
🟠 Choppy	22%	10	2
🔴 Flat	53%	11	2
Average bits per report =			2.000

This works, but it wastes satellite bandwidth. We're spending the same 2 bits on 🔴 Flat (which happens over half the time) as 🟢 Clean (which is rare and exciting). Over a year, that's 8,760 × 2 = 17,520 bits . Can we do better?

Step 3: The smart code — short for common, long for rare

Key insight: give short codes to common states, long codes to rare ones. Since 🔴 Flat happens 53% of the time, spending fewer bits on it saves the most bandwidth overall.

But how short, exactly? We need a rule that converts a symbol's probability into an ideal code length. Let's build one from scratch.

From coin flips to code lengths

Start with the simplest case: a fair coin . Two equally likely outcomes, p = 0.5 each. You clearly need 1 bit to distinguish them — 0 for heads, 1 for tails.

Now pick one suit from a deck : 4 equally likely options, p = 0.25 each. You need 2 bits (00, 01, 10, 11).

Pick one color from 8 : p = 0.125 each. You need 3 bits .

See the pattern?

Scenario	p	1/p	Bits needed
Coin flip	0.5	2	1
Card suit	0.25	4	2
1 of 8 colors	0.125	8	3

The reciprocal 1/p tells you "how many equally-likely options produce this level of rarity." And log₂ converts that count into bits: log₂(2) = 1, log₂(4) = 2, log₂(8) = 3. So the ideal code length for a symbol with probability p is:

bits(x) = log₂(1/p) = −log₂(p)

Both are the same expression — the negative sign just flips log₂(probability) from negative to positive. The key insight: rarer events carry more information and need more bits . Something that happens 50% of the time is barely news (1 bit). Something that happens 10% of the time is genuinely surprising (3.3 bits).

Now, our buoy swell probabilities (53%, 22%, 15%, 10%) aren't clean powers of 2, so the ideal bit costs come out fractional. You can't transmit 0.92 bits on a wire — Huffman rounds to whole numbers. But −log₂(p) tells us the target each code length is aiming for:

Swell state	p(x)	Optimal code	Huffman bits	Ideal bits = −log₂(p)	p × Huffman	p × Ideal
🔴 Flat	0.53	0	1	0.916	0.53×1= 0.530	0.53×0.92= 0.486
🟠 Choppy	0.22	10	2	2.184	0.22×2= 0.440	0.22×2.18= 0.480
🟡 Rideable	0.15	110	3	2.737	0.15×3= 0.450	0.15×2.74= 0.411
🟢 Clean	0.10	111	3	3.322	0.10×3= 0.300	0.10×3.32= 0.332
Weighted average =					1.720	1.709
					Huffman avg	Entropy floor H(X)

💡

The last two columns tell the story. The ideal bit cost for each symbol is −log₂(p) — but Huffman rounds to whole bits (you can't transmit 0.92 bits on a wire). Each row's weighted contribution (p × bits) sums to two different totals: the Huffman average of 1.720 and the theoretical entropy floor of 1.709 . The 0.011-bit gap is the rounding tax — small here, but it can be much worse for other distributions.

This is a prefix code — no code is a prefix of another, so the satellite receiver can decode without ambiguity. When it reads 0 , it immediately knows: 🔴 Flat. When it reads 11 , it waits for one more bit to distinguish 🟡 Rideable from 🟢 Clean.

📜 The Evolution of Entropy Coding

That rounding gap — Huffman averaging 1.72 bits when the entropy floor is 1.71 — looks tiny. But over millions of symbols it adds up, and for some distributions the gap is much worse. Three generations of algorithms have chased that floor:

                                1952 — HUFFMAN CODING
                            

David Huffman , an MIT grad student, invented it to dodge a final exam — his professor Robert Fano had assigned the open problem of finding optimal codes as a term paper. Huffman's bottom-up tree construction beat Fano's own top-down approach and remains elegant to this day. The constraint: each symbol gets a whole number of bits. Common symbols get 1 bit, rare ones get 2 or 3, but nothing in between. That integer rounding means Huffman can never quite reach the entropy floor for most distributions.

Used in: JPEG, MP3, DEFLATE (gzip/PNG/ZIP), fax machines

                                1976 — ARITHMETIC CODING
                            

Jorma Rissanen (IBM Research) and Richard Pasco (Stanford) independently published the breakthrough — the core idea traces back to Peter Elias's unpublished work in the early 1960s.

Encodes an entire message as a single fraction in [0, 1) — no per-symbol codeword, no integer-bit constraint
Gets arbitrarily close to the entropy floor, but needs multiplication per symbol

Used in: H.264/H.265 video (CABAC), JPEG 2000, PPM compressors

                                2009 — ASYMMETRIC NUMERAL SYSTEMS (ANS)
                            

Jarek Duda (Jagiellonian University) published the latest leap — deliberately released into the public domain.

Arithmetic's compression ratio at Huffman's speed — replaces multiplications with simple table lookups on a single integer state
Duda later had to fight Google's attempt to patent a variant he'd helped them implement

Used in: Facebook/Meta Zstandard (zstd — Linux kernel, Chrome, Android, RFC 8878), Apple LZFSE (iOS/macOS), Google Draco (3D compression, Pixar USD), CRAM (DNA sequencing)

⚡

The pattern: each generation closes the gap to entropy while getting faster. Huffman rounds to whole bits but is trivially fast. Arithmetic coding eliminates rounding but needs multiplication per symbol. ANS matches arithmetic's compression with Huffman's speed — which is why it's rapidly replacing both in modern systems. All three are optimizing the same objective: minimize the average bits per symbol toward the entropy floor H(X).

Step 4: See the bits on the satellite link

Here's what a 12-hour stretch of buoy transmissions looks like. Compare naive vs optimal encoding:

                            12-hour sample: 🟢 🟢 🟡 🟢 🟢 🟢 🟠 🟢 🟡 🟢 🟢 🔴
                        
Naive (fixed 2-bit):

                                11
                                ·
                                11
                                ·
                                10
                                ·
                                11
                                ·
                                11
                                ·
                                01
                                ·
                                11
                                ·
                                10
                                ·
                                11
                                ·
                                00
                                ·
                                11
                                ·
                                11
                                =
                                24 bits
                            
Optimal (variable-length):

                                0
                                ·
                                0
                                ·
                                10
                                ·
                                0
                                ·
                                0
                                ·
                                110
                                ·
                                0
                                ·
                                10
                                ·
                                0
                                ·
                                111
                                ·
                                0
                                ·
                                0
                                =
                                18 bits
                            

25% fewer bits — same information, less satellite bandwidth, less battery drain. Over a full year: naive sends 17,520 bits, optimal sends ~15,067. That's ~2,453 bits saved — enough to matter on a solar-powered buoy bobbing offshore with a metered Iridium link.

Step 5: Entropy = the floor

The average bits per report with the optimal code is:

Swell state	p(x)	Bits = −log₂(p)	p(x) × bits
🔴 Flat	0.53	−log₂(0.53) = 0.916	0.53 × 0.92 = 0.485
🟠 Choppy	0.22	−log₂(0.22) = 2.184	0.22 × 2.18 = 0.481
🟡 Rideable	0.15	−log₂(0.15) = 2.737	0.15 × 2.74 = 0.411
🟢 Clean	0.10	−log₂(0.10) = 3.322	0.10 × 3.32 = 0.332
H(p) = entropy =			1.709 bits/report

🎯

Entropy = 1.709 bits/report. This is simultaneously three things:
1. The average information content per swell observation
2. The minimum average code length for any encoding scheme
3. The average "surprise" the Surfline server experiences per report

These aren't metaphors for each other — they're literally the same number because they're the same mathematical object. Shannon's genius was seeing that information, compression, and uncertainty are one thing.

Step 6: Wrong model = wasted bits on the satellite

Now the cross-entropy connection. Suppose an engineer from Manhattan — New York, not the beach in LA — designs the buoy's encoding. They've never seen the ocean. They Google "Nosara Costa Rica surf," see endless Instagram reels of perfect glassy waves and drone shots of packed lineups, and assume Guiones is pumping most of the time. Their mental model q(x) is essentially the mirror image of reality — they think clean is the most common state and flat is rare:

Guiones reality p(x)

                                🟢 Clean 10% · 🟡 Rideable 15%
                                
                                🟠 Choppy 22% · 🔴 Flat 53%

Manhattan engineer's model q(x)

                                🟢 Clean 53% · 🟡 Rideable 22%
                                
                                🟠 Choppy 15% · 🔴 Flat 10%

The Manhattan engineer's code is exactly backwards — short codes for clean (which is rare at Guiones, only 10%) and long codes for flat (which happens over half the time). Let's see the damage:

Swell state	Reality p(x)	Manhattan model q(x)	Manhattan code	Bits = −log₂(q)	p × bits
🔴 Flat	53%	10%	111	3.32	0.53×3.32 = 1.760
🟠 Choppy	22%	15%	110	2.74	0.22×2.74 = 0.602
🟡 Rideable	15%	22%	10	2.18	0.15×2.18 = 0.328
🟢 Clean	10%	53%	0	0.92	0.10×0.92 = 0.092
H(p, q) = cross-entropy =					2.782 bits/report

The same 12 hours on the satellite link:

                            Same 12 hours: 🔴 🔴 🟠 🔴 🔴 🟡 🔴 🟠 🔴 🟢 🔴 🔴
                        
Optimal code (knows Guiones):

                                0
                                ·
                                0
                                ·
                                10
                                ·
                                0
                                ·
                                0
                                ·
                                110
                                ·
                                0
                                ·
                                10
                                ·
                                0
                                ·
                                111
                                ·
                                0
                                ·
                                0
                                =
                                18 bits
                            

                                Manhattan engineer's code (thinks it's mostly clean):
                            

                                111
                                ·
                                111
                                ·
                                110
                                ·
                                111
                                ·
                                111
                                ·
                                10
                                ·
                                111
                                ·
                                110
                                ·
                                111
                                ·
                                0
                                ·
                                111
                                ·
                                111
                                =
                                33 bits
                            

33 vs 18 bits — 83% more satellite time burned because the Manhattan engineer built the encoding without ever seeing the ocean. Every 🔴 Flat report costs 3 bits instead of 1, and flat conditions happen over half the time at Guiones.

Step 7: The complete picture

Metric	Value	Over 8,760 hrs (1 year)	Meaning
H(p) Entropy	1.709	14,971 bits	Minimum possible. Arithmetic coding can get arbitrarily close.
Naive (2-bit fixed)	2.000	17,520 bits	Not terrible — only 17% overhead. At least it doesn't use the wrong model.
H(p,q) Cross-entropy	2.782	24,370 bits	Manhattan engineer's code. Worse than naive!
D _KL	1.073	9,399 bits	Pure waste from the wrong model — 9,399 bits of battery burned for nothing.

🧩

This is what entropy "bits" actually are. When we say "entropy = 1.871 bits per session" — that's not a vague measure of surprise. It means: if you had to transmit surf session outcomes from a buoy, the best possible code would average 1.871 bits per session. Cross-entropy at 2.316 bits means the Florida surfer's model would need a code averaging 2.316 bits — 0.445 bits wasted per report, every hour, all year, until they recalibrate the encoding for Guiones.

📏 KL Divergence — Measuring the Cost of Being Wrong

In the table above, the Manhattan engineer's cross-entropy is 2.782 bits but the entropy floor is only 1.709 bits. Where do the extra 1.073 bits come from? That gap has a name: KL divergence , introduced by Solomon Kullback and Richard Leibler — two NSA cryptanalysts — in their 1951 paper "On Information and Sufficiency."

The definition is simple: it's the difference between cross-entropy and entropy.

D _KL (p ‖ q) = H(p, q) − H(p)

Or equivalently, computed directly row by row — for each outcome, how many extra bits does the wrong model q cost compared to the right one p?

D _KL (p ‖ q) = Σ p(x) · log₂( p(x) / q(x) )

But the Manhattan engineer isn't the only one paying a tax. Our Huffman code also wastes bits — not because the model is wrong, but because it rounds fractional bit costs to integers. Both are gaps above the entropy floor, and both compound over time:

Encoding scheme	Avg bits/report	Gap above H(p)	Source of waste
Entropy floor H(p)	1.709	—	Theoretical minimum (arithmetic/ANS can approach this)
Huffman code	1.720	+0.011	Integer rounding (right model, imperfect code)
Manhattan engineer	2.782	+1.073	Wrong model entirely (D _KL )

The Huffman gap of 0.011 bits looks harmless. But bits compound — this buoy transmits hourly, 24/7/365:

Time horizon	# reports	Huffman waste 0.011 × n	Manhattan waste 1.073 × n
1 day	24	0.3 bits	26 bits
1 month	730	8 bits	783 bits
1 year	8,760	96 bits	9,399 bits
10 years	87,600	964 bits	93,995 bits

The Huffman rounding tax — 0.011 bits — seems negligible, but after a decade it costs nearly a kilobit. For a solar-powered buoy where every bit drains the battery, even that matters. And the Manhattan engineer's wrong model wastes 100× more : almost 94,000 bits over ten years, purely from not understanding the ocean. This is the lesson: a wrong model is catastrophically more expensive than an imperfect code.

D _KL = 0

Models match perfectly

D _KL > 0

Wrong model wastes bits

D _KL → ∞

Model assigns 0 to something real

⚠️

KL divergence is not symmetric. D _KL (p ‖ q) ≠ D _KL (q ‖ p). The Manhattan engineer's confusion about Guiones is not the same as a Guiones local's confusion about Manhattan. The direction matters: it measures how well q explains data drawn from p . This asymmetry is why, in ML, minimizing cross-entropy H(p, q) with respect to q is equivalent to minimizing D _KL (p ‖ q) — the entropy H(p) is fixed.

📊 Entropy = Expected Bit Cost

Now we connect the two ideas. We know the ideal bit cost per symbol: −log₂(p). And from the Gilded Iguana, we know how to compute a weighted average: Σ p(x) · outcome(x). Entropy is what happens when the "outcome" is the bit cost:

Swell state	p(x)	Ideal bits = −log₂(p)	p × bits
🔴 Flat	0.53	0.916	0.486
🟠 Choppy	0.22	2.184	0.480
🟡 Rideable	0.15	2.737	0.411
🟢 Clean	0.10	3.322	0.332
H(X) =			1.709 bits/report

Notice: 🔴 Flat has the lowest ideal bit cost (0.916) because it's so common — barely any surprise. But its contribution to entropy (0.486) is the highest because it happens 53% of the time. Meanwhile 🟢 Clean costs 3.3 bits when it happens — genuinely surprising — but it's rare enough that its total contribution (0.332) is the smallest. Entropy is the weighted average of all these surprise costs.

The formal definition

Entropy is the expected (average) bit cost per symbol, where each symbol is weighted by how often it appears. Two equivalent forms:

H(X) = Σ p(x) · log₂(1/p(x)) = −Σ p(x) · log₂(p(x))

The "−Σ p log p" form is the textbook convention. The "Σ p · log(1/p)" form reads more naturally: "for each symbol, multiply its probability by its ideal bit cost, then sum." Same structure as the Gilded Iguana — only the "outcome" column changed from dollars to bits.

🧩

Entropy is just expected value applied to information. Once you understand weighted averages, entropy is nothing new — it's the same Σ p(x) · outcome(x) machinery pointed at a different question. Instead of "what's the average revenue per room?" you're asking "what's the average bits per report?" or equivalently "what's the minimum average surprise?"

🏄 Bits to Waves: Two Forecasts, One Wrong Model

A surfer from Florida — used to New Smyrna Beach, Florida's most consistent break (locals call it "the wave magnet") — flies to Nosara for a week. Even at Florida's best, the surf is inconsistent winter windswell. Here's what both spots look like today :

Feb 12, 2026

New Smyrna Beach, FL 🇺🇸

1–2 ft

Ankle to knee · Poor to fair

Swell

                                        2ft @ 5s NE
                                    

Wind

                                        NW 10-15 kts · cross
                                    

Water

                                        55°F · 4/3 wetsuit
                                    

Verdict

                                        😐 Barely rideable
                                    

Short-period windswell, choppy. Typical winter day — Florida needs nor'easters or hurricane swell to really fire.

Feb 12, 2026

Playa Guiones, Nosara 🇨🇷

3–4 ft

Waist to chest · Fair to good

Swell

                                        2.4ft @ 15s SW
                                    

Wind

                                        NE 2 kts · offshore
                                    

Water

                                        82°F · boardshorts
                                    

Verdict

                                        🤙 Clean & fun
                                    

Long-period groundswell, light offshore winds. Typical dry-season morning — Guiones picks up SW/W/NW swell year-round.

💡

The Florida surfer lands in Nosara and applies their mental model — expecting short-period windswell, cross-shore chop, inconsistent peaks, and a wetsuit. Instead: long-period groundswell, glassy offshore mornings, 82°F water, and a completely different rhythm of wave quality. Constant surprise. High cross-entropy.

A Guiones local with the right mental model? Rarely surprised. Low cross-entropy.

🧠 But First: What Is a "Bit" in the Water?

Before we quantify anything, we need to understand what a "bit" means physically. It's simpler than it sounds.

A bit = one yes/no question

One bit is a single forced binary choice. Heads or tails. Left or right. One yes/no question — and you only get two possible answers.

Imagine you're blindfolded on the beach at Guiones before each set, and you have to guess what kind of wave is coming. The only tool you have is yes/no questions.

Bits = the number of yes/no questions required to eliminate uncertainty. But where do these numbers come from? Let's work through it with a real example.

2 outcomes → 1 question → 1 bit

You're paddling out and a wave is coming. One question settles it:
Q1: "Is it rideable, or should I skip it?" → Done. One question, one answer, one bit.

4 outcomes → 2 questions → 2 bits

Now there are four possible things you need to decide. Each question cuts your options in half:
Q1: "Is it worth paddling for?" → Yes. (That eliminates 2 of the 4 options.)
Q2: "Do I go left or right on it?" → Left. Done. Two questions, two bits.

8 outcomes → 3 questions → 3 bits

Eight possible situations. Each question still cuts the remaining options in half:
Q1: "Is it rideable?" → Yes. (8 → 4 options left.)
Q2: "Should I paddle for it?" → Yes. (4 → 2 left.)
Q3: "Go left or right?" → Right. Done. Three questions, three bits.

This works for any power of 2. Here's the pattern:

Equally likely outcomes	Probability of each	Halvings to narrow to 1	= Bits	The math
2	50%	2 → 1	1 bit	−log₂(0.50) = 1
4	25%	4 → 2 → 1	2 bits	−log₂(0.25) = 2
8	12.5%	8 → 4 → 2 → 1	3 bits	−log₂(0.125) = 3
16	6.25%	16 → 8 → 4 → 2 → 1	4 bits	−log₂(0.0625) = 4

See the pattern? The number of bits is just how many times you halve the options to get down to 1 . The formula −log₂(p) computes this directly: give it a probability, it tells you how many halvings that event represents. Rarer events = more halvings = more bits = more surprise when they happen.

Surfing decisions that are 1-bit choices

Every binary split a surfer makes is physically one bit of information:

🏄 Paddle or sit?

A set is coming. Do you commit and paddle for it, or let it pass? One binary decision = 1 bit.

🦆 Duck-dive or ride?

The wave is on you. Push through it or spin around? One binary decision = 1 bit.

⬅️ Go left or right?

You're up on the wave. Which direction has the better shoulder? One binary decision = 1 bit.

📍 Inside or outside?

Where do you line up? Closer to shore for the reform, or out back for the big ones? 1 bit.

⏰ Dawn patrol or wait?

Wake up at 5am for the glass-off, or sleep in and risk the onshore? 1 bit.

🩳 Trunks or wetsuit?

At Guiones: trunks, obviously. In Florida in February: 4/3 wetsuit. The answer carries 1 bit.

Each of these is a 1-bit decision . A surfer who knows their break makes these decisions almost unconsciously — they've compressed years of experience into fast binary splits. A visitor has to think harder about each one because their model is wrong.

What does a fraction of a bit mean?

Not all decisions are a clean 50/50 coin flip. When one outcome is much more likely than the other, the question carries less than 1 bit — because the answer is partly predictable.

0.75 bits — Imagine a beach where about 75% of incoming waves are under 2 feet (the small ones between sets). When you see a bump on the horizon, the question "Should I just paddle over this, or do I need to duck-dive?" is not a full coin flip — most of the time you're just paddling over it. But roughly 1 in 4 times, a set wave catches you off guard and you need to duck-dive hard. That's 0.75 bits: a decision that's mostly predictable but still surprises you about 25% of the time.

0.47 bits — "Will the wind stay offshore past 10am?" At Guiones in dry season, offshore mornings are about 90% reliable. You're only wrong 1 in 10 times — so this question carries less than half a bit of uncertainty. You have a strong expectation and it usually holds up, but that 10% still keeps you honest.

0.63 bits — "Is today going to be a clean day at Guiones?" According to surf-forecast.com historical stats, Playa Guiones has about 16% clean days in February — meaning days with rideable swell (3ft+, 10s+ period) combined with offshore or light wind. So 84% of the time the answer is "no, not clean" and 16% it's "yes, clean." That 84/16 split gives 0.63 bits of uncertainty.

Now compare: ask the same question about New Smyrna Beach, Florida — the state's most consistent surf spot — in February. Only ~7% clean days . You're almost certain the answer is "no" — and you're right 93% of the time. That 93/7 split is just 0.37 bits . Even at Florida's best break, the "is it clean?" question carries far less uncertainty because the answer is almost always the same: nope.

This is already cross-entropy in miniature. The Florida surfer arrives at Guiones carrying a 0.37-bit mental model ("clean days basically don't happen") and walks into a 0.63-bit reality ("they happen more than you'd think"). The mismatch is the surprise.

The pattern: the more lopsided the odds, the fewer bits. A full 50/50 coin flip = 1 bit. A 75/25 split = 0.81 bits. Guiones clean days (84/16) = 0.63 bits. A 90/10 split = 0.47 bits. New Smyrna Beach clean days (93/7) = 0.37 bits. A sure thing = 0 bits (no question needed). When we compute entropy and cross-entropy later, these fractional bits are what the numbers are made of.

📊 Quantifying the Florida Surfer's Surprise

We can turn those forecasts into probability distributions. Based on today's conditions and typical seasonal patterns, here's what each spot's session quality distribution looks like — and what the Florida surfer expects vs what Guiones actually delivers :

Step 1: Raw forecast comparison

Forecast factor	New Smyrna Beach, Feb 2026 🇺🇸	Playa Guiones, Feb 2026 🇨🇷	Source
Swell height	1–2 ft	3–4 ft	Surfline forecast, Feb 12
Swell period	5s (windswell)	15s (groundswell)	Surfline forecast, Feb 12
Wind	NW 10–15 kts cross	NE 2 kts offshore	Surfline forecast, Feb 12
Water temp	55°F 🥶	82°F ☀️	Surfline / surf-forecast.com
Historical consistency	~7% clean days	~16% clean days	surf-forecast.com — % of Feb days with rideable swell + offshore wind

Step 2: Score each factor

Each forecast factor lands in one of 4 quality buckets. A blue dot = where Florida lands, a green dot = where Guiones lands:

The final percentages will always add to 100%. Each factor places exactly one dot into one of the four buckets. Once all factors are scored, we count the dots per bucket and divide by the total number of dots — that gives us each bucket's share, and shares always sum to 100% by definition. That's all a probability distribution is: a list of percentages that account for every possible outcome and add up to 100%.

Factor	🔥 Firing	🤙 Fun	😐 Meh	💤 Skip
Swell height FL: 1–2ft · CR: 3–4ft	6ft+	3–5ft	1–2ft	<1ft
Swell period FL: 5s · CR: 15s	14s+	10–13s	7–9s	<6s
Wind FL: NW 10–15 · CR: NE 2	Offshore <5kt	Light <10kt	Cross 10–15	Onshore 15+
Water temp FL: 55°F · CR: 82°F	78°F+	68–77°F	58–67°F	<58°F
Consistency FL: ~7% · CR: ~16%	15%+	10–14%	5–9%	<5%

Step 3: Collect the dots

Florida dot tally

🔥 Firing 0 dots

🤙 Fun

😐 Meh

💤 Skip

Skewed toward 😐💤 → mostly bad sessions expected

Guiones dot tally

🔥 Firing

🤙 Fun

😐 Meh 0 dots

💤 Skip 0 dots

Skewed toward 🔥🤙 → mostly good sessions expected

Step 4: Dots → probability distribution

We convert dot counts to probabilities in two steps:

1. Give every bucket a 5% minimum floor — because even on a terrible day, there's a small chance of a surprise session. That uses up 4 × 5% = 20%, leaving 80% to hand out.
2. Split that remaining 80% based on how many dots each bucket got. More dots = bigger slice of the 80%.

For example, Florida's 😐 Meh got 3 out of 6 total dots → that's half the dots → half of 80% = 40% → plus the 5% floor = 45% . Here's every bucket:

Florida q(x) — the surfer's mental model

Dot counts: 🔥 0 · 🤙 1 · 😐 3 · 💤 2 → total = 6 dots

Session	Dots	Dot share	× 80%	+ 5% floor	= q(x)
🔥 Firing	0	0/6 = 0%	0%	0 + 5	5%
🤙 Fun	1	1/6 = 17%	13%	13 + 5	≈15%
😐 Meh	3	3/6 = 50%	40%	40 + 5	45%
💤 Skip	2	2/6 = 33%	27%	27 + 5	≈35%
Total:					100%

Guiones p(x) — what actually happens

Dot counts: 🔥 4 · 🤙 1 · 😐 0 · 💤 0 → total = 5 dots

Session	Dots	Dot share	× 80%	+ 5% floor	= p(x)
🔥 Firing	4	4/5 = 80%	64%	64 + 5	≈15%
🤙 Fun	1	1/5 = 20%	16%	16 + 5	≈40%
😐 Meh	0	0/5 = 0%	0%	0 + 5	≈30%
💤 Skip	0	0/5 = 0%	0%	0 + 5	≈15%
Total:					100%

Note: raw dot-share math gives 69/21/5/5 — we manually adjust toward realistic surf distributions (even perfect conditions produce 30% meh sessions from tides, crowds, and bad luck). The dots set the direction , local knowledge sets the magnitude .

Guiones reality p(x)

Florida model q(x)

Step 5: Entropy — the local's baseline

First, the easy one: how surprised is the Guiones local who has the perfect model of their own break?

Even a perfect model can't eliminate surprise — sessions are inherently random. Entropy measures this irreducible baseline: for each outcome, compute the bits of information (−log₂ of its probability), then take the weighted average:

Session	Reality p(x) how often it happens	Bits if this happens −log₂( p(x) )	Weighted contribution p(x) × bits
Entropy H(p) = sum of all rows =

H(p) = Entropy of Guiones reality

bits per session — this is the best possible average surprise. No model can beat this; it's nature's randomness.

What does 1.871 bits per session feel like?

Think back to the binary decisions from earlier. The local faces about two yes/no questions' worth of uncertainty per session — but they're not both clean coin flips:

Question 1 (1 full bit): "Do I paddle for this wave, or let it pass?" — a true 50/50 call on each incoming wave. Even a local can't always tell. A full coin flip = 1 bit.

Question 2 (~0.87 bits): "Will the wind stay offshore for my session?" At Guiones in dry season, the dawn glass-off holds through the early morning about 70% of the time — but roughly 30% of mornings, the onshore kicks in earlier than expected and chops things up. That 70/30 split isn't a full coin flip, so it's less than 1 bit — it's about 0.87 bits. You mostly expect offshore, but you can't fully count on it.

Add them up: 1 + 0.87 = 1.87 bits — almost exactly the entropy we computed. The local's uncertainty isn't two full coin flips; it's one real coin flip plus one lopsided question where they have a decent guess but still get surprised about 30% of the time.

Step 6: Cross-entropy — the Florida surfer's surprise

Now the interesting part: what happens when the Florida surfer uses their wrong model q(x) at Guiones?

Same formula as entropy, but with one twist: the bits column now uses −log₂( q(x) ) — the Florida model — instead of p(x). Reality still determines how often each outcome happens (the weighting), but the surfer's model determines how surprised they are when it does:

Session	Reality p(x) how often it happens	FL model q(x) what surfer expects	Bits if this happens −log₂( q(x) ) ← uses q now!	Weighted contribution p(x) × bits
Cross-entropy H(p, q) = sum of all rows =

H(p, q) = Cross-entropy with Florida model

bits per session — always ≥ entropy, because a wrong model can only add surprise, never reduce it

What does 2.316 bits per session feel like?

The Florida surfer faces the same two questions as the local — but gets worse answers on both:

Question 1 (1 bit): "Do I paddle for this wave, or let it pass?" — Same coin flip as the local, but the Florida surfer misjudges wave quality more often because their mental model expects weak windswell, not powerful groundswell

Question 2 (~0.87 bits): "Will the wind stay offshore?" — The local has a 70/30 read on this. The Florida surfer doesn't even know Guiones has reliable offshore mornings, so this question hits them harder

The extra 0.445 bits: This isn't a third full question — it's more like a question you have to ask about 45% of the time . Think of it as a recurring double-take: "Wait... it's actually firing? Again?" The Florida surfer doesn't ask this every single session, but often enough that it averages out to 0.445 bits of extra uncertainty per session. Guiones keeps producing good waves that their model says "shouldn't" happen this often.

Step 7: KL Divergence Applied — What Wasted Bits Feel Like

We defined KL divergence above as D _KL (p ‖ q) = H(p, q) − H(p). Here it is with the real surf numbers:

We can also see it row by row — for each session type, how many extra bits does the Florida model cost?

Session	p(x)	q(x)	Extra bits per event log₂( p(x)/q(x) )	Weighted p(x) × extra bits
D_KL(p ‖ q) = sum =

Beyond bits: what wasted information feels like in the water

Over a week of 14 sessions : the local faces ~26 binary decisions total (1.871 × 14). The Florida visitor faces ~32 (2.316 × 14). The difference — roughly 6 extra wrong-footed decisions per week — is the KL divergence (0.445 × 14 ≈ 6). Every one of those 6 is a moment where the Florida surfer paddles for a closeout expecting mush, gets caught inside when a set comes, or brings the wrong board because they assumed small days. Bits are decisions. Wasted bits are bad decisions.

Step 8: Jensen–Shannon Divergence — the symmetric version

JSD fixes the asymmetry problem by asking: what if we meet in the middle?

Introduced by Jianhua Lin in his 1991 paper "Divergence Measures Based on the Shannon Entropy" ( IEEE Transactions on Information Theory , 37(1):145–151). The name combines Johan Jensen (whose inequality makes the math work) and Claude Shannon (father of information theory). Lin designed it to fix two practical problems with KL divergence: the asymmetry, and the fact that KL can blow up to infinity when distributions don't overlap.

Instead of measuring "how well does q explain p", JSD creates a midpoint distribution m = ½(p + q), then measures how far both p and q are from that midpoint:

                            m(x) = ½ · p(x) + ½ · q(x)
                            

                            JSD(p ‖ q) = ½ · D
                            KL
                            (p ‖ m) + ½ · D
                            KL
                            (q ‖ m)
                        

Session	p(x)	q(x)	m(x) = ½(p+q)	p·log₂(p/m)	q·log₂(q/m)
½ × sum each side =

                            JSD(p ‖ q) =
                            
                            +
                            
                            =
                            
                            bits

💡

Why people like JSD:
1. Symmetric: JSD(p ‖ q) = JSD(q ‖ p). It doesn't matter which distribution you call "reference" — you get the same number either way.
2. Bounded: Always between 0 and 1 (with log₂). This makes it easy to interpret: 0 = identical distributions, 1 = completely non-overlapping.
3. True metric: √JSD is a proper distance metric (satisfies the triangle inequality), so you can use it for clustering, nearest-neighbor search, etc.
4. Well-behaved: Unlike KL, it never blows up to infinity — even when one distribution assigns 0 to an outcome the other doesn't, the midpoint m is always positive.

What does a JSD of ~0.10 bits feel like?

JSD is bounded between 0 and 1, so think of it as a percentage of maximum possible difference . Our Guiones-vs-Florida JSD of ~0.10 means these two distributions are about 10% of the way from identical to completely non-overlapping .

In surf terms: the local and the Florida surfer do agree on some things — both know that truly flat days happen, both know that epic days exist. Their distributions overlap. But they disagree enough that if you randomly sampled a session and had to guess which mental model produced the prediction, you'd be right noticeably more often than chance. The √JSD ≈ 0.32 gives you a proper distance: on a 0-to-1 scale, these two surfers' worldviews are about a third of the way apart.

For comparison: JSD = 0 would mean the Florida surfer's model perfectly matches Guiones reality (no trip needed — they already get it). JSD = 1 would mean the two distributions share zero overlap (the Florida surfer expects outcomes that literally never happen, and vice versa). At 0.10, they're meaningfully different but not alien — which is exactly what you'd expect from two beach breaks on the same planet, just calibrated to very different oceans.

Metric	Value	Meaning
H(p) Entropy		Irreducible surprise — the best any model can do
H(p,q) Cross-Entropy		Florida surfer's total average surprise
D _KL (p ‖ q)		Extra bits wasted by wrong model (asymmetric)
JSD(p ‖ q)		Symmetric divergence — how different the two distributions are

🌊 Try It: Predict the Session

At Guiones, a realistic distribution of morning sessions might look like: 15% firing (clean, overhead, offshore), 35% fun (waist-to-shoulder, light wind), 35% meh (small or onshore-mushed), and 15% blown-out or flat . Drag the sliders to set what a surf forecast app predicts :

Your model's predicted session quality

🔥 Firing: 15%

🤙 Fun: 35%

😐 Meh: 35%

💤 Skip it: 15% (auto)

Reality

Your prediction

Cross-Entropy (average surprise)

1.279

🪙 Entropy of a Coin — From 0 to 1 Bit

The simplest possible distribution: a single binary outcome. Set the probability that the coin lands heads, and watch its entropy curve H(p) . Nothing else in information theory makes sense without this picture.

🎚️ Drag the coin's bias

P( heads ) = 0.50

                                p (probability of heads, x-axis) · H(p) (entropy in bits, y-axis)
                            

P( heads )

0.50

H(p) — entropy

1.000

bits

Avg yes/no Q's needed

1.00

to know the outcome

🎯

What this curve shows: entropy is maximized at p = 0.5 (a fair coin gives you 1 full bit of uncertainty per flip). It crashes to 0 at the edges (p = 0 or p = 1, where the outcome is certain — no question needed). Every entropy formula in this deck — for swell states, for English text, for image classes — is a generalization of this same bell-shaped surprise function.

🧮 Build It Yourself — The Info-Theory Calculator

You've seen each metric one at a time. This demo builds them all at once. Drag the sliders to define reality p(x) and your model q(x) over four outcomes. Every information-theoretic quantity recomputes on the fly — entropy, both cross-entropies, both KLs, JSD, and total variation distance.

REALITY p(x)

🔥 Firing: 15%

🤙 Fun: 35%

😐 Meh: 35%

💤 Skip: 15% (auto)

MODEL q(x)

🔥 Firing: 5%

🤙 Fun: 15%

😐 Meh: 45%

💤 Skip: 35% (auto)

Reality p(x)

Model q(x)

H(p) Entropy of reality

—

H(p, q) Cross-entropy

—

H(q, p) reverse CE

—

D_KL(p ‖ q)

—

D_KL(q ‖ p) (reverse)

—

JSD(p ‖ q) symmetric

—

🧩

Things to play with: drag the sliders so q matches p — every divergence collapses to 0 and H(p, q) lands on H(p). Then make q assign 1% to a class p says is 50% — KL shoots up and JSD stays bounded. Try the "Mirror image" preset to see why D_KL is asymmetric. This is the same playground production ML teams use to reason about loss curves, just with four classes instead of fifty thousand.

🦜 Why Should You Care?

Every AI model you've ever used — ChatGPT, image classifiers, spam filters — is trying to minimize cross-entropy . It's the standard score for "how wrong is my model?"

The model outputs predictions (a probability distribution over possible outcomes). Reality reveals what actually happened. Cross-entropy measures the gap. Training a neural network means adjusting parameters to shrink that gap, millions of times over.

🔑

Three things to remember:
1. Cross-entropy is always ≥ 0. Lower is better.
2. It hits its minimum when predictions perfectly match reality.
3. Being confidently wrong gets punished exponentially harder than being uncertain.

🎲 What "N i.i.d. Samples" Actually Means

You'll see this phrase everywhere in ML: "N i.i.d. samples from distribution p." Let's unpack it because it matters more than people realize.

📖

i.i.d. = Independent and Identically Distributed
Two separate ideas glued together:

Independent — each sample tells you nothing about the others.

Back at Guiones: was the surf firing yesterday ? Doesn't change the probability today. Each morning is a fresh draw from nature. In practice this is an approximation — ocean swells come in multi-day pulses from storm systems, so consecutive days are correlated — but the independence assumption is what makes the math tractable.

If samples weren't independent (say, good swells always come in 3-day runs), then seeing a firing session today would inform you about tomorrow, and you'd need a sequence model. The independence assumption lets us treat each observation as its own experiment.

Identically Distributed — every sample comes from the same underlying process.

The probability distribution of session quality isn't drifting between samples. But at Guiones, this is obviously season-dependent: dry season (Dec–Apr) has smaller, cleaner waves with offshore NE winds; green season (May–Nov) brings bigger S/SW groundswells but onshore afternoon winds. If you mash January data with July data, you're averaging over two different distributions, and your estimate of either one gets muddied. In practice you'd condition on season (or swell period, or wind direction) to get samples that are actually from the same distribution.

N — the number of samples you collected.

More samples = better estimate. With N=5 morning sessions you might get unlucky and see 100% mush. With N=1,000 you'll converge on the true distribution. This is the Law of Large Numbers at work.

⚠️

Why this matters for cross-entropy: The entire bridge from "theoretical cross-entropy" to "thing we can compute from data" relies on the i.i.d. assumption. If your samples aren't independent, your estimate is biased. If they're not identically distributed, you're estimating an average of different distributions — a different thing entirely.

🌧️ The Problem: We Don't Know p

The cross-entropy formula requires knowing p(x) — the true probability of each outcome:

H(p, q) = −Σₓ p(x) log q(x)

But in the real world, we almost never know p . We don't know the exact probability that a given morning at Guiones will be firing. We don't know the exact distribution of words in English. We don't know the exact probability that an image contains a toucan.

All we have is data — a log of past sessions, a corpus of text, a labeled dataset. So how do we compute cross-entropy without knowing p?

🎯

Answer: the Monte Carlo estimate. Instead of summing over the true distribution p(x), we average over our actual observed data. Each observation "votes" equally on how surprised our model should be.

🎰 Where "Monte Carlo" Came From

Before we use the Monte Carlo method, it's worth knowing its origin story — because it's one of the wildest in all of science.

🃏

Canfield solitaire — the game that started it all

                                1946 · Los Angeles
                            

January 1946. Stanisław Ulam — Polish-American mathematician, Manhattan Project veteran — wakes up one morning in Los Angeles unable to speak. The diagnosis: viral encephalitis. Emergency brain surgery follows: skull sawed open, brain tissue sprayed with antibiotics. He survives. During a slow, grueling convalescence, he passes the time playing Canfield solitaire.

Then the question hits him: what's the probability of a hand coming out successfully? He tries computing it analytically — the combinatorics are intractable. Then the insight: why not just play a hundred games and count? This is the core of the Monte Carlo method. Replace impossible math with repeated random experiments.

🖥️

ENIAC — the machine that made sampling practical

                                1947 · Los Alamos
                            

Ulam described the idea to John von Neumann , who immediately saw its power. The key was that the newly invented ENIAC computer could automate the sampling — turning a mathematical curiosity into a practical weapon. On March 11, 1947 , von Neumann sent a handwritten letter to Robert Richtmyer (head of the Los Alamos Theoretical Division) containing an 81-step pseudo code for simulating neutron diffusion in fissionable material — a problem the Manhattan Project physicists couldn't crack analytically despite having all the relevant physical constants.

That letter may be the first Monte Carlo algorithm ever written for a computer. The first actual calculations ran on ENIAC in April–May 1948 . Von Neumann's wife Klára contributed significantly to the coding effort, working alongside Metropolis on ENIAC for 32 straight days to implement the method.

What von Neumann's 81 steps simulated

Each neutron enters the material and randomly scatters off uranium nuclei. At each collision, dice are rolled: scatter (bounce off, lose energy), absorb (neutron disappears), or fission (nucleus splits, releasing 2–3 new neutrons). Repeat for millions of neutrons. The fraction that escape vs. cause new fissions determines whether the material goes critical.

Inside the 81 steps

Von Neumann's letter to Richtmyer didn't contain vague ideas — it was an explicit algorithm , written as 81 numbered instructions that ENIAC could execute. Each instruction was a primitive operation: add two numbers, compare a value, branch to a different step. Together they encoded the following loop for a single neutron:

1

Initialize the neutron

Set starting position (x,y,z), direction (θ,φ), and energy. The first neutrons enter from outside the material; later-generation neutrons are born at fission sites.

2

Roll dice for free-path length

Generate a pseudo-random number. Use it to sample from the exponential distribution governing how far a neutron travels before hitting a nucleus. This distance depends on the neutron's energy and the material's density — both tracked by the algorithm.

3

Move the neutron

Advance position along the current direction by the sampled distance. Check boundary conditions — did the neutron exit the material? If yes, tally it as escaped and terminate this neutron's history.

4

Roll dice for collision type

Generate another random number and compare it against the known cross-sections (physical probabilities) for U-235 at this neutron energy. Three outcomes:

SCATTER

Bounce off. Roll for new direction & energy loss. Go to step 2.

ABSORB

Neutron disappears into the nucleus. Terminate.

FISSION

Nucleus splits. Roll for how many new neutrons (2 or 3). Push them onto the queue. Terminate this neutron.

5

Loop until the neutron's history ends

If it scattered, repeat from step 2 with updated energy and direction. If absorbed or escaped, this neutron is done. If fission, the spawned neutrons are added to the queue and each one gets its own full run through the same loop. After all neutrons (original + offspring) are resolved, the simulation tallies: how many escaped? How many caused fission? How many were absorbed?

That's it — steps 1 through 81 encoded this loop plus all the bookkeeping: pseudo-random number generation (von Neumann's "middle-square" method), geometry checks for spherical/cylindrical material boundaries, energy-dependent cross-section lookups, and a queue for managing multiple generations of neutrons. The key insight was that none of these individual steps require solving the Boltzmann transport equation . Each step is trivial arithmetic or a random branch. The physics emerges from running millions of neutrons through this loop and counting the outcomes — which is exactly what Monte Carlo means.

🎲

The Monte Carlo Casino — namesake of the method

                                1949 · First paper
                            

The work was classified, so it needed a code name. Nicholas Metropolis suggested "Monte Carlo" — after the famous casino in Monaco — because Ulam's uncle Michał would borrow money from relatives because he "just had to go to Monte Carlo" to gamble. The first unclassified paper, Metropolis and Ulam's "The Monte Carlo Method," appeared in the Journal of the American Statistical Association in 1949.

Today, from training neural networks to pricing financial derivatives to simulating protein folding, every Monte Carlo estimate you compute is a direct descendant of a Polish mathematician's convalescent game of solitaire.

The idea

🃏→🔢

Can't compute it? Sample it.

The guarantee

N→∞

More samples → converges to truth

The trick

p(x)→∅

Don't need p — sampling handles the weighting

🎰 Monte Carlo Estimation: From Theory to Practice

The theoretical cross-entropy is an expectation under p:

H(p, q) = −Σₓ p(x) log q(x) = 𝔼ₚ[−log q(x)]

That 𝔼ₚ means "average this quantity over all possible events, weighted by how likely each truly is." We can't compute that without knowing p.

The Monte Carlo trick: if we have N i.i.d. samples x₁, x₂, ..., xₙ drawn from p, we approximate the expectation by averaging over our samples:

Ĥ(p, q) = −(1/N) Σᵢ log q(xᵢ)

Notice: p(x) disappeared entirely . We don't need to know p — we just need samples from p. Each xᵢ was already drawn according to p, so frequent outcomes naturally appear more often in our sum, and rare outcomes less. The sampling itself handles the weighting.

🌊 Simulate it: Guiones session sampling

True distribution: 🔥15% firing, 🤙35% fun, 😐35% meh, 💤15% skip. Your model predicts using the sliders. Hit "Sample" to draw random sessions and watch the Monte Carlo estimate converge.

Model: 🔥 Firing: 50%

Model: 🤙 Fun: 30%

Model: 😐 Meh: 15%

Model: 💤 Skip: 5% (auto)

← Each bar = running Ĥ as samples accumulate →

Monte Carlo Ĥ

—

−(1/N) Σ log q(xᵢ)

True H(p,q)

—

−Σ p(x) log q(x)

Draw some sessions to see convergence.

📊 Likelihood: How Probable Is the Data?

Before we get to Maximum Likelihood Estimation , we need the word it's built on: likelihood .

You have a model with adjustable parameters θ (knobs you can tune). For any setting of those knobs, the model assigns a probability to each possible outcome. The likelihood is simply: given a particular setting of θ, how probable is the data you actually observed?

Surf example: Suppose your model q_θ predicts Guiones session probabilities — but the predictions depend on θ (maybe θ controls how much weight you give to swell period vs wind speed). You observe 10 real sessions. For a given θ, the model says those 10 sessions had probabilities 0.4, 0.1, 0.3, ... The likelihood is the product of all those probabilities: how "likely" was this exact sequence of outcomes under this particular θ? A higher product means this θ explains the data better.

Maximum Likelihood is then the obvious next step: try every possible θ (in practice, use gradient descent) and pick the one that makes the observed data most probable. That's it — find the knob settings where the model says "yeah, this data was most likely to happen."

📈 From Cross-Entropy to MLE

Now we can connect the dots. You have a model q_θ and want to fit it to data using Maximum Likelihood Estimation (MLE) — find the θ that maximizes the likelihood of the observed data.

What is MLE? You have observed data (e.g. surf session outcomes, or a training corpus of text) and a model with adjustable parameters θ. The model assigns a probability q_θ(x) to each possible outcome x. Maximum Likelihood Estimation says: find the values of θ that make the data you actually observed as probable as possible under the model. In other words, pick the parameters that maximize the probability the model would have generated exactly the data you saw. That's it — the "best" model is the one that says "yeah, this data was most likely to happen."

Given N i.i.d. observations, the likelihood is the probability the model assigns to seeing all of them — a product of individual probabilities:

L(θ) = q_θ(x₁) · q_θ(x₂) · ... · q_θ(xₙ) = Πᵢ q_θ(xᵢ)

Products of tiny probabilities are numerically awful, so take the log (monotonic, so the maximizer is unchanged):

log L(θ) = Σᵢ log q_θ(xᵢ)

Divide by N to get an average:

(1/N) log L(θ) = (1/N) Σᵢ log q_θ(xᵢ)

🤯

That right-hand side is the negative of the Monte Carlo cross-entropy estimate.

Maximizing log-likelihood = Minimizing cross-entropy.

They're the same optimization with a sign flip. Every time someone says "we trained with MLE" and another says "we trained with cross-entropy loss," they're describing the exact same damn thing.

This isn't coincidence. The log in the likelihood is the same log in cross-entropy, which is the same log from optimal coding theory (Kraft-McMillan). Information theory, statistics, and ML all converge on the same objective.

🔬 Why the Empirical Distribution p̂ Shows Up

When values repeat in your N samples, count their frequency. If "fun session" appeared 350 times in 1000 samples, its empirical frequency is p̂(fun) = 0.35.

The Monte Carlo estimate rewritten by grouping identical outcomes:

−(1/N) Σᵢ log q(xᵢ) = −Σₓ p̂(x) log q(x) = H(p̂, q)

The Monte Carlo cross-entropy estimate is exactly the cross-entropy between the empirical distribution p̂ and your model q. As N → ∞, the Law of Large Numbers guarantees p̂(x) → p(x), so the estimate converges to the true cross-entropy.

🧩

The full picture:
True H(p, q) → estimated by → Monte Carlo Ĥ = H(p̂, q) → which equals → −(1/N) × log-likelihood → so minimizing it → is MLE.

Every piece connects. It's all the same thing viewed from different angles.