Why the best prediction is the conditional mean

1Random pairs and a joint table

ESL sets up a world where each case, say a customer, comes with an input X (the features) and an output Y (the thing to predict). Both are random variables: quantities whose values aren't known until a case is drawn. The book writes the random variable in capitals and one particular value it takes in lowercase. So X means "the input of a case we haven't seen yet", and x means one concrete value, like x = 1. In the book, X ∈ ℝ^p is a list of p numbers and Y ∈ ℝ is one number. Here we shrink both down so everything fits on one table.

In our running example, X can only be 0 or 1, and Y can only be 1 or 3. The joint distribution Pr(X, Y) gives the probability of every combination, and that's all there is to know about the population:

Read the bottom-right cell as: half of all cases have x = 1 and y = 3. The four cells add to 1, because every case lands in exactly one of them. The table has to be joint, not two separate lists, because the relationship lives in the pairing: when x = 0, Y is usually 1, and when x = 1, Y is usually 3. It also contains noise, since the same x can come with either y. Even a perfect rule will make some mistakes.

Adding along a row gives the probability of that x, whatever y turns out to be: Pr(X = 0) = 0.3 + 0.1 = 0.4 and Pr(X = 1) = 0.1 + 0.5 = 0.6. These row totals are called the marginal distribution of X. They'll come back in section 6 as weights.

One boundary to keep in mind for the whole page: this table is the population, every case that could ever occur. In real work you never see it. You see a finite training sample drawn from it. Equations 2.9 to 2.13 describe the ideal you'd reach if you knew the table, and later sections of ESL are about approximating it from data.

2One draw, one square

A prediction rule f sees only x and has to return one number. In our table x is 0 or 1, so a rule is just two numbers, and we'll call them dials: a = f(0), the prediction used whenever x = 0, and b = f(1), used whenever x = 1. Choosing f means setting those two dials. The rule doesn't get to see y, so both cells of a row get the same prediction.

When a case is drawn, the miss is y − f(x), the actual value minus the prediction. A loss function L(Y, f(X)) turns the miss into a penalty, and ESL uses squared error: L = (Y − f(X))². Squaring does two jobs. It removes the sign, so a miss of −2 costs the same as +2. It also grows faster than the miss: twice the miss costs four times as much. You can draw the penalty as a square whose side is the miss, so its area is the squared error.

3EPE is the average square (2.9)

One draw gives one square, but the drawn cell is random. ESL scores a rule by the average square over all draws, the expected prediction error:

EPE(f) = E(Y − f(X))² (2.9)

The E is an expectation, a probability-weighted average: evaluate the quantity in every possible situation, weight each result by how likely that situation is, and add. For our table the situations are the four cells, so E(g(X, Y)) = Σ g(x, y) · Pr(x, y), summed over the cells. Here g is the squared miss. Each cell's contribution is its probability times its squared miss:

Adding the last column gives EPE as a formula in the two dials. It covers every rule you could pick:

EPE(a, b) = 0.3(1 − a)² + 0.1(3 − a)² + 0.1(1 − b)² + 0.5(3 − b)²

Try the rule "predict the most likely y in each row", a = 1 and b = 3. The misses are 0, 2, −2 and 0, and the squares are 0, 4, 4 and 0, so EPE = 0.3·0 + 0.1·4 + 0.1·4 + 0.5·0 = 0.8.

Keep two things apart. EPE takes a whole rule and returns one number, so it compares rules to each other. And it's computed over the population, not over a training set. Training error averages over the cases you happened to observe, while EPE averages over every case weighted by how often it occurs.

4From a sum to an integral (2.10)

Real inputs and outputs are continuous, so there's no finite list of cells to add up. ESL writes the same average as an integral:

EPE(f) = ∫ [y − f(x)]² Pr(dx, dy) (2.10)

To read it, cut the (x, y) plane into tiny rectangles. Pr(dx, dy) is the probability that a case lands in the tiny rectangle at (x, y). It plays the role of a cell probability. [y − f(x)]² is that rectangle's squared miss, and ∫ adds the products over the whole plane, just like the sum in section 3. When the distribution has a density p(x, y), Pr(dx, dy) = p(x, y) dx dy. ESL uses the Pr(dx, dy) notation because it also covers discrete tables like ours, where there is no density and the integral becomes the four-cell sum.

So (2.9) and (2.10) are one quantity. (2.9) is the shorthand, and (2.10) unpacks what the E means.

5Splitting a cell's probability (the footnote)

Suppose you're told the case has x = 1. The top row can't happen any more, and the probabilities left in the bottom row, 0.1 and 0.5, add to only 0.6. Dividing them by 0.6 rescales the row so it adds to 1 again, and the result is the distribution of Y once you know x. That's conditional probability:

Pr(y | x) = Pr(x, y) / Pr(x)

The bar "|" reads "given". Doing the division for every cell:

Multiply both sides of the definition by Pr(x) and you get the factorization in ESL's footnote:

Pr(x, y) = Pr(y | x) · Pr(x)

Check it on the bottom-right cell: 0.833 × 0.6 = 0.5. In words, the chance of landing in a cell is the chance of landing in its row times the chance of that y within the row. This is just a different way of writing each cell's probability. Nothing has been computed yet, but section 6 uses exactly this split.

6Regrouping the sum by x (2.11)

ESL now rewrites EPE as

EPE(f) = E_X E_Y|X([Y − f(X)]² | X) (2.11)

Read it from the inside out. The inner E_Y|X(… | X) holds X fixed and averages the squared miss over the Y values that can occur with it, using Pr(y | x) as weights. That gives one number per x. The outer E_X averages those numbers over x, using Pr(x) as weights. The value is the same as (2.9). Only the order of adding has changed.

Here's the regrouping done step by step for the discrete table. The right-hand column of each step carries the "most likely y" rule, a = 1 and b = 3, along to show that the number doesn't change.

1. Start from the direct sum (2.10). One term per cell, squared miss times cell probability.

   EPE = Σ_x Σ_y [y − f(x)]² · Pr(x, y)

   For our rule, that's the 0.8 from section 3.
2. Split each cell probability with the footnote. Section 5 showed Pr(x, y) = Pr(y | x) · Pr(x). Substituting changes how each term is written, not its value.

   EPE = Σ_x Σ_y [y − f(x)]² · Pr(y | x) · Pr(x)

3. Pull Pr(x) out of the inner sum. The inner sum runs over y with x held fixed, so Pr(x) is the same number in every one of its terms, and a common factor can move outside a sum (0.4·1.5 + 0.4·2 = 0.4·(1.5 + 2)).

   EPE = Σ_x Pr(x) · [ Σ_y [y − f(x)]² · Pr(y | x) ]

4. Recognize the bracket as an expectation over Y given x. It's a weighted average of the squared miss with weights Pr(y | x), which add to 1 within the row, so it's E_Y|X([Y − f(X)]² | X = x). For our rule: row 0 gives 0.75·0 + 0.25·4 = 1, and row 1 gives 0.167·4 + 0.833·0 ≈ 0.667.
5. Recognize the outer sum as an expectation over X. What's left is a weighted average of the row results with weights Pr(x), which is E_X of the bracket. That's (2.11). For our rule: 0.4·1 + 0.6·0.667 = 0.8, the same value as step 1.

The general fact behind these steps is the law of total expectation, also called the tower rule or Adam's law: E(Z) = E_X(E(Z | X)). The overall average equals the average of the per-group averages, each weighted by how common its group is. For continuous variables the same steps use the density: p(x, y) = p(y | x) p(x), and the integral is evaluated over y first, inside an integral over x. Changing the order of integration like that is allowed here because the squared miss is never negative (Tonelli's theorem).

7Minimizing one row at a time (2.12)

Group the EPE formula from section 3 by row, as section 6 suggested:

EPE(a, b) = [0.3(1 − a)² + 0.1(3 − a)²] + [0.1(1 − b)² + 0.5(3 − b)²] = A(a) + B(b)

A contains only dial a and B contains only dial b. Think of them as two bills paid separately: turning a changes A and leaves B exactly where it was. Here's A at a few values:

The middle row, 0.3·0.25 + 0.1·2.25 = 0.3, is the smallest of the three. Since no term contains both dials, there's no trade-off to manage: the smallest possible total is the smallest A plus the smallest B. You can find the best a using only row 0 and the best b using only row 1.

Now connect this to (2.11). Each bill is that row's weight times its inner average: A(a) = 0.4 × [0.75(1 − a)² + 0.25(3 − a)²], since 0.4 × 0.75 = 0.3 and 0.4 × 0.25 = 0.1. The weight 0.4 is positive and doesn't contain a, so it scales the bill without moving its lowest point. So minimizing the bill means minimizing the inner average alone. ESL writes that for every x at once:

f(x) = argmin_c E_Y|X([Y − c]² | X = x) (2.12)

argmin_c means "the value of c that makes the expression smallest", so the answer is the location of the minimum, not the minimum value itself. The c stands for the one number you'll predict at this x. It's a plain constant here because x is fixed. "Minimize pointwise" means solving this small problem separately at each x.

That last observation is the boundary of the whole argument. Pointwise minimization works only because f may take any value at each x. Tie the dials, so the rule must predict one constant c everywhere, and every term contains c. Moving c toward 1 helps the y = 1 cells and hurts the y = 3 cells, so you're forced into a compromise. Setting the slope of the total to zero (section 8 explains this move) gives 0.3(1 − c) + 0.1(3 − c) + 0.1(1 − c) + 0.5(3 − c) = 0, that is, 2.2 − c = 0, so c = 2.2, and EPE(2.2) = 0.96. With free dials, best EPE = 0.633. A straight line f(x) = β₀ + β₁x also ties the predictions at different x together through two shared numbers. That's why (2.13) is called the ideal: it's the answer when nothing ties the predictions together, and linear regression and nearest neighbours are two ways of approximating it from data.

8The best prediction is the row mean (2.13)

We still need the actual minimizer of each row's U-shaped bill. At the bottom of a U the curve is flat, so its slope (derivative) is zero. On the left the slope is negative, and on the right it's positive.

The derivative of k(y − c)² with respect to c is −2k(y − c). For row 0, setting the slope of A to zero and dividing by −2 gives

0.3(1 − a) + 0.1(3 − a) = 0 → 0.3 − 0.3a + 0.3 − 0.1a = 0 → 0.6 = 0.4a, so a = 1.5

and for row 1, 0.1(1 − b) + 0.5(3 − b) = 0 gives 1.6 = 0.6b, so b ≈ 2.667. Look at what the answers are: 1.5 = (0.3·1 + 0.1·3) / 0.4 is the average y in row 0, weighted by Pr(y | x = 0), and 2.667 is the same thing for row 1. The best prediction in each row is that row's mean. In general, with weights Pr(y | x):

d/dc E([Y − c]² | X = x) = −2 (E(Y | X = x) − c) = 0  ⇒  f(x) = E(Y | X = x) (2.13)

ESL calls E(Y | X = x) the regression function: at every x, the best guess under squared error is the conditional mean.

A second route needs no calculus and shows what's left over. Write μ for the row mean E(Y | x) and split each miss into two pieces, Y − c = (Y − μ) + (μ − c). Squaring gives (Y − μ)² + 2(Y − μ)(μ − c) + (μ − c)². Now average over the row. The middle term averages to zero, because the deviations Y − μ from a mean average to zero and (μ − c) is a constant. What's left is

E[(Y − c)² | x] = Var(Y | x) + (μ − c)²

That's the row's spread plus a penalty for missing the mean. For row 0 the spread is 0.75·(1 − 1.5)² + 0.25·(3 − 1.5)² = 0.75, so row 0's inner average is 0.75 + (1.5 − c)². The second part is a square, so it's never negative, and it's zero only at c = 1.5. The first part doesn't depend on c: it's noise that no prediction can remove. Weighting both rows' leftover spreads by Pr(x) gives the 0.633 floor from section 7.

The answer depends on the loss. With absolute error |Y − c|, the same pointwise argument gives the conditional median instead of the mean (ESL eq. 2.18, a page later). In row 0, the median is 1, because 75% of the row's probability sits at y = 1. Squared error pulls the prediction toward the rarer 3, because a miss of 2 costs 4 under squaring.

9Your turn: the whole procedure on a new table

Here's a new population. X is still 0 or 1, but Y is now 0 or 4:

The first step is done for you: add the top row, 0.2 + 0.2, so Pr(X = 0) = 0.4. Each step after that leaves more for you to do.

10Check yourself without looking back

Try these from memory before opening the answers. Each answer names the section that teaches it.

11Review cards

Revisit these after a day, a week, and a month. Each one makes sense without this page open.

What does EPE(f) = E(Y − f(X))² measure?::The average squared miss of rule f over the whole population of (X, Y) pairs, each weighted by its probability.
How does ESL eq. 2.10 relate to 2.9?::Same quantity; the expectation written as an integral of squared miss times Pr(dx, dy) over the (x, y) plane.
What is Pr(y | x) for a discrete joint table?::The cell probability divided by its row total, Pr(x, y) / Pr(x), so each row adds to 1.
What does the factorization Pr(X, Y) = Pr(Y | X) Pr(X) let you do with the EPE integral?::Split it into an inner average over Y at fixed x inside an outer average over x (ESL eq. 2.11).
What is the law of total expectation?::E(Z) = E_X(E(Z | X)): the overall average equals the weighted average of per-group averages.
Why can EPE be minimized pointwise?::After grouping by x, each prediction f(x) appears only in its own term, and the weight Pr(x) cannot move that term's minimizer.
What does argmin_c mean in ESL eq. 2.12?::The value of c that makes the expression smallest (the location of the minimum, not the minimum value).
Under squared error, what is the best prediction at X = x?::The conditional mean E(Y | X = x), the regression function.
Under absolute error, what is the best prediction at X = x?::The conditional median of Y given X = x.
When does pointwise minimization break?::When predictions at different x share parameters (a constant or a straight line), so they cannot be set independently.
What part of expected squared error can no rule remove?::The spread Var(Y | X = x) at each x, weighted by Pr(x).

Checked on 2026-10-21 with Python 3.13 (standard library only) by verify_lesson.py: 34 checks passed.

Sources

• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd edition (2009; corrected 12th printing, 2017), section 2.4, equations 2.9 to 2.13, the footnote on conditioning, and eq. 2.18 for the median under absolute error.
• Law of total expectation (Wikipedia): the statement E(X) = E(E(X | Y)) and its other names (tower rule, Adam's law, law of iterated expectations).
• Fubini's theorem (Wikipedia), section on Tonelli's theorem: for a non-negative integrand, the order of integration can be swapped.
• Every worked number was recomputed by verify_lesson.py next to this page, which also checks that the row means minimize EPE over a grid of dial values and that the two-stage average equals the direct sum.