Expected Value

Why we need it (motivation)

A random variable X can take many values depending on chance. If you want to:

• •compare two gambling games,
• •decide whether an insurance policy is “fair,”
• •estimate average runtime of a randomized algorithm,
• •or reason about average loss in machine learning,

you need a single summary number.

Expected value is the theoretical long-run average: if you repeatedly draw independent samples X₁, X₂, … from the same distribution, the sample mean

tends to get close to a fixed value. That fixed value (when it exists) is E[X]. (A later node formalizes this as the Law of Large Numbers.)

Intuition first: “average with weights”

Suppose outcomes x happen with probabilities p(x). An ordinary average gives each outcome equal weight. Expected value gives outcome x the weight p(x).

So if big outcomes are rare, they still matter—but only proportionally to how often they occur.

Definition (discrete)

If X is discrete with possible values x and probability mass function P(X = x), the expectation is

You can read this as “sum over all outcomes: (value) × (chance of that value).”

Definition (continuous)

If X is continuous with probability density function f_X(x), then

This is the same idea: infinitely many possible values, so the weighted average becomes an integral.

Units and interpretation

• •E[X] has the same units as X. If X is dollars, E[X] is dollars.
• •E[X] is not necessarily a value X can actually take (e.g., the average of 0 and 1 is 0.5).

Difficulty calibration note

We’ll treat the following as core vs optional/advanced:

• •Core: compute E[X] for discrete/continuous distributions; linearity; interpretation.
• •Optional/Advanced: existence/undefined expectations; heavy tails; expectation of functions E[g(X)].

You can learn and apply expected value well with the core, then come back for the optional parts when you need them.

Discrete examples: build the pattern

For a discrete X, you need two things:

1) the set of possible values {x}, and

2) the probability of each value p(x).

Then compute the weighted sum ∑ x p(x).

Example pattern: dice

Let X be the value of a fair six-sided die. Then P(X=k)=1/6 for k∈{1,2,3,4,5,6}.

Notice 3.5 is not an outcome on the die—expected value is a planning number, not a predicted single roll.

Continuous examples: computing E[X] with integrals

For continuous X, the density f_X(x) plays the role of “probability per unit x.” The integral

is the continuous weighted average.

Example pattern: Uniform(0,1)

If X ∼ Uniform(0,1), then f_X(x)=1 for x∈[0,1] and 0 otherwise.

Expectation as “center of mass” (intuition)

A useful physical analogy: imagine each value x has “mass” p(x) (discrete) or density f(x)dx (continuous). The expected value is the balance point.

• •If probability mass shifts right, E[X] increases.
• •If you add a rare but huge outcome, E[X] can jump noticeably.

Sanity checks when computing

1) Range check: If X is always between a and b, then E[X] must lie between a and b.

2) Symmetry: If a distribution is symmetric around 0, often E[X]=0 (when it exists).

3) Weights sum to 1: For discrete, verify ∑ p(x)=1; for continuous, ∫ f(x)dx=1.

Core takeaway

Computing expectation is usually straightforward bookkeeping—until you meet distributions with extremely large values or tails. That’s where the next sections add nuance.

Why linearity matters

Many random variables are built from simpler pieces:

• •total reward = sum of per-step rewards,
• •total cost = sum of random costs,
• •total heads = sum of indicator variables,
• •ML loss over a dataset = average of per-example losses.

Computing the full distribution of a sum can be hard. Expected value often stays easy because expectation is linear.

Linearity rules (core)

For random variables X and Y (no independence required):

For constants a, b:

More generally, for any finite sum:

Mini-derivation (discrete)

Assume (X,Y) are discrete.

Start with the definition:

Split the sum:

Now notice:

so

Similarly the second term becomes E[Y]. Therefore E[X+Y]=E[X]+E[Y].

Indicators: a common trick

Define an indicator random variable I for an event A:

• •I = 1 if A happens
• •I = 0 otherwise

Then

This turns probability questions into expectation questions.

Example: Let X be the number of heads in n coin flips (not necessarily fair). Let Iᵢ indicate “flip i is heads.” Then X = ∑ᵢ Iᵢ, so

If the coin has P(heads)=p each time, then E[X]=np.

What linearity does not say

A classic confusion is to assume expectation distributes over products:

• •Generally, E[XY] ≠ E[X]E[Y].

That equality holds under independence (and some integrability conditions), but linearity alone doesn’t give it.

E[g(X)] (operator viewpoint)

Expected value is not just a number attached to X—it’s an operator that maps a random variable to a number.

Often we care about a transformed quantity g(X):

• •squared error: g(X) = (X−c)²
• •absolute deviation: g(X)=|X|
• •utility in economics: g(X)=u(X)
• •loss in ML: g(X)=ℓ(X, y)

Definition (discrete)

Definition (continuous)

This is the same weighted-average idea, just applied after transforming the outcomes.

Law of the unconscious statistician (LOUTS)

A subtle but powerful point: to compute E[g(X)], you usually do not need the distribution of Y=g(X). You can compute directly from X’s distribution using the formulas above.

When expectation does not exist (or is infinite)

So far, we’ve treated expectation as always producing a finite number. But expectation is only well-defined when the weighted sum/integral converges.

A practical sufficient condition is that the absolute expectation is finite:

• •Discrete: $$\sum_x |x|\,\mathbb{P}(X=x) < \infty$$
• •Continuous: $$\int_{-\infty}^{\infty} |x|\, f_X(x)\, dx < \infty$$

If these diverge, several things can happen:

Heavy tails (intuition)

A heavy-tailed distribution puts enough probability on huge values that the “average” never settles.

A famous example is the Cauchy distribution: it’s symmetric, but its tails are so heavy that E[X] is undefined (the integral does not converge in the required sense). That’s why sample means of Cauchy draws behave wildly even for large n.

Why this matters in practice

• •In modeling, assuming E[X] exists lets you use many theorems (LLN, variance formulas, etc.).
• •In risk/finance, rare catastrophic outcomes can dominate expectation.
• •In ML, expectation of loss is usually well-defined, but certain data distributions (extreme outliers) can make empirical averages unstable.

If you’re learning expectation for the first time, don’t let this scare you: most standard distributions used early (Bernoulli, Binomial, Uniform, Normal, Exponential) have finite expectations. But it’s important to know that “average” is not guaranteed by definition—it’s a property that may or may not hold.

Fairness and pricing (games, insurance)

A gamble is often called “fair” if its expected payoff is 0 (or if the price equals expected payout).

If you pay cost c to play and receive random payout X, then net payoff is X−c. By linearity:

A fair price is c = E[X] (ignoring risk preferences).

Expected loss (machine learning viewpoint)

In supervised learning, we often minimize expected risk:

We don’t know the true distribution 𝒟, so we approximate R(θ) with the empirical average over data:

The idea “sample average ≈ expected value” is exactly the intuition behind expected value and what the Law of Large Numbers formalizes.

Why SGD works with expectation

Stochastic Gradient Descent uses a random mini-batch to estimate the gradient of expected loss.

If g(θ) is the gradient computed from a random sample, SGD relies on it being (approximately) unbiased:

This is an expectation statement: on average, the noisy gradient points in the true direction.

Connecting expectation to what comes next

• •Variance measures spread around the mean μ = E[X] using E[(X−μ)²].
• •Entropy uses an expectation too: H(X)=E[−log p(X)] (discrete).
• •Law of Large Numbers explains the long-run convergence of sample means to E[X].
• •Game theory uses expected payoff when players randomize strategies.

Expected value is the first “global” summary of a distribution you should reach for: it’s simple, composable via linearity, and it’s the backbone of many later definitions.