Confidence Intervals

Why we need more than a point estimate

Suppose you estimate a population parameter θ (a mean, a proportion, a regression coefficient) using a point estimator θ̂ computed from sample data. Even if θ̂ is “good” (unbiased or consistent), it will vary from sample to sample.

If we only report θ̂, we hide an essential fact: a different random sample would yield a different estimate. A confidence interval is an interval estimate: a data-derived range intended to contain θ.

The key idea: θ is fixed, the interval is random

A common beginner intuition is: “After I compute an interval, there’s a 95% probability that θ is inside.” That’s not the frequentist definition.

In frequentist statistics, θ is a fixed (unknown) constant. Randomness comes from the sampling process. So a confidence interval is a random interval [L(X), U(X)] computed from random data X.

A (1 − α) CI is a procedure that satisfies the coverage property:

P( L(X) ≤ θ ≤ U(X) ) = 1 − α

The probability is over repeated sampling of X under the assumed model.

What “95% confidence” really means

Imagine repeating the same experiment many times, each time computing a 95% CI using the same method. About 95% of those intervals will contain the true θ. About 5% will miss it.

Once you have computed one interval from your one dataset, the interval is no longer random; it’s a fixed numerical range. The method used to compute it is what carries the 95% guarantee.

A quick contrast: probability statements vs confidence statements

• •Probability statement (Bayesian-style): P(θ ∈ [a, b] | data) = 0.95
• •Confidence statement (frequentist): P([L(X), U(X)] contains θ) = 0.95

They sound similar, but condition on different things. Confidence intervals do not, by default, assign probabilities to θ.

Anatomy of a typical CI

Most classical CIs follow a common template:

θ̂ ± (critical value) · se(θ̂)

• •θ̂ is your point estimate
• •se(θ̂) is its standard error (typical sampling deviation)
• •the critical value sets the desired coverage (often from a Normal or t distribution)

This is the “margin of error” idea made precise.

Where the template comes from (preview)

The template is a consequence of: (1) a sampling distribution for θ̂, often approximately Normal via the Central Limit Theorem (CLT), and (2) a pivot or standardized statistic whose distribution we can control.

You already know the CLT: sums/averages of many random variables tend toward a Normal distribution. Confidence intervals are where that knowledge becomes actionable.

Why sampling distributions matter

A confidence interval is a statement about what happens across repeated samples. So we need the distribution of θ̂ across those repeated samples: the sampling distribution.

If θ̂ has a distribution centered at θ with some spread, we can pick an interval around θ̂ that captures θ most of the time.

Standard error: the scale of estimator noise

The standard error se(θ̂) is the standard deviation of θ̂’s sampling distribution:

se(θ̂) = √Var(θ̂)

You can read it as: “How much θ̂ typically wiggles due to sampling.”

Crucially:

• •Standard deviation (SD) describes variability in raw data.
• •Standard error (SE) describes variability in an estimator across samples.

Example: mean of i.i.d. data

Let X₁, …, Xₙ be i.i.d. with mean μ and variance σ². The sample mean is

μ̂ = X̄ = (1/n) ∑ᵢ Xᵢ

Then

E[X̄] = μ

Var(X̄) = σ² / n

se(X̄) = σ / √n

So uncertainty shrinks like 1/√n. This is why sample size helps, but with diminishing returns.

Using the CLT to get an approximate Normal sampling distribution

For large n, the CLT says

(X̄ − μ) / (σ/√n) ≈ Normal(0, 1)

Equivalently,

X̄ ≈ Normal( μ, σ²/n )

This approximation is the workhorse behind many “Wald-type” confidence intervals.

Unknown σ: replacing it creates extra uncertainty

In real problems, σ is usually unknown. We estimate it with the sample standard deviation s.

When data are Normal, we have an exact result:

T = (X̄ − μ) / (s/√n) ∼ t_{n−1}

That t distribution is wider than Normal(0,1), reflecting the extra uncertainty from estimating σ.

General pattern (beyond means)

Often, for an estimator θ̂ (like an MLE), we have an asymptotic Normal approximation:

θ̂ ≈ Normal( θ, Var(θ̂) )

Then a standardized version

Z = (θ̂ − θ) / se(θ̂)

is approximately Normal(0,1). This is the bridge from sampling distribution to an interval.

What you need to build a CI in practice

To construct a CI, you typically need:

1) A point estimator θ̂

2) An estimate of its standard error se(θ̂)

3) A critical value c such that

P(−c ≤ Z ≤ c) = 1 − α

Then solve the inequality for θ.

Breathing room: why “standard error” is the star

If you remember only one operational fact:

A CI width is controlled primarily by se(θ̂).

Everything else (Normal vs t, exact vs approximate, 90% vs 95%) is a multiplier on that base uncertainty. If your SE estimate is poor, your CI will be misleading.

Why pivots are powerful

To build an interval, we want a statistic with a known distribution that does not depend on the unknown θ (or depends on it in a controllable way). Such a statistic is called a pivot.

Once we have a pivot, we can take quantiles of its distribution and algebraically rearrange to isolate θ.

The classic Normal-based (Wald) derivation

Assume we have an estimator θ̂ such that

Z = (θ̂ − θ) / se(θ̂) ≈ Normal(0, 1)

Let z_{1−α/2} be the (1 − α/2) quantile of the standard Normal. Then

P( −z_{1−α/2} ≤ Z ≤ z_{1−α/2} ) ≈ 1 − α

Substitute Z:

P( −z_{1−α/2} ≤ (θ̂ − θ)/se(θ̂) ≤ z_{1−α/2} ) ≈ 1 − α

Now solve for θ. Multiply through by se(θ̂) (positive):

P( −z_{1−α/2}·se(θ̂) ≤ θ̂ − θ ≤ z_{1−α/2}·se(θ̂) ) ≈ 1 − α

Rearrange terms:

Left inequality:

θ̂ − θ ≥ −z·se

⇒ −θ ≥ −z·se − θ̂

⇒ θ ≤ θ̂ + z·se

Right inequality:

θ̂ − θ ≤ z·se

⇒ −θ ≤ z·se − θ̂

⇒ θ ≥ θ̂ − z·se

Combine:

P( θ̂ − z_{1−α/2}·se(θ̂) ≤ θ ≤ θ̂ + z_{1−α/2}·se(θ̂) ) ≈ 1 − α

So the (approximate) (1 − α) CI is

[ θ̂ − z_{1−α/2}·se(θ̂), θ̂ + z_{1−α/2}·se(θ̂) ]

This is the famous “estimate ± margin of error.”

When to use t instead of z

If your pivot is t-distributed (common for means with unknown σ under Normal data), you replace z_{1−α/2} with t_{n−1, 1−α/2}.

Interpreting the critical value

Common two-sided critical values:

Higher confidence means a larger multiplier, hence wider intervals.

One-sided confidence intervals

Sometimes you only care about an upper bound or lower bound.

For an upper (1 − α) bound, you want

P( Z ≤ z_{1−α} ) = 1 − α

leading to an upper confidence bound:

θ ≤ θ̂ + z_{1−α}·se(θ̂)

Similarly for a lower bound.

Beyond Wald: why alternative constructions exist

The Wald CI is simple, but it can behave poorly when:

• •sample size is small
• •parameter is near a boundary (e.g., probability p near 0 or 1)
• •estimator distribution is skewed

This is why you will later see intervals like Wilson score intervals (for proportions), likelihood-based intervals, or bootstrap intervals.

Still, the pivot logic remains the same: find (or approximate) a statistic with a known distribution, then invert it.

A useful comparison table

For this node, the main goal is to master what a CI means and how the standard “critical value × SE” construction arises.

What CIs let you do

Confidence intervals are a lingua franca for uncertainty.

They let you:

• •communicate estimation uncertainty (more informative than a point estimate)
• •compare groups (do intervals overlap? cautiously interpreted)
• •connect to hypothesis tests (CI inversion)
• •reason about practical significance (is the whole interval within a “good enough” range?)

CI ↔ hypothesis test connection (inversion idea)

A two-sided (1 − α) CI for θ corresponds to a two-sided hypothesis test at level α:

H₀: θ = θ₀

Rule:

• •Reject H₀ if θ₀ is outside the (1 − α) CI.
• •Fail to reject if θ₀ is inside.

Why this works: both procedures are built from the same pivot and critical region.

Planning and sample size (precision thinking)

Suppose your CI has half-width (margin of error) m:

m = z_{1−α/2} · se(θ̂)

For a mean with known σ:

m = z · (σ/√n)

Solve for n to achieve desired precision m:

m = zσ/√n

⇒ √n = zσ/m

⇒ n = (zσ/m)²

This turns “How many samples do we need?” into a quantitative design question.

Practical interpretation: what you can safely say

Given a 95% CI [L, U], it’s correct to say:

• •“Using this method, we constructed an interval that would cover θ in 95% of repeated samples.”
• •“Values of θ far outside [L, U] are not very compatible with the data under the model.”

Be cautious with:

• •“There is a 95% probability θ is in [L, U].” (frequentist CIs don’t mean that)

Width drivers and trade-offs

CI width increases when:

• •confidence level increases (95% → 99%)
• •data are noisier (larger σ, larger Bernoulli variance p(1−p))
• •sample size decreases
• •the estimator is inefficient (larger Var(θ̂))

CI width decreases when:

• •n increases
• •you use a more efficient estimator (e.g., MLE under correct model)

Connection to MLE (a prerequisite you already know)

For many MLEs θ̂, under regularity conditions:

θ̂ ≈ Normal( θ, I(θ)^{-1} )

where I(θ) is Fisher information (or n times per-sample information). In practice we plug in θ̂:

se(θ̂) ≈ √( I(θ̂)^{-1} )

Then a Wald CI becomes

θ̂ ± z_{1−α/2} · √( I(θ̂)^{-1} )

This is one of the most common “automatic CI” outputs in statistical software.

“Compatible values” mindset

A CI is often best treated as the set of parameter values that are reasonably compatible with your observed data under the assumed model.

This mindset helps you avoid the trap of treating 0.049 vs 0.051 p-values as fundamentally different: an interval shows a continuum of plausible effects.

A final caution: the model assumptions matter

Coverage (the 95% guarantee) depends on the assumptions used to derive se(θ̂) and the pivot distribution:

• •independence assumptions
• •identical distribution / stationarity
• •correct likelihood model for MLE-based SEs
• •large-sample approximations actually being “large enough”

If those fail, your interval can under-cover (too narrow) or over-cover (too wide).