Hypothesis Testing

Hypothesis testing is a structured way to use sample data to evaluate a claim about a population.

The core difficulty is that samples vary. Even if nothing is changing in the population, random sampling will produce different means, proportions, and counts. Hypothesis testing acknowledges this uncertainty and asks:

• •If the null claim were true, how surprising would our sample evidence be?
• •Is it surprising enough that we’re willing to reject the null?

The competing statements: H₀ and Hₐ

A hypothesis test always starts with two competing hypotheses about a population parameter (a fixed but unknown number like μ, p, or λ).

• •Null hypothesis (H₀): the “status quo” or “no effect / no difference” claim. It is the claim we test as if it were true.
• •Alternative hypothesis (Hₐ): what we consider if the data provide strong evidence against H₀.

Example (mean):

• •H₀: μ = 100
• •Hₐ: μ ≠ 100

Example (proportion):

• •H₀: p = 0.10
• •Hₐ: p > 0.10

Notice that both hypotheses are statements about the population, not the sample. The sample is our window.

One-sided vs two-sided: the direction matters

The alternative hypothesis encodes what “extreme evidence” means.

• •Two-sided test: Hₐ: parameter ≠ value. Extremes are large deviations in either direction.
• •Right-tailed test: Hₐ: parameter > value. Extremes are unusually large values of the test statistic.
• •Left-tailed test: Hₐ: parameter < value. Extremes are unusually small values.

This choice is not cosmetic. It determines which tail(s) of the null distribution count as “as extreme or more extreme.”

The workflow in one sentence

1) Choose H₀/Hₐ → 2) compute a test statistic from the sample → 3) compare it to its null distribution → 4) compute a p-value → 5) reject or fail to reject using α.

A visual mental model (keep this in your head)

Under H₀, your test statistic has a distribution. You mark either:

• •a rejection region whose total area is α (fixed in advance), or
• •the p-value region: area in the tail(s) at least as extreme as the observed statistic.

Both are just shaded areas under the same “null curve.”

Static diagram: rejection region (two-sided)

Below is a generic null distribution (often approximately Normal). The two critical values cut off α/2 in each tail.

              Null distribution of test statistic (under H₀)

                       /
                      /  \
                     /    \
          __________/      \__________
         /                                \
--------|----|------------------|----|---------> t
      -c    0                  +c
     α/2                        α/2
   (reject)                   (reject)

Decision rule (two-sided): reject H₀ if t ≤ -c or t ≥ +c

Static diagram: p-value shading (right-tailed)

Here the alternative is “greater than,” so only the right tail counts.

              Null distribution of test statistic (under H₀)

                       /
                      /  \
                     /    \
          __________/      \__________
         /                                \
--------|-------------------|\\\\\\\\\\\-> t
       0                 t_obs   p-value area

p-value = P(T ≥ t_obs | H₀)

These pictures are the backbone of hypothesis testing. Everything else is computation.

Why we need a test statistic

A dataset is many numbers. A hypothesis test needs a single number that summarizes the evidence against H₀.

That number is the test statistic: a function of the sample.

Common patterns:

• •Sample mean x̄ (or a standardized version)
• •Sample proportion p̂ (or standardized)
• •Count in a Poisson process

But the raw statistic (like x̄) is hard to interpret without scale. We typically convert it into a standardized form that answers:

“How many standard errors away from the null value is the observed estimate?”

That’s what z-scores and t-scores do.

Standard error: the scale of sampling noise

If you repeatedly sample n observations from a fixed population, the statistic varies. Its standard deviation is the standard error (SE).

A key CLT-driven idea (you already know CLT) is:

• •For a mean, if σ is the population standard deviation,

so $\text{SE}(\bar X) = \sigma/\sqrt{n}$.

• •For a proportion, with true proportion p,

so $\text{SE}(\hat p) = \sqrt{p(1-p)/n}$.

Null distribution: “what we’d see if H₀ were true”

The null distribution is the sampling distribution of the test statistic assuming H₀ is true.

If H₀ specifies μ = μ₀, then under H₀:

• •$\bar X$ is centered at μ₀ (approximately)
• •the standardized statistic is centered at 0

A canonical z-test statistic for a mean with known σ is:

Under H₀ (and with CLT / Normal assumptions),

That last line is crucial: it tells you how to turn an observed Z into a tail probability.

“Extreme” depends on Hₐ

Once you have the null distribution, “extreme” means “in the tail(s) consistent with Hₐ.”

Let T be your test statistic with observed value t_obs.

• •Right-tailed (Hₐ: parameter > …):

• •Left-tailed (Hₐ: parameter < …):

• •Two-sided (Hₐ: parameter ≠ …):

For symmetric null distributions (like Normal), the two-sided p-value is often:

Rejection regions and α: deciding before seeing data

The p-value measures evidence; α is a decision threshold.

• •Significance level (α): a pre-set probability of rejecting H₀ when H₀ is true (long-run).

Decision rule:

• •Reject H₀ if p ≤ α
• •Otherwise, fail to reject H₀

This is equivalent to using critical values.

For example, in a two-sided z-test with α = 0.05:

• •critical values are ±1.96
• •reject H₀ if |Z| ≥ 1.96

This equivalence is worth seeing explicitly.

If $Z \sim \mathcal N(0,1)$ under H₀, and α = 0.05 two-sided, we choose c so that:

By symmetry:

So $c \approx 1.96$.

Then:

• •If z_obs = 2.3, it lies in the rejection region → p < 0.05 → reject.
• •If z_obs = 1.2, it lies in the non-rejection region → p > 0.05 → fail to reject.

What a p-value is (and what it is not)

A p-value is:

• •A probability computed under H₀
• •Of seeing a test statistic at least as extreme as observed

It is not:

• •The probability that H₀ is true
• •The probability that the result is “due to chance”
• •A measure of effect size

A small p-value means: “If H₀ were true, this would be rare.” It does not, by itself, tell you whether the effect is practically important.

The biggest source of confusion in hypothesis testing is mixing up:

• •the direction of Hₐ
• •the location of the rejection region
• •how the p-value is shaded
• •whether α is split

This section focuses on making the tail logic visual and automatic.

Step 1: Choose Hₐ first; it defines “extreme”

You should be able to answer: “Which sample outcomes would convince me H₀ is wrong?”

• •If only unusually large values would convince you, use a right-tailed test.
• •If only unusually small values would convince you, use a left-tailed test.
• •If either unusually large or unusually small would convince you, use a two-sided test.

This is not something you should decide after seeing data.

Step 2: α is area in the rejection region

Think of α as “how much tail area we’re willing to label as ‘reject’ when H₀ is true.”

Here are the three standard rejection-region pictures.

Right-tailed test (α all on the right)

              Null distribution under H₀

                       /
                      /  \
                     /    \
          __________/      \__________
         /                                \
--------|--------------------|\\\\\\\\\\\-> t
       0                    c
                        α (reject)

Reject if t ≥ c

Left-tailed test (α all on the left)

              Null distribution under H₀

                       /
                      /  \
                     /    \
          __________/      \__________
         /                                \
\\\\\\\\\\\|--------------------|---------> t
       c                    0
  α (reject)

Reject if t ≤ c

Two-sided test (α split into α/2 + α/2)

              Null distribution under H₀

                       /
                      /  \
                     /    \
          __________/      \__________
         /                                \
\\\\\\\|----|------------------|----|\\\\\\\-> t
      -c    0                  +c
     α/2                        α/2

Reject if t ≤ -c or t ≥ +c

Step 3: p-value shading matches the same tail(s)

The p-value is not “α in the tail.” It is the observed tail area beyond your observed statistic, using the appropriate tail rule.

If you remember one sentence, use this:

p-value = shaded area in the tail(s) of the null distribution beyond the observed statistic, in the direction(s) specified by Hₐ.

Examples of shading:

• •Right-tailed: shade to the right of t_obs
• •Left-tailed: shade to the left of t_obs
• •Two-sided: shade both tails beyond ±|t_obs|

A quick table to prevent tail mistakes

Connecting p-values to critical values (explicitly)

Suppose we do a right-tailed z-test at α = 0.05.

• •Critical value c is defined by:

So $c ≈ 1.645$.

Now compare two observed z-values:

1) z_obs = 1.2

• •p-value = P(Z ≥ 1.2) ≈ 0.115
• •Since 0.115 > 0.05, fail to reject.

2) z_obs = 2.0

• •p-value = P(Z ≥ 2.0) ≈ 0.0228
• •Since 0.0228 < 0.05, reject.

The rule “reject if p ≤ α” and the rule “reject if z_obs ≥ 1.645” always agree because they are the same geometric comparison under the null curve.

A note on “fail to reject” language

When p > α, we say fail to reject H₀, not “accept H₀.”

Why? Because the test is asymmetric:

• •It is designed to control false rejections (Type I error) at rate α.
• •It is not designed to certify H₀ is true.

Large p-values can happen because:

• •H₀ is true, or
• •H₀ is false but the effect is small, or
• •the sample size is too small / noise is high.

Hypothesis testing is a reusable template. Once you internalize the tail logic, you can apply it across many settings.

Common real-world uses

1) A/B testing (proportions)

• •H₀: conversion rate p_B − p_A = 0
• •Hₐ: p_B − p_A > 0 (or ≠ 0)
• •Test statistic: standardized difference in sample proportions

2) Quality control (means)

• •H₀: μ = target weight
• •Hₐ: μ < target weight (underfilling is the concern)

3) Healthcare / experiments

• •H₀: mean outcome difference between treatment and control is 0
• •Hₐ: difference ≠ 0

In each case:

• •Choose a parameter
• •Encode claims as H₀/Hₐ
• •Pick a statistic whose null distribution is known or approximable
• •Compute p-value, compare to α

What α controls (long-run behavior)

α is the probability of rejecting H₀ when H₀ is true:

That is a guarantee about a procedure, not about a single dataset. If you repeatedly run the same testing procedure in a world where H₀ is actually true, about α fraction of runs will (incorrectly) reject.

This motivates why α should be chosen before looking at the data: it’s part of the design of the decision rule.

Practical significance vs statistical significance

A tiny effect can be statistically significant if n is huge (SE becomes small). Conversely, a meaningful effect can fail to be significant if n is small.

Because the standardized statistic is often of the form:

Increasing n shrinks SE like $1/\sqrt{n}$, making it easier for a fixed effect to appear “many SEs away.”

In practice, you should pair hypothesis tests with:

• •effect size estimates (difference in means/proportions)
• •confidence intervals (often more interpretable)

Interpreting p-values responsibly

A correct interpretation template:

• •“Assuming H₀ is true, the probability of observing evidence at least as extreme as what we saw is p.”

Avoid:

• •“There is a p chance H₀ is true.” (That’s Bayesian territory, not what p-values mean.)
• •“p is the probability the result is random.” (Randomness is already in the model.)

Where this connects next

Hypothesis testing is closely tied to other core stats tools:

• •Confidence intervals: dual view of the same sampling logic; a two-sided α test corresponds to whether a (1−α) CI contains the null value.
• •Power and sample size: how likely you are to detect an effect when Hₐ is true.
• •Multiple testing: running many tests inflates false positives unless corrected.

Even if you don’t go deep into theory, the tail diagrams and “null distribution + shaded area” mental model will transfer directly.