Common Distributions

Why you should care

In real problems, you rarely invent a probability model from scratch. Instead, you pick a distribution family—a reusable pattern—and tune a few parameters θ to match the situation.

Examples:

• •A single yes/no outcome → Bernoulli(p)
• •Number of successes in n trials → Binomial(n, p)
• •Count of events over time/space with a stable average rate → Poisson(λ)
• •A value equally likely anywhere in an interval → Uniform(a, b)
• •A noisy measurement that clusters around a mean → Normal(μ, σ²)

Each family gives you:

• •A support (what values X can take)
• •A probability function: PMF (discrete) or PDF (continuous)
• •A small set of parameters θ (like p, λ, μ, σ)
• •Closed-form formulas for mean and variance

Support: discrete vs continuous

The first decision is the type of values X can take.

• •Discrete support: X ∈ {0, 1, 2, …} (or a finite set). Probabilities are assigned to points.
• •You use a PMF: p(x) = P(X = x)
• •Probabilities over sets A are sums:
• •P(X ∈ A) = ∑_{x ∈ A} p(x)

• •Continuous support: X ∈ ℝ or an interval. Probabilities are assigned to intervals (points have probability 0).
• •You use a PDF: f(x)
• •Probabilities over intervals are integrals:
• •P(X ∈ [c, d]) = ∫ᶜᵈ f(x) dx

A common confusion: for continuous X, f(x) is not itself a probability. It’s a density. You must integrate over an interval.

Parameterized families p(x | θ)

A distribution is often written as p(x | θ) (or f(x | θ)). The parameter set θ chooses one specific member of the family.

Examples of θ:

• •Bernoulli: θ = p where 0 ≤ p ≤ 1
• •Poisson: θ = λ where λ > 0
• •Normal: θ = (μ, σ²) where σ² > 0

These parameters control “location” (where the mass sits) and “scale/spread” (how variable it is).

A quick comparison table

In the next sections, we’ll build each one slowly: story → support → formula → mean/variance → how to compute probabilities.

Why discrete models show up everywhere

Many systems naturally produce counts:

• •Did a user click? (0/1)
• •How many defective items in a batch?
• •How many emails arrive in an hour?

Discrete distributions let you assign probability to each integer outcome and then sum the relevant probabilities.

Bernoulli(p)

Story

One trial. Two outcomes.

• •X = 1 means “success”
• •X = 0 means “failure”

Support and PMF

Support: X ∈ {0, 1}

PMF:

• •P(X = 1) = p
• •P(X = 0) = 1 − p

A compact way to write the PMF is:

• •p(x | p) = pˣ (1 − p)^(1−x), for x ∈ {0, 1}

Mean and variance

You’ll use these constantly:

• •E[X] = p
• •Var(X) = p(1 − p)

Why E[X] = p?

• •E[X] = 0·P(X=0) + 1·P(X=1) = p

Binomial(n, p)

Story

Repeat a Bernoulli trial n times, independently, with the same success probability p. Let X be the number of successes.

Examples:

• •# heads in n coin flips
• •# conversions among n visitors (with fixed conversion rate)

Support and PMF

Support: X ∈ {0, 1, …, n}

PMF:

• •P(X = k) = (n choose k) pᵏ (1 − p)^(n−k)

Where (n choose k) = n! / (k!(n−k)!).

Intuition for the formula

• •pᵏ (1 − p)^(n−k): probability of any particular sequence with k successes
• •(n choose k): number of sequences that have exactly k successes

Mean and variance

• •E[X] = np
• •Var(X) = np(1 − p)

A useful connection (preview of CLT): if n is large, a binomial can often be approximated by a normal:

• •X ≈ Normal(np, np(1 − p))

Poisson(λ)

Story

Poisson models counts of events in a fixed window when:

• •events occur independently (approximately)
• •the average rate is stable
• •two events don’t “pile up” at exactly the same instant (in a continuous-time idealization)

Examples:

• •calls arriving at a call center per minute
• •defects per meter of wire
• •decay events per time interval

Support and PMF

Support: X ∈ {0, 1, 2, …}

PMF:

• •P(X = k) = e^(−λ) λᵏ / k!

Here λ is both the rate parameter and the mean count per window.

Mean and variance

• •E[X] = λ
• •Var(X) = λ

That “mean = variance” fact is a diagnostic: if your observed counts have variance much bigger than the mean, a plain Poisson may be too simple.

Connection to binomial (rare events)

A classic approximation: if n is large and p is small but np stays moderate, then:

• •Binomial(n, p) ≈ Poisson(λ) with λ = np

This is the “rare events” regime.

Discrete probability calculations: summing

If X is discrete, you compute:

• •P(X ∈ A) = ∑_{x ∈ A} p(x)

Examples:

• •For Poisson: P(X ≥ 2) = 1 − P(X=0) − P(X=1)
• •For Binomial: P(X ≤ k) = ∑_{i=0}^k (n choose i) pⁱ (1−p)^(n−i)

In practice, you often use a CDF function from software for these sums, but it’s crucial to understand what is being summed and why.

Why continuous models matter

Many measurements are not naturally integer counts:

• •time, temperature, distance, voltage, weight

Even if the world is measured with finite precision, continuous distributions are often excellent approximations and give smooth, usable mathematics.

The key mental shift:

• •For continuous X, P(X = x) = 0 for any exact x.
• •You compute probabilities by integrating a density over an interval.

Uniform(a, b)

Story

“All values between a and b are equally plausible.”

Examples:

• •a randomized start time between 2pm and 3pm
• •an unknown phase angle assumed equally likely in [0, 2π]

Support and PDF

Support: X ∈ [a, b]

PDF:

• •f(x) = 1/(b − a) for a ≤ x ≤ b
• •f(x) = 0 otherwise

Probability of an interval

For any c, d with a ≤ c ≤ d ≤ b:

P(c ≤ X ≤ d)

= ∫ᶜᵈ 1/(b − a) dx

= (d − c)/(b − a)

So probability is proportional to interval length.

Mean and variance

• •E[X] = (a + b)/2
• •Var(X) = (b − a)² / 12

Normal(μ, σ²)

Story

The normal (Gaussian) distribution models values that cluster around an average μ with symmetric noise.

It also appears because of aggregation:

• •sums/averages of many small effects often become approximately normal (CLT)

Examples:

• •measurement error
• •heights (roughly)
• •sensor noise

Support and PDF

Support: X ∈ ℝ

PDF:

• •f(x) = (1/(σ√(2π))) exp(−(x−μ)²/(2σ²))

Parameters:

• •μ controls the center
• •σ (or σ²) controls the spread

Standardization (Z-scores)

A core technique is converting to the standard normal.

If X ∼ Normal(μ, σ²), define:

• •Z = (X − μ)/σ

Then Z ∼ Normal(0, 1).

This lets you use standard normal tables or software CDFs:

• •P(X ≤ x) = P(Z ≤ (x−μ)/σ)

The 68–95–99.7 rule (intuition)

For X ∼ Normal(μ, σ²):

• •P(μ − σ ≤ X ≤ μ + σ) ≈ 0.68
• •P(μ − 2σ ≤ X ≤ μ + 2σ) ≈ 0.95
• •P(μ − 3σ ≤ X ≤ μ + 3σ) ≈ 0.997

This is not a definition—just a helpful memory.

Continuous probability calculations: integrating

If X is continuous with PDF f:

• •P(X ∈ A) = ∫_{x ∈ A} f(x) dx

For an interval [c, d]:

• •P(c ≤ X ≤ d) = ∫ᶜᵈ f(x) dx

For the uniform, this integral is easy.

For the normal, there is no elementary antiderivative, so we use the CDF Φ(z) numerically.

A practical comparison:

Understanding this split (support → PMF/PDF → sum/integrate) prevents many downstream mistakes in statistics and ML.

Why model choice matters

A distribution is a compact set of assumptions. Choosing one is not just “picking a formula”—it’s deciding what outcomes are possible and what patterns are likely.

A good first pass is to match the data type and generative story.

Quick chooser: which distribution should I try?

Then sanity-check with:

• •support: does the distribution allow impossible values?
• •mean/variance: are they in the right ballpark?
• •shape: symmetric vs skewed; heavy tail vs light tail

How this connects to Maximum Likelihood Estimation (MLE)

In MLE, you assume data x₁, …, xₙ came from a distribution family p(x | θ) and pick θ that makes the observed data most likely.

Examples you’ll soon see:

• •Bernoulli/Binomial: MLE for p is the sample proportion
• •Poisson: MLE for λ is the sample mean
• •Normal: MLE for μ is the sample mean, and for σ² is the sample variance (with a particular denominator choice)

To do MLE well, you must recognize which likelihood matches your data (Bernoulli vs binomial vs Poisson, etc.).

How this connects to Bayesian inference

Bayesian inference updates distributions with data:

• •posterior ∝ likelihood × prior

The likelihood often comes from a “common distribution.” Examples:

• •Bernoulli likelihood + Beta prior → Beta posterior
• •Poisson likelihood + Gamma prior → Gamma posterior
• •Normal likelihood + Normal prior (for μ) → Normal posterior

Knowing the likelihood family is step 1.

How this connects to joint distributions and the CLT

• •Joint distributions: if you have multiple random variables (X, Y), you often assume they are independent with known marginals (e.g., X ∼ Poisson(λ₁), Y ∼ Poisson(λ₂)). From there you can build joint models.
• •Central Limit Theorem (CLT): sums of many independent variables often become approximately normal. That’s why normal approximations to binomial are common, and why normal noise models appear everywhere.

One more pacing note: models are approximations

A distribution is rarely “true.” It’s a simplified story that is useful if:

• •it matches the support
• •it captures the main shape
• •it predicts well enough for your decision

As you learn more, you’ll add richer families, but these five are the workhorses you’ll keep returning to.