Conjugate Priors

Definition and motivation

A conjugate prior for a likelihood family is a prior distribution p(θ) such that the posterior p(θ | data) lies in the same parametric family as p(θ). Conjugacy is valuable because it turns Bayesian Inference (the prerequisite where we update priors with likelihoods to get posteriors) into simple algebraic updates instead of requiring numerical integration or sampling.

Formal statement

If the likelihood p(x | θ) belongs to a family L and the prior p(θ) belongs to a family P, we say P is conjugate to L if for any dataset x the posterior p(θ | x) ∈ P. The algebraic consequence is that the posterior parameters are updated by adding data-dependent terms to prior parameters: these update rules are closed form and often involve sufficient statistics (sums, counts, means).

Why care?

• •Computational efficiency: closed-form updates avoid MCMC when the model uses conjugate pairs.
• •Interpretability: updates are simple additions (e.g., add successes and failures to Beta parameters).
• •Predictive distributions: marginalizing parameters often yields known, simple predictive distributions (Beta-Binomial, Negative Binomial from Gamma-Poisson mix).
• •Building blocks: conjugate pairs form the core of hierarchical models, empirical Bayes, Kalman filters, and message passing.

A simple motivating example (Beta-Binomial)

In Common Distributions (prerequisite) we learned the Binomial likelihood for n Bernoulli trials with success probability θ and the Beta distribution as a flexible prior on θ. The Beta(a,b) density is

The Binomial likelihood for k successes in n trials is

Multiply prior and likelihood to get posterior up to normalization:

which is again a Beta distribution: Beta(a+k, b+n-k). This is conjugacy: Beta is conjugate to Binomial.

Numeric example (concrete)

Let the prior be Beta(2,3) (so a=2, b=3). Observe k=8 successes in n=20 trials. Posterior is:

So the posterior mean is $\frac{10}{10+15}=\frac{10}{25}=0.4$. This single arithmetic update — add successes to a, failures to b — is the essence of conjugacy.

Intuition from sufficient statistics

Conjugate updates typically depend only on a small set of summary statistics of the data (sufficient statistics). For Binomial/Bernoulli that statistic is the count of successes k (or equivalently the sample mean times n). This is the same reason the Beta prior only needs two parameters (a,b): they act like pseudo-counts that add to the observed counts.

Preview of families we will study

• •Beta-Binomial (Bernoulli/Binomial likelihood + Beta prior)
• •Gamma-Poisson (Poisson likelihood + Gamma prior)
• •Normal-Normal (Normal likelihood with known variance + Normal prior on mean)

Each produces closed-form posterior parameters, posterior predictive distributions with closed forms, and clear sufficient statistics.

Beta-Binomial conjugacy — derivation and predictive distribution

Setup. Observations x_1, ..., x_n are iid Bernoulli(θ) (or equivalently we observe k successes in n Binomial(n,θ) trials). Prior: θ ~ Beta(a,b). Likelihood: p(k|θ)=\binom{n}{k}θ^{k}(1-θ)^{n-k}. Posterior (derive):

Hence

Numeric example. Prior Beta(1,1) (uniform), observe k=30 successes in n=100. Posterior is Beta(1+30, 1+70)=Beta(31,71). Posterior mean = 31/102 ≈ 0.3039.

Posterior predictive for a single future Bernoulli trial

The posterior predictive probability that the next trial is a success is

Numeric example: with Beta(1,1) prior and k=30, n=100, this equals (1+30)/(1+1+100)=31/102 ≈ 0.3039.

Predictive for m future trials (Beta-Binomial)

The predictive distribution for the number y of successes in m future trials (m>0) is the Beta-Binomial:

where B(·,·) is the Beta function. Numeric example: with prior Beta(2,3), observed k=8,n=20, compute probability of y=2 successes in m=5 future trials:

Posterior parameters: a'=10, b'=15. Then

You can compute B via Gamma: B(12,18)=\Gamma(12)\Gamma(18)/\Gamma(30) etc., or use software. The closed form keeps things analytic.

Gamma-Poisson conjugacy — derivation and predictive distribution

Setup. Observed counts x_1,...,x_n drawn iid Poisson(λ). Likelihood:

Prior: λ ~ Gamma(α, β) with the rate parametrization (shape α, rate β) so

Multiply prior and likelihood (ignore data-only constants):

This is Gamma(α + Σ x_i, β + n). Numeric example: prior Gamma(2,1) (α=2,β=1). Data x = {3,2,4} => Σ x_i = 9, n=3. Posterior: Gamma(2+9, 1+3) = Gamma(11,4). Posterior mean = (α')/(β') = 11/4 = 2.75.

Predictive distribution for a new observation

Marginalizing λ yields a closed-form predictive for a new count x':

where α' = α+Σ x_i and β' = β+n. This is a form of the (Poisson–Gamma) Negative Binomial-like mixture. Numeric example: With prior Gamma(2,1) and data {3,2,4} we had α'=11, β'=4. The predictive probability that next count x'=2 is

Compute numerically: Γ(13)/Γ(11)=11·12=132, so p=132/(2) · (4/5)^{11} · (1/5)^2 =66 · (4/5)^{11} · (1/25). You can plug values to obtain a numerical probability.

What these two conjugate pairs teach

• •Posterior parameter updates are additive: add observed counts or sums to prior pseudo-counts or pseudo-sums.
• •Posterior predictive distributions are mixtures with closed forms, useful for forecasting and model checking.
• •The sufficient statistics (k for Binomial, Σx for Poisson) are the only necessary summaries of the data for updating the prior.

Normal-Normal conjugacy (known variance) — closed-form posterior and derivation

Setup. Observations x_1,...,x_n are iid Normal(μ, σ^2) with known variance σ^2. Prior on the mean: μ ~ Normal(μ_0, τ_0^2). This is the standard conjugate pair for location with known variance. The posterior for μ is Normal(μ_n, τ_n^2) with

Derivation (completing the square). The likelihood is

The prior is

Multiplying and completing the square in μ gives a Normal posterior with precision (inverse variance) equal to the sum of prior precision and data precision: $1/\tau_n^2 = 1/\tau_0^2 + n/\sigma^2$. The posterior mean is a precision-weighted average of μ_0 and the sample mean \bar{x}.

Numeric example. Prior μ_0=0, τ_0^2=1. Known σ^2=2. Data: x = {1.2, 0.8, 1.5} so n=3, \bar{x}=(1.2+0.8+1.5)/3=3.5/3≈1.1667.

Compute precisions: 1/τ_0^2 = 1, n/σ^2 = 3/2 = 1.5, so 1/τ_n^2 = 2.5 => τ_n^2 = 0.4. Posterior mean μ_n = 0.4(0/1 + 31.1667/2) = 0.4*(1.75005) ≈ 0.70002. So μ|data ≈ N(0.7000, 0.4).

Posterior predictive for a new observation x_{new}

The predictive distribution integrates over μ and is Normal with mean μ_n and variance σ^2 + τ_n^2:

Numeric example: variance = 2 + 0.4 = 2.4, so predictive standard deviation ≈ 1.549.

Sufficient statistics and reduction of data

For Normal with known variance, the sufficient statistic for μ is the sample mean \bar{x} (and n). That is, the posterior depends only on n and \bar{x}, not on the full sample. In other words, the data reduce to two numbers: n and \bar{x}. This mirrors the Beta-Binomial case where the sufficient statistic is k.

Exponential family and general conjugacy structure

Many common likelihoods (Bernoulli/Binomial, Poisson, Normal, Exponential, Gamma, etc.) belong to the exponential family. A one-parameter exponential family has density (w.r.t. base measure):

where η is the natural parameter, T(x) is the sufficient statistic for a single observation, and A(η) is the log-partition function.

A natural conjugate prior for η has the form

where ξ and ν are hyperparameters encoding prior pseudo-observations and g normalizes. After observing n iid samples with sufficient statistic S = Σ T(x_i), the posterior becomes

so updates are additive: ξ' = ξ + S, ν' = ν + n. This is the generic conjugate-update pattern.

Concrete mapping to Beta-Binomial

For Bernoulli, T(x)=x and A(η)=\log(1+e^{η}). The Beta prior in θ-space corresponds to a conjugate prior in the natural parameter space via a change of variables; however, the additive-pseudo-count view is simplest: in Beta(a,b) the parameters (a-1, b-1) can be seen as pseudo-success and pseudo-failure counts that add to observed counts.

Why this matters

• •Exponential-family conjugacy yields compact, interpretable updates and exposes sufficient statistics.
• •It provides the algebraic backbone for many Bayesian algorithms (e.g., variational Bayes, belief propagation) where conjugacy leads to closed-form coordinate updates.
• •Recognizing natural parameters and sufficient statistics lets you derive conjugate priors for new exponential-family models quickly.

Practical use cases

1) A/B testing and online experimentation

In A/B tests with binary outcomes (click/no-click), Beta-Binomial conjugacy lets you update beliefs about conversion probabilities in real time. Example: uniform-prior Beta(1,1). Group A observes 30 successes in 100 trials → Beta(31,71). Group B observes 40 successes in 120 trials → Beta(41,81). The probability that p_A > p_B can be computed analytically via Beta integrals or approximated via Monte Carlo sampling from the two Betas. These closed forms make sequential decision rules easy (stop when P(p_A>p_B)>0.95).

Numeric snippet (compare posteriors). Posterior means: A = 31/102 ≈ 0.304, B = 41/122 ≈ 0.336. But the posterior distributions capture uncertainty; the probability p_A>p_B requires integrating their joint Beta densities and is tractable numerically.

2) Count data and rare-event smoothing

For estimating rates (traffic accidents per month, email arrivals), Poisson likelihood with Gamma prior gives a Gamma posterior. The Gamma prior acts as regularization: for small-sample areas it pulls extreme sample means toward the prior mean (empirical Bayes uses pooled data to set hyperparameters). Predicting future counts uses the Gamma–Poisson mixture (negative-binomial predictive) to capture over-dispersion beyond a simple Poisson.

3) Measurement, sensor fusion, and filtering

Normal-Normal conjugacy with known variance is exactly the static version of the Kalman filter's update step: combine prior estimate and new measurements weighted by their precisions. In many engineering settings where sensor noise variance is known, these closed-form updates are used continuously.

4) Hierarchical models and empirical Bayes

Conjugate models are building blocks for hierarchical Bayes. For example, modeling click rates p_i for many websites with p_i ~ Beta(α,β) and data Binomial for each site yields closed-form per-site posteriors. Empirical Bayes sets α,β via marginal likelihood (which can be computed for conjugate families), giving shrinkage estimates that pool information across units.

Connections to downstream methods

• •Variational inference and message passing exploit exponential-family conjugacy to produce closed-form coordinate updates. Learning these conjugate pairs prepares you to derive variational updates.
• •Gibbs sampling: when conditional posteriors are from conjugate families, Gibbs samplers simplify (draw from standard distributions instead of generic MCMC steps).
• •Predictive modeling: closed-form posterior predictive distributions supply analytic credible intervals and predictive checks.

Numeric case study (decision example)

Suppose a small clinic records 2 successes in 5 trials for a new treatment. With prior Beta(1,1), posterior Beta(3,4). The expected success probability is 3/7 ≈ 0.4286 and the predictive probability of next trial success is (3)/(7+1?) — careful: the predictive for a single Bernoulli is (a')/(a'+b') = 3/7 ≈ 0.4286 (same as posterior mean). If a competing treatment has posterior predictive 0.35, you might choose the new treatment. Conjugacy makes these computations trivial and transparent.

When conjugacy is not enough

Real models often lack conjugacy (complex likelihoods, non-exponential-family components). Still, conjugate models serve as approximations, initialization for numerical methods, or local updates within larger models (e.g., conditional conjugacy in parts of hierarchical models).

Summary

Conjugate priors give you closed-form Bayesian updates, analytic predictive distributions, and clear sufficient statistics. Recognizing conjugacy (or mapping a model into the exponential family) buys tractability and insight — foundational for modern Bayesian computation and modeling.