Bayesian Inference

This node builds on ideas you may already know, but it’s easy to get tripped up by missing one small piece. Here’s the explicit checklist.

Required prerequisites

1) Bayes’ theorem and conditional probability

You should be comfortable with:

• •Conditional probability: $P(A\mid B)=\frac{P(A\cap B)}{P(B)}$.
• •Bayes’ theorem:

You should also understand the “proportional” form:

where the missing factor is “whatever makes it sum/integrate to 1.” Bayesian inference uses this proportionality constantly.

2) Common distributions (Bernoulli/binomial/Poisson/normal)

You should recognize probability mass/density functions and their parameters.

• •Bernoulli: $x\in\{0,1\}$, parameter $\theta$.
• •Binomial: counts of successes in $n$ trials.
• •Poisson: counts over time/space.
• •Normal: mean/variance.

3) Likelihood and MLE

You should know that the likelihood is the same expression as $p(x\mid \theta)$ but interpreted as a function of $\theta$ for fixed observed $x$.

MLE chooses:

Bayesian inference will generalize this: it returns a distribution over $\theta$ instead of one optimizer.

Helpful (but optional) prerequisite: calculus/integration intuition

The “evidence” (also called the marginal likelihood) is:

(or a sum for discrete $\theta$). If you don’t have calculus yet, you can still learn most of Bayesian inference by treating this as “the normalization constant” and focusing on proportional reasoning:

You can do many practical updates with conjugate priors without doing the integral yourself.

A crucial clarification (common misconception)

People often say “use a flat/uninformative prior.” Two important caveats:

1) ‘Flat’ depends on parameterization. A prior that is uniform in $\theta$ is not uniform in $\phi=g(\theta)$. For example, if $\phi=\theta^2$, then a uniform prior in $\theta$ induces a non-uniform prior in $\phi$.

2) “Non-informative” is subtle. Some priors are designed to be less informative under reparameterizations (e.g., Jeffreys priors), but there is no universal free lunch.

Keep this in mind as we talk about priors: they encode assumptions, and assumptions should be made explicit.

The big idea: uncertainty about parameters is a first-class object

In frequentist statistics, parameters are fixed but unknown. In Bayesian statistics, parameters are treated as uncertain quantities described by a probability distribution.

You observe data $x$ (which might be a dataset like $x=(x_1,\dots,x_n)$), and you want to reason about an unknown parameter (or parameters) $\theta$.

Bayesian inference is the process of updating your beliefs about $\theta$ after seeing $x$.

The core equation

Bayes’ theorem in density form is:

Each term has a distinct job:

• •Prior $p(\theta)$: what you believe about $\theta$ before seeing this data.
• •Likelihood $p(x\mid\theta)$: how likely the observed data is if $\theta$ were the true parameter.
• •Posterior $p(\theta\mid x)$: what you believe after seeing data.
• •Evidence $p(x)$: the normalization constant making the posterior integrate to 1.

Often we write the update in proportional form:

That proportional form is not a shortcut; it’s a mindset: start by multiplying prior × likelihood, then normalize.

Why this is more than “just Bayes’ theorem”

Bayes’ theorem is a single identity. Bayesian inference is a workflow:

1) Choose a probabilistic model for data: $p(x\mid\theta)$.

2) Choose a prior over unknowns: $p(\theta)$.

3) Compute or approximate the posterior: $p(\theta\mid x)$.

4) Use the posterior for decisions/predictions.

This workflow forces you to express assumptions.

A useful mental picture: prior × likelihood = unnormalized posterior

Suppose $\theta$ is one-dimensional.

• •The prior is a curve over $\theta$.
• •The likelihood is another curve over $\theta$ (for the fixed observed data).
• •Multiplying them gives a curve that is large where both agree.
• •Normalization rescales that product so the area equals 1.

This “agreement by multiplication” is the heart of Bayesian updating.

Bayesian inference vs MLE (how they relate)

MLE finds a point estimate maximizing the likelihood.

Bayesian inference produces a distribution. But you can recover point estimates from the posterior:

• •MAP estimate (maximum a posteriori):

• •If the prior is uniform (and you accept that choice), MAP and MLE coincide.

The key difference: Bayesian inference quantifies uncertainty and naturally supports predictive distributions (integrating over $\theta$).

Start with the data-generating story

A Bayesian model usually begins with a story:

1) Nature draws a parameter $\theta$ from a prior $p(\theta)$.

2) Then Nature generates data $x$ from $p(x\mid\theta)$.

We only observe $x$. Bayesian inference asks: given $x$, what should we believe about $\theta$?

Prior p(θ): encoding beliefs and constraints

A prior can do several jobs:

• •Encode domain knowledge (e.g., a coin is probably not extremely biased).
• •Enforce constraints (e.g., $\theta\in[0,1]$ for probabilities).
• •Regularize inference (prevent extreme estimates with small data).

Example: probability parameter

If $\theta$ is a probability (like a Bernoulli success rate), then a natural prior is the Beta distribution:

Interpretation (informally): $\alpha-1$ looks like prior “successes,” $\beta-1$ like prior “failures.”

Likelihood p(x|θ): a function of θ when x is fixed

This is a common conceptual speed bump.

• •As a probability model, $p(x\mid\theta)$ is a distribution over possible data $x$ given $\theta$.
• •As a likelihood, $L(\theta)=p(x\mid\theta)$ is a function of $\theta$ for the observed $x$.

Important: likelihoods are not probability distributions over $\theta$, so they do not need to integrate to 1 over $\theta$.

IID datasets and likelihood factorization

If $x=(x_1,\dots,x_n)$ are IID given $\theta$, then:

That product is why data accumulates evidence quickly.

Posterior p(θ|x): updated belief

The posterior is what you use for:

• •uncertainty intervals (credible intervals),
• •point summaries (posterior mean, MAP),
• •predictive distributions (posterior predictive),
• •decision-making (expected utility).

Evidence p(x): the normalization constant with hidden power

The evidence is:

You can think of it as:

• •The probability of seeing $x$ under the whole model (prior + likelihood).
• •A measure of how well the model predicts the data before seeing it.

This becomes central in model comparison (Bayes factors), because it penalizes overly flexible models that spread probability mass too thin.

A compact comparison table

The “Bayesian update” as a sequence

If you observe data in chunks, Bayes updates are consistent.

Let data arrive as $x^{(1)}$ then $x^{(2)}$. Then:

So yesterday’s posterior becomes today’s prior. This is not just poetic; it’s computationally useful and conceptually clean.

Why conjugate priors matter

The posterior requires multiplying and normalizing:

Sometimes, that product lands in the same family as the prior. Then the posterior has a closed form, and updating is easy.

That pairing is called conjugacy.

Conjugacy is not required for Bayesian inference, but it’s the clearest way to learn the mechanics.

Beta–Binomial: the canonical example

Assume $x$ is the number of successes in $n$ Bernoulli trials with success probability $\theta$.

• •Likelihood:

• •Prior: $\theta\sim \text{Beta}(\alpha,\beta)$.

Compute the unnormalized posterior:

So:

This reveals the “pseudo-count” intuition: successes add to $\alpha$, failures add to $\beta$.

Gamma–Poisson: rates for count data

If data are Poisson with rate $\lambda$:

A conjugate prior is:

(using the rate-parameterization where density is proportional to $\lambda^{\alpha-1}e^{-\beta\lambda}$).

With $n$ IID observations and $S=\sum_i x_i$:

Again: data adds to shape, and the number of observations adds to rate.

Normal–Normal: unknown mean with known variance

If:

and prior:

then the posterior is also normal. The posterior mean becomes a precision-weighted average of the prior mean and sample mean.

Define precision as inverse variance: $\kappa=1/\sigma^2$, $\kappa_0=1/\tau_0^2$.

Let $\bar x=\frac{1}{n}\sum_i x_i$. Then:

So:

This shows a deep Bayesian theme: uncertainty shrinks with data.

Posterior predictive: predicting new data

Bayesian inference shines when you want to predict future observations $x_{\text{new}}$.

Instead of plugging in a single estimate of $\theta$, you average over the posterior:

This is called the posterior predictive distribution.

Intuition: if you are uncertain about $\theta$, your predictions should reflect that uncertainty.

Example intuition (no heavy math)

• •If the posterior over a coin’s bias is wide, your predictive probability of heads is not just “one number”; it’s informed by that width.
• •With little data, predictions are more conservative.

Credible intervals (Bayesian) vs confidence intervals (frequentist)

A Bayesian credible interval is a probability statement about the parameter:

A frequentist 95% confidence interval is a statement about repeated sampling behavior of the interval procedure, not directly about the realized parameter.

Both can be useful, but do not automatically interpret them the same way.

A gentle note on computation

Conjugate priors are beautiful, but many real models are not conjugate.

In those cases:

• •you might approximate integrals (variational inference),
• •or sample from the posterior (MCMC),
• •or use Laplace approximations.

This node focuses on building the conceptual and algebraic foundation that those methods rely on.

Why Bayesian inference is a cornerstone

Bayesian inference gives you three capabilities that show up everywhere in modern ML and statistics:

1) Uncertainty-aware learning (not just point estimates)

2) Principled regularization via priors

3) Model comparison via evidence

Let’s connect those to the nodes this unlocks.

1) Latent-variable generative modeling (Variational Autoencoders)

VAEs introduce latent variables z and parameters θ.

A typical generative story:

• •Sample latent z from a prior $p(\mathbf{z})$.
• •Generate data x from $p_\theta(\mathbf{x}\mid\mathbf{z})$.

Inference asks for the posterior over latent variables:

But $p_\theta(\mathbf{x})=\int p_\theta(\mathbf{x}\mid\mathbf{z})p(\mathbf{z})d\mathbf{z}$ is usually intractable, so we approximate with variational inference (ELBO). That is Bayesian inference scaled up.

2) Sampling-based inference (MCMC)

When the posterior is complex:

MCMC constructs a Markov chain whose stationary distribution is the posterior, enabling:

• •posterior means/variances via Monte Carlo,
• •credible intervals,
• •posterior predictive checks.

The target distribution MCMC needs is exactly the Bayesian posterior (often only known up to a normalization constant, which is fine for many MCMC algorithms).

3) Bayesian optimization

Bayesian optimization maintains a posterior over functions or surrogate-model parameters (often Gaussian processes). Data updates a prior to a posterior, then an acquisition function uses that posterior uncertainty to pick the next point to evaluate.

The key idea is exploration vs exploitation driven by posterior uncertainty.

4) Auction theory and beliefs about private values

In auction settings, bidders and the designer reason about unknown private valuations and types. Bayesian models represent beliefs about those unknowns and update from signals or observed behavior. “Bayesian” in mechanism design often literally refers to priors over types.

5) Causal inference

Many causal workflows use Bayesian inference to:

• •estimate treatment effects with uncertainty,
• •combine prior knowledge with data,
• •perform hierarchical modeling (partial pooling).

Even when causal identification is a separate question, Bayesian inference is frequently the engine used once a causal estimand is defined.

A final connection: regularization and MAP

If you’ve seen L2 regularization in regression, there is a Bayesian interpretation:

• •Gaussian prior on weights → L2 penalty in MAP.

This is a bridge between optimization-based ML and probabilistic modeling.

Summary of what you should now be ready for

After this node, you should be comfortable with:

• •reading and writing $p(\theta\mid x)$,
• •distinguishing prior vs likelihood,
• •computing simple conjugate updates,
• •understanding why evidence/normalization matters,
• •seeing why approximate inference methods exist.

That’s the conceptual toolkit you need before you dive into MCMC, variational inference, VAEs, Bayesian optimization, and Bayesian causal modeling.