Bayes Theorem

Why this concept exists (motivation)

In real problems you often face the direction mismatch:

• •You can estimate how likely some evidence is given a hypothesis (e.g., “if a patient has the disease, how likely is a positive test?”).
• •But you want the reverse: how likely the hypothesis is given evidence (e.g., “given a positive test, how likely is disease?”).

Bayes’ Theorem bridges this gap. It’s also the mathematical backbone of “updating beliefs with evidence”: start with an initial belief (a prior), observe data (the evidence), and compute an updated belief (the posterior).

The statement

For events A and B, with P(B) > 0:

P(A|B) = P(B|A)P(A) / P(B)

You’ll see several names for each term:

• •Prior: P(A)
• •Likelihood: P(B|A)
• •Posterior: P(A|B)
• •Evidence / marginal likelihood / normalizer: P(B)

Even if the formula feels simple, the meaning is subtle: probability is redistributed across hypotheses when evidence arrives.

Where it comes from (definition-level derivation)

Bayes’ Theorem is not a “special trick”—it is a direct consequence of the definition of conditional probability.

Start with the definition:

P(A|B) = P(A ∩ B) / P(B)

Also:

P(B|A) = P(A ∩ B) / P(A)

Solve the second equation for P(A ∩ B):

P(A ∩ B) = P(B|A)P(A)

Plug into the first equation:

P(A|B)

= P(A ∩ B) / P(B)

= [P(B|A)P(A)] / P(B)

That’s Bayes’ Theorem.

Intuition in one sentence

Posterior = (how well A predicts the evidence) × (how plausible A was) ÷ (how surprising the evidence is overall).

A note about language: hypothesis vs evidence

Often we treat:

• •A = “hypothesis is true”
• •B = “we observed some evidence”

But the theorem is symmetric: you can swap roles depending on what you’re trying to compute.

When it’s most useful

Bayes is most useful when:

1) You can model P(B|A) (data given hypothesis) more easily than P(A|B).

2) You have multiple competing hypotheses (A₁, A₂, …) and want to update which is most plausible.

3) Base rates matter (priors), and ignoring them would lead to bad conclusions.

Why separate probability into these pieces?

When you write P(A|B) directly, it can hide structure. Bayes explicitly factors the update into:

• •What you believed before (prior)
• •How compatible the evidence is with the hypothesis (likelihood)
• •A normalization step so everything sums to 1 (evidence)

This decomposition is powerful because each piece comes from a different source:

• •Priors can come from historical rates, domain knowledge, or earlier data.
• •Likelihood comes from a measurement model (sensor accuracy, test sensitivity/specificity).
• •Evidence comes from total probability over all ways B could happen.

The “belief update” view

Suppose A is a hypothesis and B is a newly observed fact.

Bayes:

P(A|B) ∝ P(B|A)P(A)

Read “∝” as “proportional to.” This is the key intuition:

• •Start with P(A) (prior mass).
• •Multiply by P(B|A) (compatibility with evidence).
• •Then renormalize so probabilities sum to 1.

This proportionality form is often how you reason informally:

1) Hypotheses that better predict the evidence get boosted.

2) Hypotheses that poorly predict the evidence get penalized.

3) Priors still matter—rare things stay rare unless the evidence is very strong.

Discrete hypotheses: odds-style thinking

Imagine two hypotheses A and ¬A (not A). Using Bayes on both:

P(A|B) = P(B|A)P(A)/P(B)

P(¬A|B) = P(B|¬A)P(¬A)/P(B)

Take the ratio (posterior odds):

P(A|B) / P(¬A|B)

= [P(B|A)P(A)] / [P(B|¬A)P(¬A)]

This shows:

Posterior odds = Likelihood ratio × Prior odds

Where likelihood ratio = P(B|A) / P(B|¬A).

This is useful because the annoying P(B) cancels, and you can see the “strength of evidence” as a ratio.

What each term means (carefully)

Let’s tie each term to an interpretation you can check:

• •Prior P(A): Before seeing B, how probable is A?
• •Likelihood P(B|A): If A were true, how often would we see B?
• •Posterior P(A|B): After seeing B, how probable is A?
• •Evidence P(B): In the whole world (across all hypotheses), how often would we see B?

A common conceptual trap is to confuse P(B|A) with P(A|B). They can be wildly different.

A miniature numeric example (no full story yet)

Suppose:

• •P(A) = 0.01 (A is rare)
• •P(B|A) = 0.90 (evidence is common if A is true)
• •P(B|¬A) = 0.05 (false alarm rate)

We’ll compute P(A|B), but first note: we need P(B). That’s not optional.

Using total probability:

P(B) = P(B|A)P(A) + P(B|¬A)P(¬A)

Compute:

P(B)

= 0.90·0.01 + 0.05·0.99

= 0.009 + 0.0495

= 0.0585

Now Bayes:

P(A|B)

= (0.90·0.01) / 0.0585

= 0.009 / 0.0585

≈ 0.1538

Even with strong evidence, a rare prior can keep the posterior moderate.

Summary table: “parts of Bayes”

Why P(B) is the “normalizing constant”

Bayes’ Theorem divides by P(B). This ensures that the posterior is a valid probability.

If you compute unnormalized weights:

w(A) = P(B|A)P(A)

then the normalized posterior is:

P(A|B) = w(A) / ∑ₖ w(Aₖ)

Where {Aₖ} are mutually exclusive, exhaustive hypotheses.

So P(B) is exactly:

P(B) = ∑ₖ P(B|Aₖ)P(Aₖ)

That is the Law of Total Probability.

Two-hypothesis case: A vs ¬A

If hypotheses are A and ¬A:

P(B)

= P(B|A)P(A) + P(B|¬A)P(¬A)

This formula is the workhorse behind medical-test and spam-filter calculations.

Multi-class case: A₁, A₂, …, Aₙ

If you have n hypotheses:

• •Aᵢ ∩ Aⱼ = ∅ for i ≠ j
• •∪ᵢ Aᵢ is the whole sample space

Then:

P(B) = ∑ᵢ P(B|Aᵢ)P(Aᵢ)

And Bayes becomes:

P(Aᵢ|B) = P(B|Aᵢ)P(Aᵢ) / ∑ⱼ P(B|Aⱼ)P(Aⱼ)

Why this matters conceptually

The evidence term answers: “How often would we see B regardless of which hypothesis is true?”

• •If B is very common overall, then seeing it doesn’t tell you much (posterior stays close to prior).
• •If B is rare overall, then seeing it is informative, and posteriors can shift dramatically.

This matches everyday reasoning: a surprising clue carries more information than a mundane one.

A common technique: compute numerator and denominator separately

When doing Bayes problems by hand:

1) Compute the numerator: P(B|A)P(A).

2) Compute P(B) using total probability.

3) Divide.

This prevents mistakes like forgetting the ¬A term or miscomputing complements.

Visualization intuition: probability mass reallocation

Think of the prior probabilities across hypotheses as “mass” that sums to 1.

• •Multiplying by P(B|Aᵢ) shrinks/expands each mass depending on how consistent that hypothesis is with the evidence.
• •Dividing by P(B) rescales so the total mass returns to 1.

This is the essence of Bayesian updating.

Small checklist for correctness

Before finalizing a Bayes calculation:

• •Are your hypotheses mutually exclusive and exhaustive?
• •Did you compute P(B) as a weighted sum over hypotheses?
• •Does your posterior stay in [0, 1]?
• •If evidence is uninformative (P(B|A) ≈ P(B|¬A)), does posterior stay near prior?

Why Bayes shows up everywhere

Bayes’ Theorem is the simplest formal model of learning from data:

• •You have uncertainty about a hidden state (disease / spam / which class).
• •You observe a noisy signal (test result / words / sensor output).
• •You infer the hidden state.

Even many modern ML systems can be described as “compute something proportional to likelihood × prior, then normalize.”

Medical testing (base rate matters)

Medical test problems are the classic Bayes showcase because humans often ignore priors.

Key terms you’ll see:

• •Sensitivity = P(positive | disease)
• •Specificity = P(negative | no disease)
• •False positive rate = P(positive | no disease) = 1 − specificity

What you usually want is:

P(disease | positive)

That is Bayes with A = disease, B = positive.

The important lesson: even a highly accurate test can yield many false positives when the disease is rare.

Spam filtering / text classification (discrete version)

Suppose:

• •A = “email is spam”
• •B = “email contains the word ‘free’”

Then Bayes says:

P(spam | contains ‘free’)

= P(contains ‘free’ | spam)P(spam) / P(contains ‘free’)

This is the skeleton of naïve Bayes classifiers (where B is many word-features). You will later learn more advanced versions, but the core update logic is identical.

Sensor fusion / robotics (belief update repeated over time)

In tracking problems:

• •Prior represents your belief about a robot’s position before the latest sensor reading.
• •Likelihood represents sensor noise (how likely a reading is for each position).
• •Posterior is your updated belief.

Repeated Bayes updates over time lead to filters like the Kalman filter and particle filter (conceptually Bayesian, though implementation details differ).

Bayes as the gateway to Bayesian inference

Bayes’ Theorem for events is the entry point to Bayes for distributions.

Event version:

P(A|B) = P(B|A)P(A)/P(B)

Distribution version (preview):

p(θ|D) ∝ p(D|θ)p(θ)

Where:

• •θ is a parameter (a continuous hypothesis)
• •D is data
• •p(·) are probability densities

This node unlocks that next step.

Quick comparison: Frequentist vs Bayesian (high-level)

You don’t need to “choose a side” to use Bayes’ Theorem; you just need to be clear about what probabilities represent in your problem.