Joint Distributions

Why we need it

Single-variable distributions (Bernoulli, Poisson, Normal, …) answer questions like “How likely is X = 3?” But many systems involve multiple random variables simultaneously.

Examples:

• •X = number of visitors to a website, Y = number of purchases.
• •X = a patient’s test result, Y = underlying condition.
• •X = temperature, Y = electricity demand.

To model relationships, we need a distribution over tuples of outcomes.

Definition (discrete)

Let X and Y be discrete random variables. The joint pmf is

p_{X,Y}(x,y) = P(X = x, Y = y).

It must satisfy:

• •Nonnegativity: p_{X,Y}(x,y) ≥ 0
• •Normalization:

∑_x ∑_y p_{X,Y}(x,y) = 1

Think of p_{X,Y} as a table: rows are x values, columns are y values, and each cell is the probability of that pair.

Definition (continuous)

If X and Y are continuous, we use a joint pdf f_{X,Y}(x,y) such that probability is obtained by integrating over regions:

P((X,Y) ∈ A) = ∬_A f_{X,Y}(x,y) dx dy.

Requirements:

• •f_{X,Y}(x,y) ≥ 0
• •∬_{ℝ²} f_{X,Y}(x,y) dx dy = 1

Crucial subtlety: For continuous variables, f_{X,Y}(x,y) is a density, not a probability. So P(X = x, Y = y) = 0 for exact points, even though f_{X,Y}(x,y) can be positive.

Support (where the mass/density lives)

A joint distribution often only assigns nonzero probability/density on a particular set:

• •Discrete: a set of pairs (x,y)
• •Continuous: a region in the plane

For example, if X is uniform on [0,1] and Y = 1 − X, then (X,Y) lies only on the line y = 1 − x. This is a joint distribution, but it’s not a regular 2D pdf (it’s “singular” on a line). In this lesson we focus on standard pmfs and pdfs where the formulas below apply cleanly.

Joint CDF as an alternative

Another way to specify the joint distribution is the joint CDF:

F_{X,Y}(x,y) = P(X ≤ x, Y ≤ y).

For continuous distributions, you can (when differentiable) recover the joint pdf via partial derivatives:

f_{X,Y}(x,y) = ∂²/∂x∂y F_{X,Y}(x,y).

This connects joint distributions to multivariable calculus: joint CDFs are like “accumulated volume,” joint pdfs are like “density per unit area.”

A mental model

• •Joint pmf/pdf: “How likely is each pair (x,y)?”
• •Marginal: “What do I believe about X alone?”
• •Conditional: “Given I learned Y = y, what do I believe about X now?”

Those last two are the real workhorses—and they both come directly from the joint.

Why marginals matter

Even if you build a full model of (X,Y), you frequently need statements about just one variable:

• •“What is the distribution of demand X regardless of price Y?”
• •“What’s the probability Y exceeds a threshold, regardless of X?”

Marginals let you ignore variables by aggregating over all their possible values.

Discrete marginals

Starting from p_{X,Y}(x,y), the marginal pmfs are

p_X(x) = ∑_y p_{X,Y}(x,y)

p_Y(y) = ∑_x p_{X,Y}(x,y).

Interpretation: Fix x and add up the probabilities of all pairs (x,y) across all y.

This is literally the “row sum” (or “column sum”) of the joint table.

Continuous marginals

From a joint pdf f_{X,Y}(x,y), the marginal pdfs are

f_X(x) = ∫_{−∞}^{∞} f_{X,Y}(x,y) dy

f_Y(y) = ∫_{−∞}^{∞} f_{X,Y}(x,y) dx.

Interpretation: Fix x and integrate the density along the vertical line at that x.

Regions and supports

In practice, the integration limits often come from the support.

Example: Suppose f_{X,Y}(x,y) = c on the triangle 0 ≤ x ≤ 1, 0 ≤ y ≤ 1 − x (and 0 otherwise). Then

• •For a given x ∈ [0,1], y ranges from 0 to 1 − x.

So

f_X(x) = ∫_0^{1−x} c dy = c(1 − x), 0 ≤ x ≤ 1.

The lesson: marginalization is conceptually simple, but you must be careful about the geometry of where the joint density is nonzero.

A table viewpoint (discrete)

If you have a joint pmf table, marginals are easy bookkeeping.

The bottom-right cell becomes 1 because the whole table sums to 1.

From marginals to probabilities

Once you have marginals, you can compute 1D probabilities:

• •Discrete: P(X ∈ S) = ∑_{x∈S} p_X(x)
• •Continuous: P(a ≤ X ≤ b) = ∫_a^b f_X(x) dx

Marginalization generalizes

For more variables, the idea is identical.

If you have p_{X,Y,Z}(x,y,z), then

p_{X,Y}(x,y) = ∑_z p_{X,Y,Z}(x,y,z)

and similarly for continuous with integrals.

This “sum/integrate out what you don’t need” pattern will show up everywhere later (Bayes nets, latent-variable models, expectation-maximization, variational inference).

Why conditionals matter

Information changes beliefs. If you observe Y = y, your uncertainty about X should typically shrink or shift.

Conditional distributions formalize this update.

Discrete conditional pmf

If p_Y(y) > 0, define

p_{X|Y}(x|y) = P(X = x | Y = y) = p_{X,Y}(x,y) / p_Y(y).

This is the key identity:

• •The joint splits into “marginal × conditional”:

p_{X,Y}(x,y) = p_{X|Y}(x|y) p_Y(y).

Continuous conditional density

For continuous variables, we use conditional densities (informally, “density of X given Y = y”):

f_{X|Y}(x|y) = f_{X,Y}(x,y) / f_Y(y), when f_Y(y) > 0.

And similarly

f_{X,Y}(x,y) = f_{X|Y}(x|y) f_Y(y).

“Normalize a slice” intuition

Fix y.

• •The function x ↦ f_{X,Y}(x,y) is a slice of the joint density.
• •Its total mass in x is f_Y(y):

f_Y(y) = ∫ f_{X,Y}(x,y) dx.

So dividing by f_Y(y) forces the slice to integrate to 1:

∫ f_{X|Y}(x|y) dx

= ∫ f_{X,Y}(x,y)/f_Y(y) dx

= (1/f_Y(y)) ∫ f_{X,Y}(x,y) dx

= (1/f_Y(y)) f_Y(y)

= 1.

That’s what a conditional density is: the renormalized slice.

Bayes’ rule emerges immediately

Once you accept

p_{X,Y}(x,y) = p_{X|Y}(x|y)p_Y(y) = p_{Y|X}(y|x)p_X(x),

you can equate them to get Bayes’ rule:

p_{X|Y}(x|y) = p_{Y|X}(y|x) p_X(x) / p_Y(y).

And the denominator is just a marginalization:

p_Y(y) = ∑_x p_{Y|X}(y|x)p_X(x) (discrete)

f_Y(y) = ∫ f_{Y|X}(y|x) f_X(x) dx (continuous).

So “Bayes” is not a separate topic from joint distributions—it’s a rearrangement of the joint.

Independence as a special structure

X and Y are independent if learning Y tells you nothing about X:

p_{X|Y}(x|y) = p_X(x) for all x,y with p_Y(y) > 0.

Equivalent factorizations:

p_{X,Y}(x,y) = p_X(x)p_Y(y)

f_{X,Y}(x,y) = f_X(x)f_Y(y).

Independence is strong. Many variables are not independent, but may still be uncorrelated or weakly dependent. Later nodes (covariance/correlation, mutual information) quantify dependence in different ways.

Chain rule for more variables

For three variables (X,Y,Z), the joint can be factored as

p_{X,Y,Z}(x,y,z)

= p_{X|Y,Z}(x|y,z) p_{Y|Z}(y|z) p_Z(z).

More generally, for X₁,…,X_n:

p(x₁,…,x_n) = ∏_{i=1}^n p(x_i | x₁,…,x_{i−1}).

This is not “assumptions”—it’s always true when the conditionals exist. Assumptions enter when you simplify the conditionals (e.g., Markov properties).

Computing probabilities of events

With a joint distribution, probabilities of events about (X,Y) become sums/integrals over regions.

Discrete:

P((X,Y) ∈ A) = ∑_{(x,y)∈A} p_{X,Y}(x,y).

Continuous:

P((X,Y) ∈ A) = ∬_A f_{X,Y}(x,y) dx dy.

This is often the cleanest way to solve problems like P(X + Y ≤ 1) or P(X ≤ Y).

Expectation with a joint

Many quantities of interest are functions g(X,Y). The expected value is

E[g(X,Y)] = ∑_x ∑_y g(x,y) p_{X,Y}(x,y) (discrete)

E[g(X,Y)] = ∬ g(x,y) f_{X,Y}(x,y) dx dy (continuous).

Special cases:

• •E[X] = ∬ x f_{X,Y}(x,y) dx dy
• •E[XY] = ∬ xy f_{X,Y}(x,y) dx dy

A key identity (law of total expectation) links conditionals back to marginals:

E[X] = E[E[X|Y]].

You can see it directly in the discrete case:

E[E[X|Y]]

= ∑_y E[X|Y=y] p_Y(y)

= ∑_y (∑_x x p_{X|Y}(x|y)) p_Y(y)

= ∑_y ∑_x x (p_{X,Y}(x,y)/p_Y(y)) p_Y(y)

= ∑_x ∑_y x p_{X,Y}(x,y)

= E[X].

This is one of the most useful “bridge” theorems in probability.

Dependence: beyond independence

Independence is all-or-nothing. In practice you often ask: “How dependent are X and Y?”

Two major upcoming tools depend on joints:

Joint distributions are the raw material those measures are made from.

A geometric view for continuous joints

For a continuous f_{X,Y}:

• •High density regions in the (x,y)-plane correspond to likely pairs.
• •Contours (level sets) show the “shape” of dependence.
• •If contours are circular (as in an isotropic Gaussian), dependence is weak.
• •If contours are elongated along a diagonal, X and Y move together.

Even before computing correlation, you can often “see” dependence from the joint density.

Practical modeling patterns

1) Start with a generative story (conditionals):

• •Choose Y ~ p_Y
• •Then choose X | Y=y ~ p_{X|Y}(

· | y)

This defines a valid joint via p_{X,Y}(x,y) = p_{X|Y}(x|y)p_Y(y).

2) Or start with a joint and derive everything else:

• •Compute p_Y via marginalization
• •Compute p_{X|Y} via division

Both are common. In machine learning, you often specify p(y) and p(x|y) (mixture models, Naive Bayes), and marginalize y to get p(x).

A note on vectors and multivariate distributions

When you have many variables, you often package them into a random vector X ∈ ℝᵈ.

• •The “joint distribution of d variables” is exactly the distribution of X.
• •A marginal corresponds to selecting some coordinates of X.

You’ll later see multivariate normals, covariance matrices, and transformations of random vectors. Joint distributions are the foundation.