Ensemble Methods

Why ensembles exist (motivation)

Many ML models have a frustrating property: small changes in the training data can cause large changes in the learned model. Decision trees are a classic example—if you perturb the dataset slightly, the splits can change, leading to very different trees.

Ensemble methods are a systematic way to convert:

• •instability → stability
• •high variance → lower variance (often)
• •weak learners → strong learner (often)

The core idea is surprisingly simple: instead of trusting one model, train multiple base learners and aggregate their predictions.

Definition

An ensemble method constructs a predictor:

• •Base learners: h₁(x), h₂(x), …, h_M(x)
• •Ensemble: H(x)

and defines H(x) as an aggregation of the hᵢ(x).

Typical forms:

• •Regression (averaging):
• •H(x) = (1/M) ∑ᵢ hᵢ(x)
• •Classification (majority vote):
• •H(x) = mode{h₁(x), …, h_M(x)}
• •Weighted versions:
• •H(x) = ∑ᵢ αᵢ hᵢ(x)

Intuition: “wisdom of the crowd” needs diversity

If every base learner makes the same mistakes, the ensemble gains nothing.

Ensembles work when base learners are:

1. 1)Individually better than random (even slightly)
2. 2)Diverse (errors not perfectly correlated)

A helpful mental model: if each model has some error “noise,” averaging cancels noise—but only if those noises aren’t identical.

A variance-reduction sketch (why averaging helps)

Suppose we do regression and each model predicts:

hᵢ(x) = f(x) + εᵢ

where f(x) is the true target function (or the expected prediction) and εᵢ is a zero-mean error term.

The ensemble average is:

H(x) = (1/M) ∑ᵢ hᵢ(x)

= (1/M) ∑ᵢ (f(x) + εᵢ)

= f(x) + (1/M) ∑ᵢ εᵢ

So the ensemble error is (1/M) ∑ᵢ εᵢ.

If the εᵢ are independent with Var(εᵢ) = σ², then:

Var(H(x) − f(x))

= Var((1/M) ∑ᵢ εᵢ)

= (1/M²) ∑ᵢ Var(εᵢ)

= (1/M²) · M · σ²

= σ² / M

That 1/M is the dream scenario.

If errors are correlated, you don’t get the full 1/M; this is why creating diversity is a central design goal.

Two big families

Ensemble methods mainly fall into two camps:

This lesson focuses on: aggregation, bagging, random forests, and boosting—what they optimize, why they work, and how to reason about their behavior using the bias–variance lens.

Why talk about aggregation separately?

Training many models is only half the story. The other half is: how do we combine them so the result is better than each one?

Aggregation defines how information flows from base learners hᵢ(x) to the final ensemble H(x). Different choices correspond to different assumptions about:

• •whether predictions are comparable,
• •whether models are equally trustworthy,
• •whether models specialize in different regions.

Common aggregation rules

1) Regression: simple averaging

If hᵢ(x) outputs a real number (e.g., house price), the default aggregation is:

H(x) = (1/M) ∑ᵢ hᵢ(x)

Why this makes sense:

• •It preserves the scale of the target.
• •It reduces variance when base learners’ errors aren’t perfectly correlated.

2) Classification: majority voting

If each hᵢ(x) outputs a class label in {0,1} (or {1,…,K}), a typical ensemble is:

H(x) = mode{h₁(x), …, h_M(x)}

Why this can help:

• •A single overfit tree may misclassify a point, but if most other trees classify it correctly, the vote corrects it.

3) Probability averaging (soft voting)

Often, classifiers output estimated probabilities pᵢ(y=k | x). Then you can average probabilities and choose argmax:

p̄(y=k | x) = (1/M) ∑ᵢ pᵢ(y=k | x)

H(x) = argmaxₖ p̄(y=k | x)

Soft voting tends to be more stable than hard voting because it uses confidence information.

4) Weighted combinations

Sometimes models shouldn’t be equal. Boosting is the archetype:

H(x) = sign(∑ᵢ αᵢ hᵢ(x)) (binary classification with hᵢ(x) ∈ {−1, +1})

or for regression:

H(x) = ∑ᵢ αᵢ hᵢ(x)

Here αᵢ is learned from performance (later models may get different weights).

Diversity: the hidden requirement

Aggregation works best when base learners disagree in useful ways.

How do we create diversity?

• •Different training subsets (bagging)
• •Different feature subsets (random forests)
• •Different training emphasis (boosting reweights data)
• •Different model types (stacking / heterogeneous ensembles—beyond this node)

A key point: diversity that increases error is not helpful. We want “decorrelated” errors while keeping each hᵢ(x) reasonably accurate.

Bias–variance lens for aggregation

You already know the bias–variance tradeoff. Aggregation changes that balance:

• •Averaging many high-variance models tends to reduce variance.
• •Combining many weak models sequentially can reduce bias by expanding the function class.

A simplified but useful heuristic:

• •Bagging → variance ↓
• •Boosting → bias ↓ (often) and variance can ↑ if overfit

A small derivation: averaging with correlated errors

Assume M models, each with error εᵢ, Var(εᵢ)=σ², and pairwise Corr(εᵢ, εⱼ)=ρ for i≠j.

Let H be the average. Then:

Var(H − f)

= Var((1/M) ∑ᵢ εᵢ)

= (1/M²) Var(∑ᵢ εᵢ)

Now expand Var of a sum:

Var(∑ᵢ εᵢ)

= ∑ᵢ Var(εᵢ) + 2∑_{i<j} Cov(εᵢ, εⱼ)

= Mσ² + 2 · (M(M−1)/2) · (ρσ²)

= Mσ² + M(M−1)ρσ²

So:

Var(H − f)

= (1/M²)[Mσ² + M(M−1)ρσ²]

= (σ²/M) + ( (M−1)/M ) ρσ²

As M → ∞, this approaches ρσ².

Interpretation:

• •If ρ = 0 (independent errors), variance → 0 as M grows.
• •If ρ > 0, there is a floor: you can’t average below the shared correlated component.

This is why random forests deliberately reduce correlation (via feature subsampling), not just train many trees.

Why bagging works

Decision trees are typically:

• •low bias (they can fit complex patterns)
• •high variance (they change a lot with data perturbations)

Bagging targets that exact profile.

The plan:

1. 1)Create many slightly different datasets via bootstrap resampling.
2. 2)Train the same learner on each dataset (often an unpruned tree).
3. 3)Aggregate predictions (vote/average).

You get a “committee” of trees that individually overfit in different ways, and the vote cancels much of that overfitting.

Bootstrap resampling (the source of diversity)

Given training set D of size N, a bootstrap sample Dᵢ is created by sampling N points with replacement from D.

Key facts:

• •Some points appear multiple times.
• •Some points don’t appear at all.

A useful calculation: expected fraction of unique points in a bootstrap sample.

Probability a specific point is not selected in one draw: 1 − 1/N.

Not selected in N draws:

(1 − 1/N)ᴺ ≈ e^{−1} ≈ 0.368

So expected fraction left out is ≈ 36.8%, and included is ≈ 63.2%.

Those “left out” points are the out-of-bag (OOB) set for that tree.

Bagging algorithm (conceptual)

For m = 1..M:

• •D_m ← bootstrap(D)
• •train base model h_m on D_m

Return ensemble H(x) = average/vote of h_m(x)

Out-of-bag evaluation (a free validation signal)

Because each training point is excluded from about 36.8% of bootstrap samples, you can estimate generalization error without a separate validation set:

• •For each point (xⱼ, yⱼ), collect predictions from trees where xⱼ was OOB.
• •Aggregate those predictions and compare to yⱼ.

This yields an OOB error estimate that is often close to cross-validation for bagged trees.

Random forests: bagging + feature randomness

Bagging alone reduces variance, but trees can still be correlated because strong predictors dominate splits.

Random forests add a second diversity lever:

• •At each split, consider only a random subset of features.

If there are d features total, you choose m_try features per split:

• •Classification default: m_try ≈ √d
• •Regression default: m_try ≈ d/3

Why it helps:

• •Different trees choose different top-level splits.
• •This reduces correlation ρ between tree errors.
• •Lower ρ pushes ensemble variance down (see correlated variance formula earlier).

Random forest prediction

• •Regression: H(x) = (1/M) ∑ᵢ hᵢ(x)
• •Classification: majority vote or probability averaging

Interpreting random forests via bias–variance

• •Individual tree: low bias, high variance
• •Bagging/forest: slightly higher bias (sometimes) but much lower variance

Often the net result is better test performance.

Practical hyperparameters (what they do)

Feature importance (connection, but be cautious)

Random forests often report feature importance:

• •Mean decrease in impurity (fast, but biased toward high-cardinality features)
• •Permutation importance (more reliable, but more expensive)

This is a great tool, but not a proof of causality.

When bagging/forests shine

• •Tabular data with non-linear interactions
• •Mixed feature types
• •Minimal preprocessing
• •Need strong baseline quickly

When they struggle

• •Extrapolation outside training range (trees are piecewise constant)
• •Very high-dimensional sparse linear problems (linear models may dominate)
• •Tasks requiring smooth function approximation unless many trees

Why boosting exists (a different failure mode)

Bagging is great when your base learner is already powerful but unstable.

What if your base learner is too weak and underfits?

• •High bias
• •Systematic errors

Boosting attacks bias by building an additive model step-by-step, where each new learner focuses on what previous learners got wrong.

The central idea: additive modeling

Boosting constructs:

H_M(x) = ∑_{m=1..M} α_m h_m(x)

The sequence matters: hₘ is chosen based on the errors of H_{m−1}.

Two big boosting views:

1. 1)Reweighting view (AdaBoost-style): focus more on misclassified points.
2. 2)Gradient view (Gradient Boosting): each learner fits the negative gradient (residual-like signal) of a loss function.

We’ll sketch both, because they illuminate the “why.”

AdaBoost intuition (binary classification)

Assume labels y ∈ {−1, +1}, and weak learners output h(x) ∈ {−1, +1}.

AdaBoost maintains a weight distribution over training points:

• •Start with uniform weights wᵢ = 1/N.
• •Train a weak learner h₁ that minimizes weighted classification error.
• •Increase weights on misclassified points so the next learner pays more attention.

At step m, weighted error is:

err_m = ∑ᵢ wᵢ [h_m(xᵢ) ≠ yᵢ]

Then compute learner weight:

α_m = (1/2) ln((1 − err_m)/err_m)

Update data weights:

wᵢ ← wᵢ · exp(−α_m yᵢ h_m(xᵢ))

Check what this does:

• •If yᵢ = h_m(xᵢ), then yᵢ h_m(xᵢ)=+1 ⇒ multiply by exp(−α_m) (downweight)
• •If yᵢ ≠ h_m(xᵢ), then yᵢ h_m(xᵢ)=−1 ⇒ multiply by exp(+α_m) (upweight)

Finally normalize w so ∑ᵢ wᵢ = 1.

The final classifier:

H(x) = sign(∑ₘ α_m h_m(x))

Why it reduces bias: each stage targets the “hard” points not yet handled.

Risk: if you keep adding learners, you can start fitting noise (variance can rise), especially with very complex base learners.

Gradient Boosting intuition (general loss)

Gradient boosting generalizes boosting to arbitrary differentiable losses.

We want to minimize empirical risk:

L(H) = ∑ᵢ ℓ(yᵢ, H(xᵢ))

We build H as an additive model:

H_m(x) = H_{m−1}(x) + η · f_m(x)

where f_m is a new base learner (often a small tree), and η ∈ (0,1] is a learning rate.

Why the “gradient” appears

We want to choose f_m that most reduces L. In function space, the steepest descent direction is the negative gradient.

Define pseudo-residuals:

rᵢ^{(m)} = − ∂ℓ(yᵢ, H(xᵢ)) / ∂H(xᵢ) evaluated at H = H_{m−1}

Then fit the next learner to these residuals:

• •f_m(x) ≈ r^{(m)}

So gradient boosting is like:

• •“Compute what direction would reduce loss at each point.”
• •“Fit a model to that direction.”
• •“Take a small step.”

Example: squared error regression

Let ℓ(y, H) = (1/2)(y − H)².

Compute:

∂ℓ/∂H = ∂(1/2)(y − H)² / ∂H

= (1/2) · 2(y − H) · (−1)

= −(y − H)

So the negative gradient is:

r = −(∂ℓ/∂H) = y − H

That’s exactly the residual.

Thus gradient boosting with squared loss repeatedly fits residuals.

Why small trees (stumps) are common

Boosting often uses weak learners such as:

• •depth-1 stumps
• •depth-2 or depth-3 trees

They have high bias individually, but the additive combination builds complexity gradually. This helps control overfitting and makes training more stable.

Regularization knobs in boosting

Bias–variance view

Boosting is often described as:

• •bias ↓ because the additive model expands expressiveness
• •variance can either stay manageable (with regularization) or explode (if overly aggressive)

This is why boosting tends to be powerful but sensitive to tuning.

Start from the problem: what’s failing?

Use the bias–variance diagnosis you already know.

• •If your model overfits wildly (high variance): prefer bagging / random forests.
• •If your model underfits (high bias): prefer boosting.

This is not absolute, but it’s a strong default.

A decision table

Interpreting H(x) in practice

Even though an ensemble is “many models,” you still ask:

• •What is H(x) optimizing?
• •Is H(x) stable under resampling?
• •Are base learners too correlated?

For random forests, stability increases with M; for boosting, stability depends on regularization.

Computational tradeoffs

Boosting’s sequential nature is the price you pay for its targeted bias reduction.

Connection to “model averaging” beyond trees

Ensembles are not limited to trees.

• •You can bag neural nets.
• •You can average multiple logistic regressions trained on different bootstrap samples.

However, trees are a perfect match because:

• •they’re unstable (bagging helps a lot)
• •they can be weak learners (boosting works well)

A note on stacking (beyond this node)

Another ensemble family is stacking, where a meta-model learns how to combine base learners. It’s powerful, but adds complexity and risks leakage. For this node, focus on the foundational trio: bagging, random forests, boosting.

Practical checklist

Before choosing/tuning ensembles:

1. 1)Ensure train/test split is correct (time series needs time-aware split).
2. 2)Confirm metric (accuracy vs AUC vs RMSE) matches the goal.
3. 3)For boosting: start with small depth, small η, then increase M.
4. 4)For forests: increase M until performance plateaus; tune m_try and leaf size.

Where this node leads

Understanding ensembles unlocks:

• •feature importance and model interpretation techniques
• •modern gradient boosting systems (regularization, shrinkage, column subsampling)
• •uncertainty estimation via ensembles
• •robust baselines for tabular ML pipelines