Logistic Regression

The problem it solves

In binary classification, each example has features x ∈ ℝᵈ and a label y ∈ {0, 1}. You want a model that, given x, outputs a probability that the label is 1:

• •Output near 1 ⇒ “very likely positive”
• •Output near 0 ⇒ “very likely negative”

Many models can produce a hard decision (0/1), but logistic regression is designed to produce a calibrated probability in [0, 1].

Why we don’t just use linear regression

A linear model like w·x + b can be any real number: negative, > 1, etc. That’s not a valid probability.

We need two ingredients:

1. 1)A linear predictor (score):

z = w·x + b

This is the raw “evidence” for the positive class.

2. 2)A squashing function that maps ℝ → [0, 1].

Logistic regression chooses the sigmoid (logistic) function:

σ(z) = 1 / (1 + e^(−z))

So the model is:

ŷ = P(y = 1 | x) = σ(w·x + b)

Intuition: score → probability

The score z measures where x lies relative to a hyperplane.

• •Decision boundary is where z = 0

w·x + b = 0

• •If z is large and positive, σ(z) ≈ 1
• •If z is large and negative, σ(z) ≈ 0

So logistic regression is a linear classifier in geometry, but a probabilistic model in output.

Odds and log-odds (why sigmoid is a natural choice)

A key reason logistic regression is so standard is that it models the log-odds as linear.

Define odds:

odds = P(y=1|x) / P(y=0|x) = p / (1 − p)

Log-odds (logit):

logit(p) = log(p / (1 − p))

Logistic regression assumes:

log(p / (1 − p)) = w·x + b

Solve for p:

Let z = w·x + b.

p / (1 − p) = e^z

p = e^z (1 − p)

p = e^z − e^z p

p + e^z p = e^z

p(1 + e^z) = e^z

p = e^z / (1 + e^z) = 1 / (1 + e^(−z)) = σ(z)

This shows sigmoid isn’t arbitrary: it’s what you get when you say “log-odds are linear in features.”

Notation note: include bias in w

Sometimes we fold the bias into the weight vector by adding a constant feature x₀ = 1.

Define extended feature vector x̃ = [1, x₁, …, x_d] and w̃ = [b, w₁, …, w_d]. Then:

z = w̃·x̃

This can simplify implementations and derivations.

Summary

Logistic regression is:

• •A linear scoring function z = w·x + b
• •A probabilistic output ŷ = σ(z)
• •A training objective that matches Bernoulli labels via maximum likelihood

Why start with a linear predictor?

The linear predictor is the simplest way to combine features:

z = w·x + b = ∑ⱼ wⱼ xⱼ + b

Motivation:

• •It’s interpretable: each feature contributes additively.
• •It’s scalable: works well in high dimensions.
• •It’s a strong baseline: many problems are close to linearly separable after good feature engineering.

Geometry: a hyperplane decision boundary

The set of points where the model is indifferent (predicts 0.5) is where ŷ = 0.5.

Since σ(0) = 0.5, we have:

ŷ = 0.5 ⇔ z = 0 ⇔ w·x + b = 0

That equation describes a hyperplane.

• •w is perpendicular (normal) to the hyperplane.
• •b shifts the hyperplane.

Predicted class often uses a threshold:

predict 1 if ŷ ≥ 0.5 (equivalently z ≥ 0)

Why the sigmoid specifically?

We want a function with:

• •Output in (0, 1)
• •Smooth, differentiable (for gradient descent)
• •Monotonic increasing (higher score ⇒ higher probability)

Sigmoid has these properties.

Key values:

• •σ(0) = 1/2
• •σ(z) → 1 as z → +∞
• •σ(z) → 0 as z → −∞

Sensitivity: sigmoid derivative

Training needs gradients. Sigmoid has a famously convenient derivative.

Let p = σ(z) = 1 / (1 + e^(−z)).

Differentiate:

p = (1 + e^(−z))^(−1)

∂p/∂z = −1 · (1 + e^(−z))^(−2) · ∂/∂z (1 + e^(−z))

∂/∂z (1 + e^(−z)) = −e^(−z)

So:

∂p/∂z = (1 + e^(−z))^(−2) · e^(−z)

Now rewrite in terms of p:

p = 1 / (1 + e^(−z))

1 − p = e^(−z) / (1 + e^(−z))

Therefore:

p(1 − p) = [1 / (1 + e^(−z))] · [e^(−z) / (1 + e^(−z))] = e^(−z) / (1 + e^(−z))²

Thus:

∂p/∂z = p(1 − p)

This compact form is one reason logistic regression is so convenient.

Interpretability: weight signs and magnitudes

Because log-odds are linear:

log(p/(1−p)) = w·x + b

Each weight wⱼ has a direct interpretation:

• •Increasing xⱼ by 1 increases log-odds by wⱼ (holding others fixed).
• •If wⱼ > 0, that feature pushes toward class 1.
• •If wⱼ < 0, it pushes toward class 0.

Caution: interpretation depends on feature scaling. If one feature is measured in large units, its weight will tend to be smaller.

A tiny comparison table

This sets up the next step: choosing a loss that matches Bernoulli labels.

Why we need a special loss

For classification, we don’t just want “close numeric values.” We want:

• •confident correct predictions to be rewarded
• •confident wrong predictions to be punished strongly
• •a probabilistic interpretation (so “0.9” means something)

Binary cross-entropy (BCE) comes directly from maximum likelihood estimation for a Bernoulli model.

Bernoulli model for labels

Assume for each input x, the label y is drawn as:

P(y = 1 | x) = p

P(y = 0 | x) = 1 − p

with p = σ(z) and z = w·x + b.

The Bernoulli probability mass function can be written compactly as:

P(y | x) = pʸ (1 − p)^(1−y)

because:

• •if y = 1 ⇒ p¹(1−p)⁰ = p
• •if y = 0 ⇒ p⁰(1−p)¹ = 1−p

Likelihood for a dataset

Given N i.i.d. examples {(xᵢ, yᵢ)}:

L(w, b) = ∏ᵢ pᵢ^(yᵢ) (1 − pᵢ)^(1−yᵢ)

where pᵢ = σ(w·xᵢ + b).

Maximizing a product is awkward, so take logs:

log L = ∑ᵢ [ yᵢ log pᵢ + (1 − yᵢ) log(1 − pᵢ) ]

Maximum likelihood is equivalent to minimizing negative log-likelihood:

J(w, b) = − log L = − ∑ᵢ [ yᵢ log pᵢ + (1 − yᵢ) log(1 − pᵢ) ]

Often we average over N:

J = (1/N) ∑ᵢ ℓᵢ

with per-example loss:

ℓ = −[ y log p + (1 − y) log(1 − p) ]

That is the binary cross-entropy loss.

Why BCE behaves the way we want

Consider one example.

• •If y = 1, the loss is ℓ = −log p
• •If p = 0.9 ⇒ ℓ ≈ 0.105
• •If p = 0.01 ⇒ ℓ ≈ 4.605 (large penalty)

• •If y = 0, the loss is ℓ = −log(1 − p)
• •If p = 0.1 ⇒ ℓ ≈ 0.105
• •If p = 0.99 ⇒ ℓ ≈ 4.605

So BCE strongly penalizes confident mistakes.

Gradient: the clean “prediction minus label” form

This is the workhorse result that makes training simple.

For one example:

p = σ(z), z = w·x + b

ℓ = −[ y log p + (1 − y) log(1 − p) ]

We compute ∂ℓ/∂z.

Step 1: derivative of ℓ with respect to p:

∂ℓ/∂p = −[ y · (1/p) + (1 − y) · (−1/(1 − p)) ]

∂ℓ/∂p = − y/p + (1 − y)/(1 − p)

Step 2: chain rule with ∂p/∂z = p(1 − p):

∂ℓ/∂z = (∂ℓ/∂p)(∂p/∂z)

∂ℓ/∂z = ( − y/p + (1 − y)/(1 − p) ) · p(1 − p)

Distribute p(1 − p):

∂ℓ/∂z = −y(1 − p) + (1 − y)p

∂ℓ/∂z = −y + yp + p − yp

∂ℓ/∂z = p − y

So the derivative w.r.t. the score is:

∂ℓ/∂z = (p − y) = (ŷ − y)

Now apply z = w·x + b:

∂z/∂w = x

∂z/∂b = 1

Thus:

∇_w ℓ = (ŷ − y)x

∂ℓ/∂b = (ŷ − y)

For the full dataset (averaged):

∇_w J = (1/N) ∑ᵢ (ŷᵢ − yᵢ)**xᵢ

∂J/∂b = (1/N) ∑ᵢ (ŷᵢ − yᵢ)

This is the key computational loop: predict p, compute error (p − y), accumulate gradients.

Convexity (a practical perk)

For standard logistic regression (no hidden layers), the BCE objective is convex in (w, b). That means:

• •there is a single global minimum
• •gradient descent won’t get trapped in bad local minima (though it can still be slow)

This is a major difference from neural networks, where the objective is non-convex.

Training with gradient descent (putting the pieces together)

A typical training step:

1. 1)Compute zᵢ = w·xᵢ + b
2. 2)Compute ŷᵢ = σ(zᵢ)
3. 3)Compute gradients:

• •g_w = (1/N) ∑ᵢ (ŷᵢ − yᵢ)**xᵢ
• •g_b = (1/N) ∑ᵢ (ŷᵢ − yᵢ)

4. 4)Update parameters (learning rate η):

w ← w − η g_w

b ← b − η g_b

Because you already know gradient descent, the main learning here is: BCE + sigmoid makes the gradient become “prediction minus label.”

Decision thresholds and costs

The model outputs a probability ŷ. Turning it into a label requires a threshold t.

• •Default: t = 0.5
• •Predict 1 if ŷ ≥ t

But in many real applications, false positives and false negatives have different costs.

Examples:

• •Medical screening: prefer fewer false negatives ⇒ lower threshold
• •Spam filtering: prefer fewer false positives ⇒ higher threshold

So logistic regression naturally supports probability-based decision-making.

Evaluation metrics (quick orientation)

Accuracy is not always enough, especially with class imbalance.

Common choices:

• •Precision, recall, F1
• •ROC curve and AUC
• •Log loss (the same BCE, on held-out data)

BCE is a natural metric because it evaluates probability quality, not just hard labels.

Regularization (brief but important)

To reduce overfitting, add a penalty on w.

L2 regularization (ridge):

J_reg = J + (λ/2)‖w‖²

Gradient adds:

∇_w J_reg = ∇_w J + λw

(Usually the bias b is not regularized.)

L1 regularization (lasso) encourages sparsity, but its gradient uses subgradients and optimization needs more care.

Numerical stability: logits and “BCE with logits”

Directly computing log(σ(z)) can cause issues when z is very large in magnitude.

In practice, libraries use a stable form often called binary cross-entropy with logits, where you pass z (the logit) directly.

This is a practical detail, but it matters for robust training.

Connection to neural networks

Logistic regression is a 1-layer neural network:

• •Input: x
• •Linear layer: z = w·x + b
• •Activation: σ(z)
• •Loss: BCE

When you later learn neural networks, you’ll generalize the linear layer to multiple layers and nonlinearities. The final layer for binary classification often remains a sigmoid (or a 2-class softmax), and the loss remains cross-entropy.

So mastering logistic regression means you already understand:

• •linear layers
• •activations
• •cross-entropy objectives
• •gradient-based training

You’re standing right at the entrance to Neural Networks.