Lagrange Multipliers

The problem it solves (why)

Unconstrained optimization asks:

• •Minimize or maximize f(x) over all x in ℝⁿ.

But many optimization problems have equality constraints:

• •Minimize f(x) subject to g(x) = c.

The constraint restricts you to a feasible set—typically a curve (in 2D), a surface (in 3D), or a manifold/hypersurface (in higher dimensions). On that set, the “best” point is not usually where ∇f(x) = 0, because you’re not allowed to move in all directions.

So we need a different optimality condition that respects the constraint.

The key idea (intuition)

Imagine you’re walking along the constraint curve g(x) = c. At a best feasible point, you can’t move along the curve to improve f.

That means:

• •Every feasible direction (tangent to the constraint set) must have zero first-order change in f.

In gradient language:

• •The gradient ∇f points in the direction of steepest increase.
• •If you project ∇f onto the tangent directions and there’s any component left, you could move along the constraint to increase f.

So at a constrained extremum, ∇f must have no component in any tangent direction.

What vectors are orthogonal to every tangent direction? The normal vector to the constraint surface.

And what gives a normal vector to the level set g(x) = c? The gradient ∇g.

That yields the geometric condition:

• •At a constrained extremum: ∇f(x*) is parallel to ∇g(x*).

Parallel means “same direction up to scaling,” so:

• •∇f(x*) = λ ∇g(x*) for some scalar λ.

λ is the Lagrange multiplier.

The canonical statement

For one equality constraint:

Optimize f(x) subject to g(x) = c.

If x* is a solution (under regularity conditions like ∇g(x*) ≠ 0), then there exists λ such that:

• •∇f(x*) = λ ∇g(x*)
• •g(x*) = c

This converts the constrained problem into a system of equations.

What λ “means”

λ is not just a trick variable. It often measures sensitivity: how much the optimal value changes if you relax the constraint slightly.

Define the optimal value function:

• •v(c) = optimum of f(x) subject to g(x) = c

Under suitable conditions, one can show:

• •dv/dc = λ

Interpretation: λ is the “shadow price” of tightening/loosening the constraint.

When to use (and when not)

Lagrange multipliers are designed for:

• •Equality constraints (g(x) = c)
• •Smooth (differentiable) functions

If you have inequalities (g(x) ≤ c), you need the generalized framework: KKT Conditions.

A mental picture to keep

• •g(x) = c is a level set.
• •∇g is perpendicular to that level set.
• •At the best point on the level set, ∇f must also be perpendicular to it.
• •Therefore ∇f ∥ ∇g.

That single picture explains the method—and helps you remember why it works.

Why this condition must hold

Suppose you’re constrained to g(x) = c, and you’re at some feasible point x.

A feasible direction d is one that keeps you on the constraint to first order. Using a first-order approximation:

• •g(x + εd) ≈ g(x) + ε ∇g(x)·d

To stay feasible (keep g = c) for small ε, we need:

• •∇g(x)·d = 0

So feasible directions d are exactly the vectors orthogonal to ∇g(x). These directions form the tangent space to the constraint surface.

Now look at how the objective changes:

• •f(x + εd) ≈ f(x) + ε ∇f(x)·d

At a constrained local optimum, we can’t improve f by moving in any feasible direction. That means:

• •∇f(x)·d = 0 for all d such that ∇g(x)·d = 0

In linear algebra terms:

• •∇f is orthogonal to the entire tangent space.

The orthogonal complement of the tangent space is the span of the normal vector ∇g. Therefore:

• •∇f(x*) ∈ span{∇g(x*)}

So there exists λ such that:

• •∇f(x*) = λ ∇g(x*)

This is the “gradient parallelism” condition.

What can go wrong (regularity)

If ∇g(x*) = 0, the constraint surface may not have a well-defined normal at x* (think cusp or degenerate point). Then the simple condition may fail or be incomplete.

A standard regularity assumption for one constraint is:

• •∇g(x*) ≠ 0

For multiple constraints, you need gradients of constraints to be linearly independent (more later).

Constrained extrema vs. unconstrained stationary points

Unconstrained optimum: ∇f(x*) = 0.

Constrained optimum: ∇f(x*) doesn’t need to be 0. It only needs to have no component tangent to the feasible set.

Geometrically:

• •Unconstrained optimum: level sets of f “close around” x*.
• •Constrained optimum: the level set of f is tangent to the constraint set at x*.

Turning geometry into equations

If x ∈ ℝ² with constraint g(x, y) = c, the method yields:

• •∂f/∂x = λ ∂g/∂x
• •∂f/∂y = λ ∂g/∂y
• •g(x, y) = c

That’s 3 equations in 3 unknowns (x, y, λ).

In ℝⁿ, it becomes n + 1 equations in n + 1 unknowns.

Multiple equality constraints

If you have m constraints:

• •g₁(x) = c₁, …, gₘ(x) = cₘ

Then the feasible set is the intersection of m level sets. The tangent space is orthogonal to each ∇gᵢ, so the normal space is spanned by them.

The condition generalizes to:

• •∇f(x*) = ∑ᵢ₌₁ᵐ λᵢ ∇gᵢ(x*)
• •gᵢ(x*) = cᵢ for all i

This is the same “gradient lives in the span of constraint gradients” idea.

Quick comparison: one constraint vs. many

What this gives you (and what it doesn’t)

Solving the Lagrange equations gives candidates. You still need to:

• •Check feasibility
• •Compare objective values (global optimum) or use second-order tests (local classification)
• •Consider boundary/degenerate cases if assumptions fail

In constrained problems, it’s common to find:

• •Multiple candidate points
• •Some maxima, some minima, some saddle-like points restricted to the constraint

So treat the gradient parallelism equations as a principled way to generate a shortlist of candidates.

Why introduce the Lagrangian (motivation)

Gradient parallelism is a geometric statement, but algebraically it’s nicer to express everything as “take derivatives and set them to zero.”

The Lagrangian packages the objective and constraint into one scalar function whose stationary points encode the Lagrange conditions.

For one constraint g(x) = c, define:

• •ℒ(x, λ) = f(x) − λ(g(x) − c)

(You’ll also see ℒ = f + λ(g − c). The sign convention doesn’t matter as long as you’re consistent; λ will flip sign.)

How the stationary conditions reproduce the method

Take partial derivatives:

• •∇xℒ(x, λ) = ∇f(x) − λ ∇g(x)
• •∂ℒ/∂λ = −(g(x) − c)

Set them to zero:

1) ∇xℒ = 0 ⇒ ∇f(x) − λ ∇g(x) = 0 ⇒ ∇f(x) = λ ∇g(x)

2) ∂ℒ/∂λ = 0 ⇒ g(x) − c = 0 ⇒ g(x) = c

So stationary points of ℒ correspond exactly to the Lagrange multiplier equations.

Why λ is “free” to enforce the constraint

Think of λ as adjusting the penalty for violating the constraint. At the solution, λ tunes the trade-off so that a stationary point occurs exactly on the feasible set.

This is different from naive penalty methods like minimizing f(x) + ρ(g(x) − c)² for a large ρ:

• •Penalty methods approximate constraints.
• •Lagrange multipliers enforce the constraint exactly (in the mathematical conditions).

Multiple constraints in Lagrangian form

For constraints gᵢ(x) = cᵢ:

• •ℒ(x, λ) = f(x) − ∑ᵢ₌₁ᵐ λᵢ (gᵢ(x) − cᵢ)

Stationarity gives:

• •∇f(x) − ∑ᵢ λᵢ ∇gᵢ(x) = 0
• •gᵢ(x) = cᵢ for all i

Second-order thinking (local min/max classification)

Solving first-order conditions gives candidates. To classify them, you need to understand curvature along feasible directions.

A practical approach:

• •Parameterize the constraint locally and check second derivative along that parameter, or
• •Use second-order constrained conditions (bordered Hessian), or
• •In many problems, simply compare objective values across candidates if the feasible set is simple/compact.

At difficulty 4, it’s worth knowing a bit of structure without turning this into a full course:

Tangent-space second-order idea

Let d be a feasible direction at x*, i.e., ∇g(x*)·d = 0.

For a constrained local minimum, you expect:

• •dᵀ ∇²f(x*) d ≥ 0 for all feasible d (after accounting for curvature of the constraint).

The clean way to incorporate constraint curvature uses the Hessian of the Lagrangian:

• •∇²xℒ(x, λ) = ∇²f(x) − λ ∇²g(x*)

Then you examine dᵀ ∇²xℒ d over tangent directions d.

You don’t always need this in practice, but it’s the right object to look at: once you’ve “encoded” the constraint in ℒ, the Hessian of ℒ governs local behavior on the manifold.

Relationship to constrained optimization algorithms

Even if you don’t solve analytically, the Lagrangian viewpoint drives algorithms:

• •Method of multipliers / augmented Lagrangian
• •Sequential quadratic programming (SQP)
• •Dual ascent methods

These will reappear when you study Lagrangian Duality and KKT Conditions.

Summary of the workflow

1) Write constraint(s) in standard form g(x) = c.

2) Form ℒ(x, λ) = f(x) − ∑ λᵢ(gᵢ(x) − cᵢ).

3) Solve:

• •∇xℒ = 0
• •∂ℒ/∂λᵢ = 0

4) Evaluate candidates and pick the correct optimum type.

The method is compact, systematic, and scales naturally to multiple constraints—exactly why it’s a cornerstone for later topics like KKT and duality.

Why Lagrange multipliers matter in machine learning

Many ML training problems are constrained, either explicitly or implicitly:

• •Margin constraints in support vector machines
• •Normalization constraints (e.g., probabilities sum to 1)
• •Conservation constraints in probabilistic models
• •Equality constraints introduced to split variables in optimization (ADMM-style ideas)

Even when the original problem is unconstrained, constraints appear when you derive dual problems or enforce structure.

Support Vector Machines (where you’ll see it next)

In a (hard-margin) linear SVM, you choose a separating hyperplane (w, b) that maximizes margin subject to classification constraints.

A common equivalent formulation:

Minimize ½‖w‖² subject to

• •yᵢ(w·xᵢ + b) ≥ 1 for all i

Those are inequality constraints, so the full story uses KKT. But the core pattern is identical:

• •Build a Lagrangian with multipliers
• •Stationarity links primal variables (w, b) to multipliers
• •The dual problem emerges by eliminating primal variables

Even before inequalities, the equality-constraint case teaches the essential muscle memory: “Introduce λ, take derivatives, solve stationarity + constraints.”

KKT conditions: equality constraints are the base case

KKT conditions generalize Lagrange multipliers to include inequalities g(x) ≤ 0 via multipliers αᵢ ≥ 0 and complementary slackness:

• •αᵢ gᵢ(x) = 0

If you understand equality constraints well, KKT feels like a small extension rather than a new world.

Lagrangian duality: λ becomes an optimization variable

In equality constraints, we usually solve for λ as part of the stationarity system.

In duality, you instead treat λ as defining a lower bound (for minimization) and optimize that bound:

• •Dual function: q(λ) = inf_x ℒ(x, λ)
• •Dual problem: maximize q(λ) over λ

The Lagrangian is the bridge between primal and dual.

Sensitivity / shadow price interpretation (practical intuition)

Consider a parameterized constraint g(x) = c. If λ* is large in magnitude at the solution, the optimum is very sensitive to changes in c.

A simple mental model:

• •If tightening the constraint slightly makes the objective much worse, |λ| is large.
• •If the constraint doesn’t really “bind” the optimum (or you can adjust with little cost), |λ| is small.

In economics and resource allocation, λ is literally the “value” of an additional unit of resource.

A concrete ML-adjacent example: normalization constraint

Suppose you optimize over probability vectors p ∈ ℝⁿ with constraint ∑ᵢ pᵢ = 1.

That’s an equality constraint. The gradient-parallelism idea says:

• •∇f(p) = λ ∇(∑ᵢ pᵢ)

But ∇(∑ᵢ pᵢ) = 1 (the all-ones vector). So the condition becomes:

• •∂f/∂pᵢ = λ for all i

Meaning: at optimum (with only that constraint), all partial derivatives must be equal. This is a powerful structural fact you can reuse.

(If you also require pᵢ ≥ 0, that’s inequality territory → KKT.)

How to recognize a “Lagrange multipliers problem” quickly

You’re likely in this regime if:

• •The constraint is an equation like x² + y² = 1, or ∑ xᵢ = 1.
• •The constraint defines a smooth curve/surface.
• •The objective is differentiable.
• •You want stationary candidates on that surface.

One more conceptual connection: projection viewpoint

Another way to interpret Lagrange multipliers:

• •At optimum, the gradient of f projected onto the tangent space is 0.

Equivalently:

• •∇f lies entirely in the normal space, i.e., span{∇gᵢ}.

This “projection to tangent space” language shows up in manifold optimization and constrained gradient methods.

Where you go next

• •For inequality constraints and complementarity: KKT Conditions
• •For turning constraints into a dual maximization: Lagrangian Duality
• •For a flagship ML use case: Support Vector Machines

Lagrange multipliers are the conceptual seed for all three.