Deep Learning

Why before how

Deep learning is not “just bigger neural nets.” It’s a strategy for representing complicated functions using many simple, reusable parts (layers), and for learning representations that make downstream prediction easy.

The core object is a composed function:

At each layer $\ell$, we maintain an activation / representation vector $\mathbf{h}^{\ell}$ (often written $h^\ell$ when shape is clear):

• •$\mathbf{h}^{0} = \mathbf{x}$ (the input)
• •$\mathbf{h}^{\ell} = f_{\ell}(\mathbf{h}^{\ell-1};\theta_{\ell})$
• •output $\hat{\mathbf{y}} = \mathbf{h}^{L}$

A very common concrete layer is affine + nonlinearity:

Depth matters because it changes what is easy to represent and what is easy to learn.

Minimal working mental model: a 2-layer network on a simple task

You already know backprop and SGD; let’s anchor deep learning in one concrete “small but real” example.

Task: binary classification in 2D. Input $\mathbf{x} \in \mathbb{R}^2$, label $y\in\{0,1\}$. Suppose the decision boundary is not linearly separable (e.g., two moons).

A 2-layer MLP (one hidden layer) is:

Interpretation:

• •Layer 1 creates a set of learned “features” (half-spaces gated by ReLU).
• •Layer 2 mixes those features into a probability via logistic regression.

Even here, you can see the deep learning pattern:

1) Representation: $\mathbf{h}^1$ is not hand-designed—it’s learned.

2) Composition: the model builds a nonlinear function from simple parts.

3) Trainability: success depends on gradients flowing from $\hat{y}$ back to $\mathbf{W}^1$.

Now scale that idea: more layers, richer inductive biases (convolution, attention), and careful conditioning (normalization/residuals) to make training stable.

Checkpoint: what “deep” adds

Before going further, keep these three questions in mind:

1) What family of functions does this architecture represent?

2) What representations will intermediate layers tend to discover?

3) Will gradients and signals propagate stably through depth?

Deep learning is largely the practice of answering those three questions well.

Why depth is not just “more parameters”

You could increase width (more units per layer) or increase depth (more layers). Both add parameters, but they add different representational structure.

A useful mental model:

• •Width adds “parallel feature templates.”
• •Depth adds “feature reuse and hierarchy.”

Depth encourages distributed hierarchical representations:

• •Early layers: local/simple patterns.
• •Middle layers: combinations of patterns.
• •Late layers: task-level abstractions.

In images, this often looks like edges → textures → parts → objects. In language, characters/subwords → local syntax → semantics.

The forward pass as representation building

Write the network as repeated transformations:

Think of $\mathbf{h}^{\ell}$ as a coordinate system the network invents. Learning aims to make the final layer’s problem “simple” (often linearly separable).

A very common pattern is:

where Norm might be BatchNorm, LayerNorm, RMSNorm, etc.

A little math: how composition shapes sensitivity

Deep nets are compositions, so their derivatives are products (chains) of Jacobians.

Let $\mathbf{h}^{\ell} \in \mathbb{R}^{d_\ell}$. Define the Jacobian

Then:

This single equation explains a lot:

• •If typical singular values of $\mathbf{J}^\ell$ are > 1, gradients can explode.
• •If they are < 1, gradients can vanish.
• •If they cluster near 1, training tends to be stable.

You don’t need to compute these Jacobians explicitly to benefit from this mental model; it motivates initialization, normalization, and residual connections.

Checkpoint: what you should carry forward

• •Representations $\mathbf{h}^\ell$ are the real “product” of deep learning.
• •The chain-of-Jacobians view predicts optimization pathologies.
• •Architecture is about shaping both representations and Jacobians.

Why inductive bias is the point of architecture

Without assumptions, learning in high dimensions is sample-inefficient (curse of dimensionality). Architectural choices encode assumptions like:

• •locality
• •translation equivariance
• •temporal recurrence
• •permutation invariance/equivariance
• •long-range interactions

These biases restrict the function class to something that matches the world.

A comparison table of major deep learning architectures

We’ll focus on CNNs and sequence models (RNNs/attention), since they are canonical deep learning building blocks.

CNNs: locality + weight sharing

A 2D convolution layer applies a kernel over local neighborhoods. If you already know the convolution operation, the key deep-learning additions are:

1) Channels: kernels map $C_{in}$ input channels to $C_{out}$ output channels.

2) Stacking: repeated convs grow the receptive field.

A simplified expression (single output channel) is:

With multiple channels:

Why it helps: weight sharing means you learn “edge detector” once and reuse it across the image. Locality reduces parameters and encourages features to be local.

Common CNN design motifs:

• •small kernels (3×3) stacked
• •pooling or strided conv for downsampling
• •residual blocks (ResNet)

RNNs: recurrence for sequences

An RNN maintains a state $\mathbf{h}_t$ updated over time:

This encodes an inductive bias: “the present depends on a compressed summary of the past.”

Training issue: backprop through time multiplies many Jacobians across timesteps, causing vanishing/exploding gradients. LSTMs/GRUs mitigate this with gating, roughly creating more stable paths for gradient flow.

Attention/Transformers: content-based routing

Self-attention computes a weighted average of value vectors using query-key similarity.

Given matrices $\mathbf{Q},\mathbf{K},\mathbf{V}$:

The inductive bias shifts from locality/recurrent state to learned interactions between all positions.

Transformers add:

• •positional information (positional encodings)
• •residual connections
• •normalization
• •MLP blocks

Checkpoint: choosing an architecture

A practical decision rule:

• •If your data has grid locality (images, spectrograms): start with CNNs.
• •If your data is sequential and you need streaming/low latency: consider RNNs.
• •If you need long-range dependencies and can afford batch processing: attention/Transformers.

Architecture is not just accuracy—it’s compute, latency, memory, and data efficiency.

Why this section exists

In shallow models, SGD “just works” surprisingly often. In deep models, optimization can fail even when the model is expressive enough.

The chain-of-Jacobians view tells you why: signals and gradients must propagate through many transformations. If their magnitudes drift, training becomes unstable.

We’ll build a stable mental model in three steps:

1) initialization tries to keep variance roughly constant across layers

2) normalization actively stabilizes distributions during training

3) residual connections provide easy paths for gradient flow

1) Initialization as variance control

Consider a layer:

Assume $h_i$ are i.i.d. with mean 0 and variance $\mathrm{Var}(h_i)=\sigma_h^2$. If weights have mean 0 and variance $\mathrm{Var}(W_{ij})=\sigma_w^2$, then (roughly):

where $n$ is fan-in.

To keep $\mathrm{Var}(z_j)$ from blowing up with depth, choose $\sigma_w^2 \propto 1/n$.

Two famous schemes:

• •Xavier/Glorot (good for tanh-like activations): $\sigma_w^2 \approx \frac{2}{\text{fan-in}+\text{fan-out}}$
• •He/Kaiming (good for ReLU): $\sigma_w^2 \approx \frac{2}{\text{fan-in}}$

These are not magic constants; they are attempts to keep forward activations and backward gradients in a reasonable range.

2) Normalization as conditioning control

Even with good initialization, distributions drift as parameters update. Normalization layers reduce internal covariate shift and improve conditioning.

BatchNorm (BN)

For a mini-batch, BN normalizes pre-activations per feature:

• •$\mu_B, \sigma_B^2$ computed over the batch
• •$\gamma,\beta$ are learned scale/shift

Pros: strong stabilizer, often speeds up CNN training.

Cons: batch-size dependence; tricky for RNNs/online/very small batches.

LayerNorm (LN)

LN normalizes across features within a single example:

Pros: works well in Transformers; independent of batch size.

RMSNorm

RMSNorm scales by root-mean-square without subtracting the mean:

Often used in modern LLM stacks for simplicity and stability.

3) Residual connections as gradient highways

A residual block computes:

Differentiate w.r.t. $\mathbf{h}^{\ell}$:

The identity term $\mathbf{I}$ ensures there is always a path with derivative near 1, which combats vanishing gradients.

This is a key reason very deep networks (ResNets, deep Transformers) are trainable.

Checkpoint: the stability toolkit

When a deep model won’t train, ask:

• •Are activations exploding/vanishing? (initialization, normalization)
• •Are gradients unstable? (residuals, normalization, learning rate)
• •Is the loss numerically unstable? (log-sum-exp tricks, mixed precision care)

These are not “details”—they are often the difference between success and failure.

Why practice looks different from theory

In theory, you can specify $f_\theta$ and run SGD. In practice, deep learning is an engineering loop:

1) pick an architecture with the right inductive bias

2) ensure optimization is stable (normalization, residuals, schedules)

3) regularize and validate (to generalize)

4) scale data/compute appropriately

Let’s connect the concepts to concrete workflows.

A practical blueprint: from data to model

Step 1: Represent input and output

• •Images: tensors $\in \mathbb{R}^{C\times H\times W}$
• •Text: token IDs → embeddings $\in \mathbb{R}^{T\times d}$
• •Tabular: normalized numeric + embeddings for categoricals

Decide output:

• •classification: softmax / sigmoid
• •regression: linear head (maybe with bounded activation)
• •sequence-to-sequence: autoregressive decoder or encoder-decoder

Step 2: Choose an inductive bias

• •Spatial locality? Use CNN or vision transformer with patches.
• •Long-range dependencies? Use attention.
• •Need streaming? Consider RNNs or efficient attention variants.

Step 3: Choose a loss and ensure numerical stability

For classification with logits $\mathbf{s}$ and label $y$:

Compute with stable log-sum-exp:

This prevents overflow in $e^{s_k}$.

Step 4: Optimization choices (SGD family)

Even if you know SGD, deep learning often uses schedules and adaptive methods.

Learning rate schedules (cosine decay, step decay, warmup) can be as important as the optimizer.

Step 5: Regularize for generalization

You already know L1/L2/dropout. In deep learning, common additional regularizers include:

• •data augmentation (especially vision)
• •early stopping
• •label smoothing
• •stochastic depth / drop-path (deep residual nets)

Worked mental model: “depth creates features, architecture chooses which features are easy”

Tie back to our earlier 2-layer classifier:

• •adding layers lets the model build intermediate representations that progressively linearize the task
• •CNN bias makes “edge-like” features easy to learn everywhere
• •attention bias makes “copy/align” behavior easy to learn across positions

Deep learning succeeds when your architecture makes the right representations cheap to discover with gradient descent.

Connections forward: why this node unlocks attention and meta-learning

• •Attention mechanisms refine the idea of learned representations by letting the model decide where to read from in its own activations.
• •Meta-learning treats the learning process itself as an object: deep nets can learn representations that adapt quickly, or learn optimizers/updates.

Final checkpoint

If you can explain:

1) $f_\theta$ as a composition of layers,

2) $\mathbf{h}^\ell$ as learned representations,

3) inductive bias as the reason architectures differ,

4) trainability as controlling Jacobian products,

…then you have a working deep learning “tech tree” model that scales to modern architectures.