Sequence-to-Sequence Modeling

Why this paradigm exists

Many real problems are not “predict a single label” but “produce a whole sequence whose length may differ from the input.” Examples:

• •Machine translation: English sentence → French sentence
• •Summarization: article → short summary
• •Speech recognition: audio frames → text tokens
• •Code generation: docstring → function

The defining feature is variable-length in, variable-length out, and output tokens depend on each other.

In our running example:

• •Source sequence x has length $S=5$ (English tokens)
• •Target sequence y has length $T=5$ (French tokens)

The goal is to model the conditional probability of the entire output sequence given the input sequence:

Autoregressive factorization (the key probabilistic move)

We don’t predict the whole sequence at once. We factorize it using the chain rule:

Here $y_{<t}$ means $(y_1,\ldots,y_{t-1})$.

This single equation captures the seq2seq contract:

• •At step $t$, the model has seen the entire input $x_{1:S}$.
• •It also conditions on the outputs it already produced.
• •It outputs a distribution over the next token $y_t$.

Encoder–decoder: turning the contract into an architecture

To implement this, we split responsibilities:

1. 1)Encoder reads the input sequence and produces internal representations.
2. 2)Decoder generates the output sequence one token at a time.

Historically, the encoder and decoder were RNNs (LSTM/GRU). Today, they are often Transformers—but the encoder/decoder roles persist.

A classic mental model:

• •The encoder produces a “memory” of the source.
• •The decoder is a “writer” that consults that memory to decide what to write next.

The bottleneck that motivates attention

Early seq2seq used a single vector (often the final encoder hidden state) as a summary of the entire source. This works for very short sentences but degrades on longer ones: the decoder must squeeze everything about $x_{1:S}$ through a fixed-size bottleneck.

Our five-word example is short, but it already hints at a deeper need: when the decoder generates “vertes” (green), it should access information from source position 3. If all information is compressed into a single vector, the decoder has to remember precise token-level details across steps.

This is why alignment/attention matters: it provides a direct path from each output step back to the relevant encoder states.

What you should be able to say after this section

• •Seq2seq is about $P(\mathbf{y} \mid \mathbf{x})$ where both sides are sequences.
• •Decoding is autoregressive: predict $y_t$ given $y_{<t}$ and the input.
• •Encoder–decoder splits the job into reading and writing.
• •A single-vector summary can be an information bottleneck, motivating attention.

Why understand the no-attention version first?

Attention can feel like “extra machinery.” But it’s easier to appreciate once you’ve seen what the encoder–decoder is trying to do on its own—and where it struggles.

We’ll describe a standard formulation using hidden states and probability distributions. Even if you later use a Transformer, the ideas map cleanly:

• •Encoder produces a sequence of representations (or a final summary).
• •Decoder maintains a state and emits tokens via softmax.

Step 1: Represent tokens as vectors

Tokens are discrete symbols. Models convert them into continuous vectors.

• •Input embedding for token $x_i$: e(xᵢ) ∈ ℝᵈ
• •Output embedding for token $y_{t-1}$ (previous token): e(y_{t-1})

Step 2: Encoder produces hidden states

Let the encoder produce hidden states hᵢ (one per source position):

• •$H = (h_1,\ldots,h_S)$ where each $h_i \in \mathbb{R}^{d_h}$

In an RNN encoder, a typical recurrence is:

Intuition:

• •$h_i$ is a contextual representation of the prefix $x_{1:i}$.
• •In a bidirectional encoder, $h_i$ can represent both left and right context.

Even in a Transformer encoder, you still end up with one vector per token position; you can still call them $h_i$.

Step 3: The simplest bottleneck: a single context vector

A classic early approach defines a single vector c summarizing the source, for example the last hidden state:

(There are other choices: pooling over $H$, or concatenating last states of a bidirectional RNN. But the core idea is “fixed-size summary.”)

Step 4: Decoder state and token generation

The decoder maintains a hidden state sₜ and emits a distribution over the next token:

1) Update decoder state:

2) Produce logits and probabilities:

The softmax is your prerequisite: it turns logits into a probability distribution over the vocabulary.

Training objective: teacher forcing + cross-entropy

During training, we usually feed the true previous token rather than the model’s sampled token (teacher forcing). The loss is negative log-likelihood:

Where $y_t^{\*}$ is the ground-truth token.

Inference: start/end tokens and decoding strategies

At inference time:

• •Start with a special token: $y_0 = \langle BOS \rangle$
• •Repeatedly sample/choose tokens until $\langle EOS \rangle$ or a max length.

Decoding choices:

Why the bottleneck is a real problem

If $c$ is fixed, then every output token must be generated from the same compressed summary.

Imagine generating “vertes” at the end. The decoder must:

• •Remember that “green” occurred in the input.
• •Remember that it modifies “apples” and thus should agree in gender/number.
• •Place it at the correct point in the output.

With a single $c$, the model can learn this sometimes, but it scales poorly with longer sequences and complex reorderings.

A “breathing room” intuition

Think of the encoder summary vector $c$ as trying to be a whole paragraph’s worth of meaning squeezed into one sticky note.

You can write a good sticky note.

But if the decoder could instead flip through the original paragraph whenever needed, it would make fewer mistakes.

That “flipping through” is exactly what attention adds.

What you should be able to do after this section

• •Write the factorization $\prod_t P(y_t \mid y_{<t}, x)$.
• •Describe how an encoder produces hidden states and (optionally) a single context vector.
• •Explain training with teacher forcing and cross-entropy.
• •Articulate the fixed-size bottleneck problem as the main motivation for attention.

Why attention: the decoder needs target-step-specific context

When producing $y_t$, the decoder doesn’t need the entire source equally.

• •To generate “Je”, focus on “I”.
• •To generate “mange”, focus on “eat”.
• •To generate “pommes”, focus on “apples”.
• •To generate “vertes”, focus back on “green”.

So instead of a single fixed $c$, we want a different context vector $c_t$ at each decoding step.

The objects (anchored to the node’s symbols)

You are given:

• •Encoder states: $H=(h_1,\ldots,h_S)$, one vector per source position.
• •Attention weights at step $t$: $\alpha_t = (\alpha_{t,1},\ldots,\alpha_{t,S})$.

Constraints:

• •$\alpha_{t,i} \ge 0$
• •$\sum_{i=1}^{S} \alpha_{t,i} = 1$ (because it’s softmax-normalized)

Interpretation:

• •$\alpha_{t,i}$ is “how much the decoder at time $t$ attends to source position $i$.”

Step 1: Compute attention scores (alignment scores)

We first compute an unnormalized score for each source position $i$.

A common pattern:

Where:

• •$s_{t-1}$ is the decoder’s previous hidden state (a summary of what has been generated so far).
• •$h_i$ is the encoder representation of source token $x_i$.

Typical scoring functions (you’ll see these in literature):

You don’t need to memorize them all right now—the pattern is what matters: compare the decoder’s needs to each encoder state.

Step 2: Softmax into attention weights αₜ

Convert scores into a distribution over positions:

This is exactly where your softmax prerequisite shows up.

Numerical stability reminder (important in real implementations):

Step 3: Build the context vector cₜ as a weighted sum of encoder states

Now we compute:

Interpretation:

• •$c_t$ lives in the same vector space as $h_i$.
• •If $\alpha_{t,3}$ is large, $c_t$ is pulled toward $h_3$.

This makes attention feel like a soft pointer into the source.

Step 4: Use cₜ to update the decoder and produce yₜ

A common formulation:

Then output distribution:

Sometimes the context is also fed directly into the output layer, e.g. concatenate $[s_t; c_t]$.

Anchoring back to our 5-word example: what αₜ should look like

Let’s label source positions:

1:I, 2:eat, 3:green, 4:apples, 5:today

A plausible alignment pattern:

• •When $y_1=\text{Je}$, attention peaks on source 1.
• •When $y_2=\text{mange}$, attention peaks on source 2.
• •When $y_4=\text{pommes}$, attention peaks on source 4.
• •When $y_5=\text{vertes}$, attention peaks on source 3.

So $\alpha_5$ might heavily weight position 3.

Inline “alignment table” diagram

Below is a rough alignment matrix (rows are target steps t, columns are source positions i). Darker means larger α.

This is the alignment concept: for each output step, the model forms a distribution over input positions.

Why attention improves information flow (conceptually)

Without attention, the only path from input token $x_i$ to output decision at time $t$ is:

$x_i \to$ encoder computations $\to$ single summary $c \to$ decoder state $s_t \to y_t$

With attention:

$x_i \to h_i \to c_t \to s_t \to y_t$

Now each output step has a direct, learnable channel to any encoder state.

A careful note: attention weights are not always “true explanations”

It’s tempting to say “αₜ tells you exactly which input word caused the output.” Often it correlates with alignment, but:

• •The decoder state $s_{t-1}$ also carries information.
• •Multiple layers and nonlinearities can mix information.

So treat attention as:

• •A useful alignment-like mechanism
• •Sometimes an interpretability aid
• •Not a perfect causal explanation by default

What you should be able to do after this section

• •Define $H=(h_1,\ldots,h_S)$ and $\alpha_t$.
• •Write $\alpha_{t,i}=\text{softmax}(e_{t,i})$ and $c_t=\sum_i \alpha_{t,i}h_i$.
• •Explain attention as a per-output-step soft selection over source positions.
• •Connect attention back to reorderings like placing “vertes” after “pommes”.

From “alignment” to modern architectures

The attention we described is often called encoder–decoder attention or cross-attention:

• •Queries come from the decoder state (what we need now)
• •Keys/values come from encoder states $H$ (what the source contains)

Even if you haven’t learned Transformer math yet, the conceptual mapping is straightforward:

• •Our score function $\text{score}(s_{t-1}, h_i)$ becomes “query–key compatibility.”
• •Our weighted sum $c_t=\sum_i \alpha_{t,i}h_i$ becomes “weighted sum of values.”

This is exactly why understanding seq2seq clarifies attention: it tells you what problem attention is solving.

Real tasks and what changes

Seq2seq setups vary mainly in:

1) What counts as the “sequence” on the input side

2) What output vocabulary/tokenization looks like

3) How decoding is constrained

Examples:

The encoder can be anything that outputs a sequence of vectors $H$:

• •RNN/Transformer over tokens
• •CNN/ViT over image patches
• •Acoustic model over spectrogram frames

As long as you have $H$, the decoder can attend over it.

Decoding details that matter in applications

Even with a good model, the way you decode changes behavior:

• •Beam search often improves translation quality, but can produce bland outputs.
• •Length normalization is common because log-probabilities favor short sequences.

A typical beam objective may look like:

where $\gamma$ controls length penalty.

Exposure bias (a practical training vs inference gap)

Training uses teacher forcing: decoder sees true $y_{t-1}^{\*}$.

Inference uses its own previous prediction $\hat{y}_{t-1}$.

So the model may drift if it makes an early mistake. This issue is called exposure bias.

Mitigations you may hear about:

• •Scheduled sampling
• •Sequence-level training objectives (e.g., policy gradient)
• •Stronger models + better decoding

You don’t need to solve exposure bias now, but you should recognize it as a recurring theme in seq2seq.

Why attention is a stepping stone to Transformers

In Transformers:

• •The encoder uses self-attention to build $H$.
• •The decoder uses masked self-attention to model $P(y_t \mid y_{<t})$.
• •The decoder uses cross-attention to incorporate the source (our $\alpha_t$ idea).

So the conceptual flow becomes:

1) Encode source into $H$

2) For each decoding step t, compute attention distribution $\alpha_t$ over $H$

3) Use the resulting context to predict $y_t$

If you understand that, the jump to “multi-head attention” is mostly an engineering/generalization step: do it several times in parallel with different learned projections.

Returning one last time to the mini-scenario

Our translator succeeded not because it memorized the entire input in one vector, but because at each step it can ask:

• •“What part of the source is relevant right now?”

When producing “vertes”, it can attend back to “green” even though that occurred earlier and has already influenced other outputs.

That dynamic reading behavior is the essence of seq2seq alignment.

What you should be able to do after this section

• •Recognize encoder–decoder attention as cross-attention.
• •Name common seq2seq applications and what differs.
• •Explain exposure bias and why decoding strategy matters.
• •See how this node unlocks the more general Attention Mechanisms node.