# Soft Alignment in Sequence-to-Sequence Models

> One thread of 5 from the "attention" map, covering the 17 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Self-Attention & Transformers, Attention as Learned Weighting & Gating.

Pioneered by Bahdanau et al.'s alignment mechanism in neural machine translation, this thread develops attention as a soft, differentiable weighting of source representations to guide decoding. Attention computes normalized weights over encoder states at each decoder step, enabling models to jointly learn alignment and prediction. This formulation becomes the template for all downstream task adaptations.


**Generating Sequences With Recurrent Neural Networks** (2013) \cite{graves2013generating}

What it did: Introduces soft alignment via Gaussian kernels for sequence conditioning  *(the tool's reading)*

A soft-alignment mechanism using mixtures of Gaussian kernels determines which parts of a conditioning sequence the network attends to. This enables sequence-to-sequence prediction without requiring fixed alignment.  *(the tool's reading)*

> “Section […] introduces an extension to the prediction network that allows it to condition its outputs on a short annotation sequence whose alignment with the predictions is unknown.”
>
> ✓ verified: found word for word in the paper's own text


**Neural Machine Translation by Jointly Learning to Align and Translate** (2014) \cite{bahdanau2014neural}

What it did: Generalizes soft alignment to flexible learned weighting over all annotations  *(the tool's reading)*

Instead of fixed Gaussian windows, a learned scoring function computes soft weights over encoder annotations at each decoder step. This avoids the fixed-length bottleneck by dynamically aligning target generation to source representations.  *(the tool's reading)*

> “Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to.”
>
> ✓ verified: found word for word in the paper's own text


**End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results** (2014) \cite{chorowski2014end}

What it did: Adapts alignment mechanism to relative position-aware scoring  *(the tool's reading)*

The attention mechanism learns position-dependent scoring in speech recognition contexts. Relative position information is incorporated into the soft weighting over encoder states.  *(the tool's reading)*

> “The context is a weighted sum of annotations: c_o = ∑_i=1^I α_o,i h_i, where α_o,i is a normalized weight for each annotation h_i. This effectively means that the decoder selects each annotation h_i with a certainty α_o,i.”
>
> ✓ verified: found word for word in the paper's own text


**Show, Attend and Tell: Neural Image Caption Generation with Visual Attention** (2015) \cite{xu2015show}

> “Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed.”
>
> ✓ verified: found word for word in the paper's own text


**Teaching Machines to Read and Comprehend** (2015) \cite{hermann2015teaching}

> “The representation r of the document d is formed by a weighted sum of these output vectors. These weights are interpreted as the degree to which the network attends to a particular token in the document when answering the query”
>
> ✓ verified: found word for word in the paper's own text


**Attention-Based Models for Speech Recognition** (2015) \cite{chorowski2015attention}

> “An attention-based recurrent sequence generator (ARSG) is a recurrent neural network that stochastically generates an output sequence (y_1, …, y_T) from an input x.”
>
> ✓ verified: found word for word in the paper's own text


**Effective Approaches to Attention-based Neural Machine Translation** (2015) \cite{luong2015effective}

What it did: Introduces local attention attending to source word subsets  *(the tool's reading)*

Attention is split into global (all source words) and local (source subset) variants for neural machine translation. The local approach selectively attends to a time-dependent window of source positions.  *(the tool's reading)*

> “The idea of a global attentional model is to consider all the hidden states of the encoder when deriving the context vector c_t.”
>
> ✓ verified: found word for word in the paper's own text


**A Neural Attention Model for Abstractive Sentence Summarization** (2015) \cite{rush2015neural}

> “A similar issue in machine translation inspired […] to instead utilize an attention-based contextual encoder that constructs a representation based on the generation context.”
>
> ✓ verified: found word for word in the paper's own text


**Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems** (2015) \cite{raffel2015feed}

What it did: Simplifies attention to feedforward context vector computation  *(the tool's reading)*

A stripped-down feedforward attention mechanism computes weighted averages without recurrent complexity. This makes soft alignment applicable to simpler architectures while maintaining the core weighting principle.  *(the tool's reading)*

> “Attention mechanisms allow for a more direct dependence between the state of the model at different points in time. Following the definition from […], given a model which produces a hidden state h_t at each time step, attention-based models compute a “context” vector c_t as the weighted mean of the state sequence h by”
>
> ✓ verified: found word for word in the paper's own text


**Long Short-Term Memory-Networks for Machine Reading** (2016) \cite{cheng2016long}

> “This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens.”
>
> ✓ verified: found word for word in the paper's own text


**Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation** (2016) \cite{wu2016google}

What it did: Connects attention across non-adjacent encoder-decoder layers  *(the tool's reading)*

Attention aligns decoder bottom layers to encoder top layers rather than symmetric layer pairing. This improves parallelism while maintaining the soft-weighted alignment mechanism.  *(the tool's reading)*

> “Our attention module is similar to […]. More specifically, let 𝐲_i-1 be the decoder-RNN output from the past decoding time step (in our implementation, we use the output from the bottom decoder layer). Attention context 𝐚_i for the current time step is computed according to the following formulas:”
>
> ✓ verified: found word for word in the paper's own text


**Bidirectional Attention Flow for Machine Comprehension** (2016) \cite{seo2016bidirectional}

> “One of the key factors to the advancement has been the use of neural attention mechanism, which enables the system to focus on a targeted area within a context paragraph (for MC) or within an image (for Visual QA), that is most relevant to answer the question […].”
>
> ✓ verified: found word for word in the paper's own text


**Deep Biaffine Attention for Neural Dependency Parsing** (2016) \cite{atmani2016deep}

> “In […]'s […] model, the (bidirectional) LSTM's recurrent output vector for each word is concatenated with each possible head's recurrent vector, and the result is used as input to an MLP that scores each resulting arc.”
>
> ✓ verified: found word for word in the paper's own text


**An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data** (2016) \cite{song2016end}

> “With a soft attention mechanism, each joint within a frame is assigned a spatial attention weight based on the joint-selection gates. This enables our model to adaptively focus more on those discriminative joints.”
>
> ✓ verified: found word for word in the paper's own text


**Convolutional Sequence to Sequence Learning** (2017) \cite{gehring2017convolutional}

What it did: Applies attention to every convolutional decoder layer independently  *(the tool's reading)*

Weighted context vectors are computed at each decoder layer step using the same scoring mechanism. Multiple attention layers add minimal overhead while enabling deep convolutional sequence processing.  *(the tool's reading)*

> “We introduce a separate attention mechanism for each decoder layer.”
>
> ✓ verified: found word for word in the paper's own text


**A Deep Reinforced Model for Abstractive Summarization** (2017) \cite{paulus2017deep}

> “At each decoding step t, we use an intra-temporal attention function to attend over specific parts of the encoded input sequence in addition to the decoder's own hidden state and the previously-generated word […].”
>
> ✓ verified: found word for word in the paper's own text


**LSTM Fully Convolutional Networks for Time Series Classification** (2017) \cite{karim2017lstm}

> “The attention mechanism is a technique often used in neural translation of text, where a context vector C is conditioned on the target sequence y. As discussed in Bahdanau et al. […], the context vector c_i depends on a sequence of annotations (h_1,..., h_T_x) to which an encoder maps the input sequence.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Soft Alignment in Sequence-to-Sequence Models** → **Self-Attention & Transformers** (2017): Scaled dot-product unifies alignment
- **Soft Alignment in Sequence-to-Sequence Models** → **Attention as Learned Weighting & Gating** (2015): Weights generalize beyond alignment