# How "attention" developed

> Generated by Lineage from the 84 papers in these 5 threads.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

Attention evolved from a biological vision concept into a learnable neural mechanism, progressing from task-specific soft alignment in sequence models to a universal, efficient architectural primitive underlying modern deep learning.

_[Write your framing paragraph here: which thread matters for your work, and why.]_


## 1. Saliency & Biological Vision (2012–2021)

*Attention as selective visual processing in biological systems and computational saliency models.*

Treats attention as the cognitive/biological mechanism of selective gaze and fixation in visual processing. Papers in this thread focus on detecting salient regions, modeling human eye fixations, and understanding visual attention as a perceptual phenomenon rather than as a learnable neural module. This thread remains largely parallel to neural attention mechanisms throughout the corpus.


**Visual Saliency Based on Scale-Space Analysis in the Frequency Domain** (2012) \cite{li2012visual}

What it did: Established saliency as biological visual selection via frequency analysis  *(the tool's reading)*

Introduced scale-space analysis of amplitude spectrum using Hypercomplex Fourier Transform to model bottom-up saliency mechanisms that competitively select salient regions for prioritized neural processing.  *(the tool's reading)*

> “Visual attention facilitates our ability to rapidly locate the most important information in a scene […].”
>
> ✓ verified: found word for word in the paper's own text


**Salient Object Detection: A Benchmark** (2015) \cite{borji2015salient}

What it did: Clarified distinction between salient object detection and fixation prediction  *(the tool's reading)*

Refined saliency modeling by formally separating salient object detection from attention-driven fixation prediction and objectness detection, grounding all approaches in biological visual selectivity.  *(the tool's reading)*

> “VISUAL attention, the astonishing capability of human visual system to selectively process only the salient visual stimuli in details, has been investigated by multiple disciplines such as cognitive psychology, neuroscience, and computer vision […].”
>
> ✓ verified: found word for word in the paper's own text


**What Makes for Effective Detection Proposals?** (2015) \cite{hosang2015what}

What it did: Connected saliency to computational interest-point detection frameworks  *(the tool's reading)*

Drew explicit analogy between biological saliency-driven attention and interest point detectors, positioning both as mechanisms for focusing computation on salient locations.  *(the tool's reading)*

> “Interest points allow for focusing attention to the most salient and distinctive locations in an image, greatly reducing computation for subsequent tasks such as classification, retrieval, matching, and detection.”
>
> ✓ verified: found word for word in the paper's own text


**DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection** (2015) \cite{li2015deepsaliency}

What it did: Incorporated perceptual features driving human gaze allocation  *(the tool's reading)*

Extended saliency models to encompass human perceptual behaviors including color, contrast, and semantic features that direct biological gaze toward image regions.  *(the tool's reading)*

> “Specifically, the saliency value of the red flower in Figure […](a) is higher than the yellow one, which is consistent with human perception (paying more attention to the objects with red bright colors […]).”
>
> ✓ verified: found word for word in the paper's own text


**Video Salient Object Detection via Fully Convolutional Networks** (2017) \cite{wang2017video}

What it did: Extended salient object detection to temporal visual threads  *(the tool's reading)*

Advanced saliency research beyond static images to video by maintaining distinction between fixation prediction and salient object detection in temporal domain.  *(the tool's reading)*

> “Saliency detection has been extensively studied in computer vision, and saliency models in general can be categorized into visual attention prediction or salient object detection. The former methods […] try to predict scene locations where a human observer may fixate.”
>
> ✓ verified: found word for word in the paper's own text


**Attention mechanisms in computer vision: A survey** (2021) \cite{guo2021attention}

What it did: Generalized attention as adaptive feature reweighting across modalities  *(the tool's reading)*

Broadened attention mechanism definition from purely biological saliency to dynamic computational processes that recalibrate feature weights across multiple dimensions and modalities.  *(the tool's reading)*

> “In computer vision, {the} attention mechanism can be regarded as a dynamic weight recalibration process based on the input features.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis of this thread: what it enabled, what it left unsolved.]_


## 2. Soft Alignment in Sequence-to-Sequence Models (2013–2017)

*Attention as learned alignment between encoded and decoded sequences in recurrent architectures.*

Pioneered by Bahdanau et al.'s alignment mechanism in neural machine translation, this thread develops attention as a soft, differentiable weighting of source representations to guide decoding. Attention computes normalized weights over encoder states at each decoder step, enabling models to jointly learn alignment and prediction. This formulation becomes the template for all downstream task adaptations.


**Generating Sequences With Recurrent Neural Networks** (2013) \cite{graves2013generating}

What it did: Introduces soft alignment via Gaussian kernels for sequence conditioning  *(the tool's reading)*

A soft-alignment mechanism using mixtures of Gaussian kernels determines which parts of a conditioning sequence the network attends to. This enables sequence-to-sequence prediction without requiring fixed alignment.  *(the tool's reading)*

> “Section […] introduces an extension to the prediction network that allows it to condition its outputs on a short annotation sequence whose alignment with the predictions is unknown.”
>
> ✓ verified: found word for word in the paper's own text


**Neural Machine Translation by Jointly Learning to Align and Translate** (2014) \cite{bahdanau2014neural}

What it did: Generalizes soft alignment to flexible learned weighting over all annotations  *(the tool's reading)*

Instead of fixed Gaussian windows, a learned scoring function computes soft weights over encoder annotations at each decoder step. This avoids the fixed-length bottleneck by dynamically aligning target generation to source representations.  *(the tool's reading)*

> “Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to.”
>
> ✓ verified: found word for word in the paper's own text


**End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results** (2014) \cite{chorowski2014end}

What it did: Adapts alignment mechanism to relative position-aware scoring  *(the tool's reading)*

The attention mechanism learns position-dependent scoring in speech recognition contexts. Relative position information is incorporated into the soft weighting over encoder states.  *(the tool's reading)*

> “The context is a weighted sum of annotations: c_o = ∑_i=1^I α_o,i h_i, where α_o,i is a normalized weight for each annotation h_i. This effectively means that the decoder selects each annotation h_i with a certainty α_o,i.”
>
> ✓ verified: found word for word in the paper's own text


**Show, Attend and Tell: Neural Image Caption Generation with Visual Attention** (2015) \cite{xu2015show}

> “Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed.”
>
> ✓ verified: found word for word in the paper's own text


**Teaching Machines to Read and Comprehend** (2015) \cite{hermann2015teaching}

> “The representation r of the document d is formed by a weighted sum of these output vectors. These weights are interpreted as the degree to which the network attends to a particular token in the document when answering the query”
>
> ✓ verified: found word for word in the paper's own text


**Attention-Based Models for Speech Recognition** (2015) \cite{chorowski2015attention}

> “An attention-based recurrent sequence generator (ARSG) is a recurrent neural network that stochastically generates an output sequence (y_1, …, y_T) from an input x.”
>
> ✓ verified: found word for word in the paper's own text


**Effective Approaches to Attention-based Neural Machine Translation** (2015) \cite{luong2015effective}

What it did: Introduces local attention attending to source word subsets  *(the tool's reading)*

Attention is split into global (all source words) and local (source subset) variants for neural machine translation. The local approach selectively attends to a time-dependent window of source positions.  *(the tool's reading)*

> “The idea of a global attentional model is to consider all the hidden states of the encoder when deriving the context vector c_t.”
>
> ✓ verified: found word for word in the paper's own text


**A Neural Attention Model for Abstractive Sentence Summarization** (2015) \cite{rush2015neural}

> “A similar issue in machine translation inspired […] to instead utilize an attention-based contextual encoder that constructs a representation based on the generation context.”
>
> ✓ verified: found word for word in the paper's own text


**Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems** (2015) \cite{raffel2015feed}

What it did: Simplifies attention to feedforward context vector computation  *(the tool's reading)*

A stripped-down feedforward attention mechanism computes weighted averages without recurrent complexity. This makes soft alignment applicable to simpler architectures while maintaining the core weighting principle.  *(the tool's reading)*

> “Attention mechanisms allow for a more direct dependence between the state of the model at different points in time. Following the definition from […], given a model which produces a hidden state h_t at each time step, attention-based models compute a “context” vector c_t as the weighted mean of the state sequence h by”
>
> ✓ verified: found word for word in the paper's own text


**Long Short-Term Memory-Networks for Machine Reading** (2016) \cite{cheng2016long}

> “This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens.”
>
> ✓ verified: found word for word in the paper's own text


**Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation** (2016) \cite{wu2016google}

What it did: Connects attention across non-adjacent encoder-decoder layers  *(the tool's reading)*

Attention aligns decoder bottom layers to encoder top layers rather than symmetric layer pairing. This improves parallelism while maintaining the soft-weighted alignment mechanism.  *(the tool's reading)*

> “Our attention module is similar to […]. More specifically, let 𝐲_i-1 be the decoder-RNN output from the past decoding time step (in our implementation, we use the output from the bottom decoder layer). Attention context 𝐚_i for the current time step is computed according to the following formulas:”
>
> ✓ verified: found word for word in the paper's own text


**Bidirectional Attention Flow for Machine Comprehension** (2016) \cite{seo2016bidirectional}

> “One of the key factors to the advancement has been the use of neural attention mechanism, which enables the system to focus on a targeted area within a context paragraph (for MC) or within an image (for Visual QA), that is most relevant to answer the question […].”
>
> ✓ verified: found word for word in the paper's own text


**Deep Biaffine Attention for Neural Dependency Parsing** (2016) \cite{atmani2016deep}

> “In […]'s […] model, the (bidirectional) LSTM's recurrent output vector for each word is concatenated with each possible head's recurrent vector, and the result is used as input to an MLP that scores each resulting arc.”
>
> ✓ verified: found word for word in the paper's own text


**An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data** (2016) \cite{song2016end}

> “With a soft attention mechanism, each joint within a frame is assigned a spatial attention weight based on the joint-selection gates. This enables our model to adaptively focus more on those discriminative joints.”
>
> ✓ verified: found word for word in the paper's own text


**Convolutional Sequence to Sequence Learning** (2017) \cite{gehring2017convolutional}

What it did: Applies attention to every convolutional decoder layer independently  *(the tool's reading)*

Weighted context vectors are computed at each decoder layer step using the same scoring mechanism. Multiple attention layers add minimal overhead while enabling deep convolutional sequence processing.  *(the tool's reading)*

> “We introduce a separate attention mechanism for each decoder layer.”
>
> ✓ verified: found word for word in the paper's own text


**A Deep Reinforced Model for Abstractive Summarization** (2017) \cite{paulus2017deep}

> “At each decoding step t, we use an intra-temporal attention function to attend over specific parts of the encoded input sequence in addition to the decoder's own hidden state and the previously-generated word […].”
>
> ✓ verified: found word for word in the paper's own text


**LSTM Fully Convolutional Networks for Time Series Classification** (2017) \cite{karim2017lstm}

> “The attention mechanism is a technique often used in neural translation of text, where a context vector C is conditioned on the target sequence y. As discussed in Bahdanau et al. […], the context vector c_i depends on a sequence of annotations (h_1,..., h_T_x) to which an encoder maps the input sequence.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis of this thread: what it enabled, what it left unsolved.]_


## 3. Recurrent Glimpse-Based Visual Attention (2014–2018)

*Attention as a sequential, reinforcement-learning driven mechanism for selective spatial processing of images.*

Frames attention as a control problem where an RNN sequentially selects 'glimpses' or spatial regions to process, using reinforcement learning or differentiable soft-attention. Pioneered by models that treat visual attention as an iterative, foveation-like process. This thread bridges biological vision intuitions with learnable neural mechanisms but remains confined to recurrent, step-wise processing.


**Recurrent Models of Visual Attention** (2014) \cite{mnih2014recurrent}

What it did: Frames visual attention as sequential glimpse selection via RNN and reinforcement learning  *(the tool's reading)*

This paper introduces attention-based visual processing as a control problem where a recurrent agent sequentially selects limited image regions (glimpses) for processing at each time step. The approach uses an RNN combined with policy gradient learning for end-to-end training, moving beyond prior greedy or hardwired selection strategies.  *(the tool's reading)*

> “Instead humans focus attention selectively on parts of the visual space to acquire information when and where it is needed, and combine information from different fixations over time to build up an internal representation of the scene […], guiding future eye movements and decision making.”
>
> ✓ verified: found word for word in the paper's own text


**Multiple Object Recognition with Visual Attention** (2014) \cite{ba2014multiple}

What it did: Scales recurrent glimpse-based attention to real-world images and multiple objects  *(the tool's reading)*

This work extends the recurrent attention model (RAM) framework to handle real-world image complexity and multiple objects simultaneously, whereas prior RAM work only demonstrated performance on cluttered digit classification and toy control tasks.  *(the tool's reading)*

> “We present an attention-based model for recognizing multiple objects in images. The proposed model is a deep recurrent neural network trained with reinforcement learning to attend to the most relevant regions of the input image.”
>
> ✓ verified: found word for word in the paper's own text


**DRAW: A Recurrent Neural Network For Image Generation** (2015) \cite{gregor2015draw}

What it did: Replaces discrete REINFORCE-trained glimpses with fully differentiable Gaussian-filter attention  *(the tool's reading)*

DRAW introduces a fully differentiable two-dimensional spatial attention mechanism using 2D Gaussian filters that determine where the network reads from and writes to, explicitly mimicking human foveation. This replaces the discrete, REINFORCE-trained glimpse policy of prior RAM approaches while combining attention with sequential variational auto-encoding.  *(the tool's reading)*

> “Unlike the aforementioned works, we consider an explicitly two-dimensional form of attention, where an array of 2D Gaussian filters is applied to the image, yielding an image `patch' of smoothly varying location and zoom.”
>
> ✓ verified: found word for word in the paper's own text


**Dynamic Memory Networks for Visual and Textual Question Answering** (2016) \cite{xiong2016dynamic}

What it did: Adapts recurrent attention mechanism to gate facts in question-answering tasks  *(the tool's reading)*

This paper reinterprets attention as gating over input facts using global knowledge from questions and memory states to determine relevance, proposing an attention-based GRU variant. The mechanism shifts recurrent glimpse-based attention from visual navigation to multi-fact filtering in reasoning tasks.  *(the tool's reading)*

> “We implement this attention by associating a single scalar value, the attention gate g^t_i, with each fact f_i during pass t.”
>
> ✓ verified: found word for word in the paper's own text


**End-to-End Comparative Attention Networks for Person Re-Identification** (2016) \cite{liu2016end}

What it did: Applies recurrent LSTM-based attention to spatial masking for person re-identification  *(the tool's reading)*

This work implements recurrent attention via an LSTM that generates softmax-weighted spatial masks over CNN feature maps, enabling selective focus on discriminative local regions in person images. The approach maintains the recurrent glimpse paradigm while applying it to fine-grained feature selection rather than whole-image navigation.  *(the tool's reading)*

> “Briefly, similar to human visual processing, attention-based algorithms tend to selectively concentrate on a part of the information, and at the same time ignore other perceived information. Such a mechanism is usually called attention and can be employed to adaptively localize discriminative parts or regions of person images.”
>
> ✓ verified: found word for word in the paper's own text


**An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data** (2016) \cite{song2016end}

What it did: Extends recurrent attention to dual spatial-temporal gating in skeleton action recognition  *(the tool's reading)*

This paper frames attention as soft, content-dependent gates operating across both spatial dimensions (skeleton joints) and temporal dimensions (frames) within LSTM networks. The work generalizes recurrent attention beyond single-modality vision tasks to structured skeletal and temporal data for action understanding.  *(the tool's reading)*

> “With a soft attention mechanism, each joint within a frame is assigned a spatial attention weight based on the joint-selection gates. This enables our model to adaptively focus more on those discriminative joints.”
>
> ✓ verified: found word for word in the paper's own text


**Pedestrian Alignment Network for Large-scale Person Re-Identification** (2017) \cite{zheng2017pedestrian}

What it did: Replaces explicit recurrent LSTM attention with implicit CNN feature-based alignment  *(the tool's reading)*

This paper proposes that attention emerges implicitly from CNN training for identity classification without extra annotations or explicit LSTM/gating mechanisms. The work critiques the computational overhead of recurrent LSTM-based attention, shifting toward emergent alignment learned directly from the primary task.  *(the tool's reading)*

> “The proposed network thus takes advantage of this attention mechanism to adaptively locate and align pedestrians within a bounding box.”
>
> ✓ verified: found word for word in the paper's own text


**Attention-based Deep Multiple Instance Learning** (2018) \cite{ilse2018attention}

What it did: Reinterprets attention as permutation-invariant learned pooling for instance aggregation  *(the tool's reading)*

This paper abstracts attention away from sequential glimpse selection or spatial gating, reframing it as a trainable, permutation-invariant weighted-average pooling operator that aggregates instance embeddings. The mechanism replaces pre-defined pooling operators with a learnable two-layer neural network, moving attention from spatial/temporal selection to aggregate-level weighting.  *(the tool's reading)*

> “Interestingly, the proposed MIL pooling corresponds to a version of the attention mechanism […]. The main difference is that typically in the attention mechanism all instances are sequentially dependent while here we assume that all instances are independent.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis of this thread: what it enabled, what it left unsolved.]_


## 4. Self-Attention & Transformers (2017–2021)

*Attention as a universal, parallelizable mechanism for computing relationships between all positions in a sequence.*

Inaugurated by the Transformer, this thread redefines attention as scaled dot-product self-attention over positions within a single sequence (or modality), computing query-key-value relationships. Eliminates recurrence and decoder-encoder asymmetry, enabling fully parallel training and inference. Rapidly becomes the foundational architecture for language models, vision models, and multimodal systems, with subsequent work optimizing efficiency, interpretability, and structural variants.


**A Structured Self-attentive Sentence Embedding** (2017) \cite{lin2017structured}

What it did: Self-attention computes weights via MLP without external inputs  *(the tool's reading)*

Paper [30] introduces self-attention as an MLP-based weighting mechanism operating on LSTM hidden states, enabling attention purely from internal representations. This establishes attention operating without separate key-value inputs.  *(the tool's reading)*

> “Our aim is to encode a variable length sentence into a fixed size embedding. We achieve that by choosing a linear combination of the n LSTM hidden vectors in H. Computing the linear combination requires the self-attention mechanism. The attention mechanism takes the whole LSTM hidden states H as input, and outputs a vector of weights 𝐚:”
>
> ✓ verified: found word for word in the paper's own text


**Attention Is All You Need** (2017) \cite{vaswani2017attention}

What it did: Scaled dot-product attention replaces RNNs entirely in Transformers  *(the tool's reading)*

Paper [34] defines attention as query-key-value mapping with scaled dot-product computation and removes RNNs/convolutions completely. The Transformer becomes the first model relying entirely on self-attention for representation.  *(the tool's reading)*

> “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.”
>
> ✓ verified: found word for word in the paper's own text


**Graph Attention Networks** (2017) \cite{velivckovic2017graph}

What it did: Masked self-attention extends to graph-structured data  *(the tool's reading)*

Paper [38] applies masked self-attention to graph neighborhoods instead of sequences, computing learned importance weights. This generalizes Transformer attention beyond sequential data.  *(the tool's reading)*

> “One of the benefits of attention mechanisms is that they allow for dealing with variable sized inputs, focusing on the most relevant parts of the input to make decisions.”
>
> ✓ verified: found word for word in the paper's own text


**Image Transformer** (2018) \cite{parmar2018image}

What it did: Self-attention scales to image generation via sequence modeling  *(the tool's reading)*

Paper [41] applies scaled dot-product self-attention to image generation, treating pixels as sequence positions. This extends Transformer attention to visual domains with tractable likelihood.  *(the tool's reading)*

> “Each self-attention layer computes a d-dimensional representation for each position, that is, each channel of each pixel. To recompute the representation for a given position, it first compares the position's current representation to other positions' representations, obtaining an attention distribution over the other positions. This distribution is then used to weight the contribution of the other positions' representations to the next representation for the position at hand.”
>
> ✓ verified: found word for word in the paper's own text


**Self-Attention with Relative Position Representations** (2018) \cite{shaw2018self}

What it did: Relative position representations augment scaled dot-product attention  *(the tool's reading)*

Paper [42] extends Transformer self-attention by incorporating relative position representations alongside dot-product computation. This adds structural awareness to standard attention weights.  *(the tool's reading)*

> “Each attention head operates on an input sequence, x = (x_1, …, x_n) of n elements where x_i ∈ℝ^d_x, and computes a new sequence z = (z_1, …, z_n) of the same length where z_i ∈ℝ^d_z.”
>
> ✓ verified: found word for word in the paper's own text


**QANet: Combining Local Convolution with Global Self-Attention for\n Reading Comprehension** (2018) \cite{yu2018qanet}

What it did: Self-attention replaces max pooling in reading comprehension  *(the tool's reading)*

Paper [45] combines convolutions exclusively with self-attention for reading comprehension, using dot-product query-key similarity. This demonstrates self-attention as pooling replacement across modalities.  *(the tool's reading)*

> “For the self-attention-layer, we adopt the multi-head attention mechanism defined in […] which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product.”
>
> ✓ verified: found word for word in the paper's own text


**Self-Attention Generative Adversarial Networks** (2018) \cite{zhang2018self}

What it did: Self-attention enters GANs for spatial image feature weighting  *(the tool's reading)*

Paper [46] introduces self-attention into GAN generators and discriminators, computing spatial feature responses as weighted sums. This marks first self-attention use in adversarial training.  *(the tool's reading)*

> “The self-attention module calculates response at a position as a weighted sum of the features at all positions, where the weights – or attention vectors – are calculated with only a small computational cost.”
>
> ✓ verified: found word for word in the paper's own text


**OCNet: Object Context Network for Scene Parsing** (2018) \cite{yuan2018ocnet}

What it did: Sparse self-attention approximates dense relation matrices efficiently  *(the tool's reading)*

Paper [48] proposes interlaced sparse self-attention to approximate dense position-weighted aggregation matrices. This reduces quadratic computational cost while preserving self-attention semantics.  *(the tool's reading)*

> “The self-attention scheme […] calculates the context at one position as a aggregation of all positions in a sentence (at the encoder stage).”
>
> ✓ verified: found word for word in the paper's own text


**Deep Audio-Visual Speech Recognition** (2018) \cite{afouras2018deep}

What it did: Dual self-attention extends Transformer to audio-visual modeling  *(the tool's reading)*

Paper [49] extends Transformer self-attention to dual encoder-decoder setup for multimodal speech recognition. This maintains standard scaled dot-product mechanism across new modalities.  *(the tool's reading)*

> “the architecture incorporates a novel dual attention mechanism that can operate over visual input only, audio input only, or both.”
>
> ✓ verified: found word for word in the paper's own text


**Generating Long Sequences with Sparse Transformers** (2019) \cite{child2019generating}

What it did: Sparse factorizations reduce attention complexity from O(n²) to O(n√n)  *(the tool's reading)*

Paper [54] introduces sparse factorizations of full attention matrices via connectivity patterns. This dramatically reduces quadratic scaling without sacrificing performance on long sequences.  *(the tool's reading)*

> “A self-attention layer maps a matrix of input embeddings X to an output matrix and is parameterized by a connectivity pattern S = {S_1,..., S_n}, where S_i denotes the set of indices of the input vectors to which the ith output vector attends.”
>
> ✓ verified: found word for word in the paper's own text


**Unified Language Model Pre-training for Natural Language Understanding\n and Generation** (2019) \cite{dong2019unified}

What it did: Masked self-attention enables unified language model pre-training  *(the tool's reading)*

Paper [55] uses Transformer self-attention with specific masking patterns to control context conditioning. This applies standard mechanisms to unified bidirectional pre-training.  *(the tool's reading)*

> “In each Transformer block, multiple self-attention heads are used to aggregate the output vectors of the previous layer.”
>
> ✓ verified: found word for word in the paper's own text


**Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned** (2019) \cite{voita2019analyzing}

What it did: Individual attention heads specialize; pruning them preserves performance  *(the tool's reading)*

Paper [58] analyzes multi-head self-attention confidence scores and introduces head pruning via stochastic gates. This reveals that individual heads specialize without affecting overall performance.  *(the tool's reading)*

> “The multi-head attention mechanism obtains h (i.e. one per head) different representations of (Q, K, V), computes scaled dot-product attention for each representation, concatenates the results, and projects the concatenation through a feed-forward layer.”
>
> ✓ verified: found word for word in the paper's own text


**Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting** (2019) \cite{li2019enhancing}

What it did: Convolutional self-attention incorporates local context into dot-product  *(the tool's reading)*

Paper [59] modifies Transformer attention for time series by using causal convolutions to produce queries and keys. This replaces point-wise dot-product with locally-contextual attention matching.  *(the tool's reading)*

> “After these linear projections, the scaled dot-product attention computes a sequence of vector outputs: 𝐎_h = Attention(𝐐_h, 𝐊_h, 𝐕_h)= softmax(𝐐_h𝐊_h^T/√(d_k)·𝐌)𝐕_h.”
>
> ✓ verified: found word for word in the paper's own text


**Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism** (2019) \cite{shoeybi2019megatron}

What it did: Multi-head self-attention parallelization enables billion-parameter training  *(the tool's reading)*

Paper [61] exploits parallelism across multi-head self-attention computation for massive language models. This addresses engineering of Transformer attention at unprecedented scale.  *(the tool's reading)*

> “As shown in Figure […]b, for the self attention block we exploit inherent parallelism in the multihead attention operation, partitioning the GEMMs associated with key (K), query (Q), and value (V) in a column parallel fashion such that the matrix multiply corresponding to each attention head is done locally on one GPU.”
>
> ✓ verified: found word for word in the paper's own text


**Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer** (2019) \cite{raffel2019exploring}

> “Self-attention is a variant of attention […] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”
>
> ✓ verified: found word for word in the paper's own text


**Axial Attention in Multidimensional Transformers** (2019) \cite{ho2019axial}

What it did: Axial attention generalizes to multidimensional tensors with savings  *(the tool's reading)*

Paper [63] generalizes standard self-attention to axial variants aligned with multiple tensor dimensions. This saves O(N^(d-1)/d) computation over full self-attention.  *(the tool's reading)*

> “Self-attention is a neural network operation that is able to transform a sequence y_1, …, y_N into a sequence y'_1, …, y'_N, where each y'_i depends on all y_i by way of a single vectorizable computation […].”
>
> ✓ verified: found word for word in the paper's own text


**Reformer: The Efficient Transformer** (2020) \cite{kitaev2020reformer}

What it did: Locality-sensitive hashing approximates attention in O(L log L) time  *(the tool's reading)*

Paper [64] replaces dot-product attention with LSH-based approximation reducing complexity from O(L²) to O(L log L). This enables efficient attention on very long sequences.  *(the tool's reading)*

> “The standard attention used in the Transformer is the scaled dot-product attention […]. The input consists of queries and keys of dimension d_k, and values of dimension d_v. The dot products of the query with all keys are computed, scaled by √(d_k), and a softmax function is applied to obtain the weights on the values.”
>
> ✓ verified: found word for word in the paper's own text


**MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers** (2020) \cite{wang2020minilm}

What it did: Value vector interactions become explicit self-attention knowledge  *(the tool's reading)*

Paper [65] extends self-attention distillation by including scaled dot-products between values as knowledge. This adds new information beyond attention distribution matrices.  *(the tool's reading)*

> “The attention mechanism […] has been a highly successful neural network component for NLP tasks, which is also crucial for pre-trained LMs.”
>
> ✓ verified: found word for word in the paper's own text


**Longformer: The Long-Document Transformer** (2020) \cite{beltagy2020longformer}

What it did: Sparsified attention patterns scale self-attention linearly with length  *(the tool's reading)*

Paper [66] proposes sparse attention patterns replacing quadratic self-attention, scaling linearly with sequence length. This becomes drop-in replacement for standard Transformer attention.  *(the tool's reading)*

> “To address this challenge, we sparsify the full self-attention matrix according to an “attention pattern” specifying pairs of input locations attending to one another.”
>
> ✓ verified: found word for word in the paper's own text


**Synthesizer: Rethinking Self-Attention in Transformer Models** (2020) \cite{tay2020synthesizer}

What it did: Synthetic attention learns without explicit dot-product query-key  *(the tool's reading)*

Paper [67] proposes learning attention patterns directly without computing dot-product alignment matrices. This reimagines self-attention beyond content-based mechanisms.  *(the tool's reading)*

> “The fundamental role of dot product self-attention is to learn self-alignment, i.e., to determine the relative importance of a single token with respect to all other tokens in the sequence.”
>
> ✓ verified: found word for word in the paper's own text


**Language Models are Few-Shot Learners** (2020) \cite{brown2020language}

What it did: Alternating sparse and dense patterns scale attention to 175B parameters  *(the tool's reading)*

Paper [68] extends GPT-2 with alternating dense and locally-banded sparse attention patterns. This enables Transformer scaling to unprecedented 175 billion parameters.  *(the tool's reading)*

> “We use the same model and architecture as GPT-2 […], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer […].”
>
> ✓ verified: found word for word in the paper's own text


**Visual Transformers: Token-based Image Representation and Processing for Computer Vision** (2020) \cite{wu2020visual}

What it did: Spatial attention converts image features into semantic tokens  *(the tool's reading)*

Paper [69] uses content-dependent spatial attention to aggregate image features adaptively. This shifts from treating patches equally to token-based representation.  *(the tool's reading)*

> “Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context.”
>
> ✓ verified: found word for word in the paper's own text


**Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings** (2020) \cite{wang2020conv}

What it did: Low-rank structure of attention context mapping revealed theoretically  *(the tool's reading)*

Paper [70] demonstrates that Transformer self-attention context matrices are low-rank both theoretically and empirically. This justifies efficient approximations of attention.  *(the tool's reading)*

> “Here, each token's representation is updated by attending to all other tokens in the previous layer. This operation is key for retaining long-term information, giving Transformers the edge over recurrent models on long sequences. However, attending to all tokens at each layer incurs a complexity of O(n^2) with respect to sequence length.”
>
> ✓ verified: found word for word in the paper's own text


**Deformable DETR: Deformable Transformers for End-to-End Object Detection** (2020) \cite{zhu2020deformable}

What it did: Deformable attention samples sparse locations instead of all positions  *(the tool's reading)*

Paper [72] proposes deformable attention modules that attend to small sampling location subsets rather than all features. This pre-filters prominent keys while maintaining multi-head mechanism.  *(the tool's reading)*

> “Given a query element (e.g., a target word in the output sentence) and a set of key elements (e.g., source words in the input sentence), the multi-head attention module adaptively aggregates the key contents according to the attention weights that measure the compatibility of query-key pairs.”
>
> ✓ verified: found word for word in the paper's own text


**An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale** (2020) \cite{dosovitskiy2020image}

What it did: Vision Transformer applies standard self-attention directly to image patches  *(the tool's reading)*

Paper [73] applies unchanged multi-head self-attention to image patch sequences without CNN inductive biases. This demonstrates that standard Transformer attention suffices for vision.  *(the tool's reading)*

> “For each element in an input sequence 𝐳∈ℝ^N × D, we compute a weighted sum over all values 𝐯 in the sequence. The attention weights A_ij are based on the pairwise similarity between two elements of the sequence and their respective query 𝐪^i and key 𝐤^j representations.”
>
> ✓ verified: found word for word in the paper's own text


**Training data-efficient image transformers & distillation through attention** (2020) \cite{touvron2020training}

What it did: Distillation tokens enable attention-based knowledge transfer for efficiency  *(the tool's reading)*

Paper [74] frames attention as trainable associative memory and uses distillation tokens for knowledge transfer. This extends Transformer attention training beyond standard supervision.  *(the tool's reading)*

> “Multi-head Self Attention layers (MSA). The attention mechanism is based on a trainable associative memory with (key, value) vector pairs.”
>
> ✓ verified: found word for word in the paper's own text


**TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation** (2021) \cite{chen2021transunet}

What it did: Self-attention models global dependencies in medical image segmentation  *(the tool's reading)*

Paper [75] applies Transformer self-attention to tokenized image patches for segmentation combined with U-Net. This demonstrates attention capturing long-range spatial dependencies in medical imaging.  *(the tool's reading)*

> “Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms, but can result in limited localization abilities due to insufficient low-level details.”
>
> ✓ verified: found word for word in the paper's own text


**Transformer in Transformer** (2021) \cite{han2021transformer}

What it did: Nested self-attention between and within patches improves vision  *(the tool's reading)*

Paper [77] extends standard multi-head self-attention with hierarchical structure: local patch attention plus patch-level attention. This adds nested organization to vision transformers.  *(the tool's reading)*

> “The scaled dot-product attention is applied on Q,K,V: Attention(Q,K,V) = softmax(QK^T/√(d_k))V.”
>
> ✓ verified: found word for word in the paper's own text


**CvT: Introducing Convolutions to Vision Transformers** (2021) \cite{wu2021cvt}

What it did: Convolutional projections replace linear projections before self-attention  *(the tool's reading)*

Paper [78] modifies Transformer attention by using depth-wise separable convolutions instead of position-wise linear projections. This incorporates local spatial context before attention computation.  *(the tool's reading)*

> “The goal of the proposed Convolutional Projection layer is to achieve additional modeling of local spatial context, and to provide efficiency benefits by permitting the undersampling of K and V matrices.”
>
> ✓ verified: found word for word in the paper's own text


**ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training** (2021) \cite{touvron2021resmlp}

What it did: Simple linear layers replace data-dependent self-attention token mixing  *(the tool's reading)*

Paper [79] demonstrates that fixed linear layers remove query-key weighting while maintaining competitive accuracy. This questions necessity of data-dependent attention mechanisms.  *(the tool's reading)*

> “Whereas in self-attention the weights to aggregate information from other patches are data dependent through queries and keys, in ResMLPthe weights are not data dependent and only based on absolute positions of patches.”
>
> ✓ verified: found word for word in the paper's own text


**SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers** (2021) \cite{xie2021segformer}

What it did: Hierarchical local-to-global attention simplifies semantic segmentation  *(the tool's reading)*

Paper [80] combines local attention (convolution-like) with global non-local attention in hierarchical encoder. This creates efficient multi-scale self-attention design.  *(the tool's reading)*

> “Attention(Q, K, V) = Softmax(QK^𝖳/√(d_head))V.”
>
> ✓ verified: found word for word in the paper's own text


**CoAtNet: Marrying Convolution and Attention for All Data Sizes** (2021) \cite{dai2021coatnet}

What it did: Depthwise convolution and self-attention unify via relative attention  *(the tool's reading)*

Paper [81] unifies convolution and self-attention by decomposing attention into static kernels plus input-adaptive scoring. This reveals structural similarity between mechanisms.  *(the tool's reading)*

> “In comparison, self-attention allows the receptive field to be the entire spatial locations and computes the weights based on the re-normalized pairwise similarity between the pair (x_i, x_j):”
>
> ✓ verified: found word for word in the paper's own text


**PVT v2: Improved baselines with pyramid vision transformer** (2021) \cite{wang2021pvt}

What it did: Average pooling creates linear-complexity self-attention for vision  *(the tool's reading)*

Paper [82] replaces convolution with average pooling before self-attention, reducing complexity from quadratic to linear with spatial resolution. This enables efficient attention in vision transformers.  *(the tool's reading)*

> “Different from SRA […] which uses convolutions for spatial reduction, linear SRA uses average pooling to reduce the spatial dimension (i.e., h× w) to a fixed size (i.e., P× P) before the attention operation.”
>
> ✓ verified: found word for word in the paper's own text


**MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision\n Transformer** (2021) \cite{mehta2021mobilevit}

What it did: Local self-attention blocks combine CNN inductive bias with transformer  *(the tool's reading)*

Paper [83] introduces MobileViT blocks replacing local convolution processing with self-attention for non-local dependencies. This merges CNN spatial inductive bias with Transformer global modeling.  *(the tool's reading)*

> “The computational cost of self-attention in vision transformers is O(N^2d).”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis of this thread: what it enabled, what it left unsolved.]_


## 5. Attention as Learned Weighting & Gating (2014–2021)

*Attention as adaptive, learnable weights applied to features, representations, or memory structures.*

Treats attention as a general-purpose weighted pooling or gating mechanism applied to diverse structures: memory banks, feature maps, graph neighborhoods, or temporal sequences. Attention computes scalar or vector scores for each element (via neural networks or similarity functions) and aggregates via weighted averaging. This thread encompasses memory networks, spatial attention in CNNs, graph attention, and attention-based pooling for multiple instance learning.


**Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models** (2014) \cite{kiros2014unifying}

What it did: Attention conceived as joint alignment mechanism for multimodal tasks  *(the tool's reading)*

The paper envisions attention as a future mechanism that would jointly learn to align caption parts with image regions. This establishes attention as a learned weighting problem across modalities.  *(the tool's reading)*

> “We are developing an attention-based model that jointly learns to align parts of captions to images and use these alignments to determine where to attend next, thus dynamically modifying the vectors used for conditioning the decoder.”
>
> ✓ verified: found word for word in the paper's own text


**End-To-End Memory Networks** (2015) \cite{sukhbaatar2015end}

What it did: Attention operationalized as learned softmax weights over vectors  *(the tool's reading)*

Attention is now concretely implemented as softmax-weighted probability distributions computed via inner products between query vectors and embeddings. Multi-step extension enables sequential weighted selection across computational hops.  *(the tool's reading)*

> “Recently there has been a resurgence in models of computation using explicit storage and a notion of attention […]; manipulating such a storage offers an approach to both of these challenges.”
>
> ✓ verified: found word for word in the paper's own text


**Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks** (2015) \cite{ren2015faster}

> “We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with “attention” mechanisms, the RPN component tells the unified network where to look.”
>
> ✓ verified: found word for word in the paper's own text


**Hierarchical Question-Image Co-Attention for Visual Question Answering** (2016) \cite{lu2016hierarchical}

What it did: Learned weights extended to bidirectional co-attention across modalities  *(the tool's reading)*

Attention expands from unidirectional weighting to co-attention, where question-guided visual weights and image-guided question weights are jointly learned. Parallel and alternating strategies demonstrate flexible weight learning architectures.  *(the tool's reading)*

> “So far, all attention models for VQA in literature have focused on the problem of identifying “where to look” or visual attention. In this paper, we argue that the problem of identifying “which words to listen to” or question attention is equally important.”
>
> ✓ verified: found word for word in the paper's own text


**Pointer Sentinel Mixture Models** (2016) \cite{merity2016pointer}

What it did: Attention as learned score mechanism for content-based retrieval  *(the tool's reading)*

Inner product scoring followed by softmax creates learned weights determining when to retrieve from memory versus predict. Mixture model framework lets the pointer learn when to use attention-based selection.  *(the tool's reading)*

> “Models with soft attention or memory components have been proposed to help deal with this challenge, aiming to allow for the retrieval and use of relevant previous hidden states, in effect increasing hidden state capacity and providing a path for gradients not tied to timesteps.”
>
> ✓ verified: found word for word in the paper's own text


**Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer** (2016) \cite{zagoruyko2016paying}

What it did: Attention as learned spatial importance weighting for CNNs  *(the tool's reading)*

Attention now applies learned weights to spatial regions rather than sequence elements. Activation-based and gradient-based maps provide different mechanisms for learning which regions matter for decisions.  *(the tool's reading)*

> “To that end, here we consider attention as a set of spatial maps that essentially try to encode on which spatial areas of the input the network focuses most for taking its output decision (e.g., for classifying an image), where, furthermore, these maps can be defined w.r.t. various layers of the network so that they are able to capture both low-, mid-, and high-level representation information.”
>
> ✓ verified: found word for word in the paper's own text


**Modeling Relational Data with Graph Convolutional Networks** (2017) \cite{schlichtkrull2017modeling}

> “A potential way to overcome this limitation is to introduce an attention mechanism, i.e. to replace the normalization constant 1/c_i,r with data-dependent attention weights a_ij,r, where ∑_j,ra_ij,r=1.”
>
> ✓ verified: found word for word in the paper's own text


**Squeeze-and-Excitation Networks** (2017) \cite{hu2017squeeze}

What it did: Attention as learned channel-wise gating mechanism  *(the tool's reading)*

SE blocks learn to gate and reweight channels rather than spatial regions or sequence positions. This lightweight gating mechanism learns resource allocation across feature dimensions in a computationally efficient manner.  *(the tool's reading)*

> “Attention can be interpreted as a means of biasing the allocation of available computational resources towards the most informative components of a signal […].”
>
> ✓ verified: found word for word in the paper's own text


**Deep Contextualized Word Representations** (2018) \cite{peters2018deep}


**Attentive Statistics Pooling for Deep Speaker Embedding** (2018) \cite{okabe2018attentive}

What it did: Attention learns weighted statistics beyond means  *(the tool's reading)*

A small neural network module learns scalar importance scores per frame, normalized via softmax into weights. First application of learned attention weights to compute both weighted means and weighted standard deviations.  *(the tool's reading)*

> “An attention model works in conjunction with the original DNN and calculates a scalar score e_t for each frame-level feature”
>
> ✓ verified: found word for word in the paper's own text


**Attention U-Net: Learning Where to Look for the Pancreas** (2018) \cite{oktay2018attention}

> “Attention coefficients, α_i ∈ [0,1], identify salient image regions and prune feature responses to preserve only the activations relevant to the specific task as shown in Figure […].”
>
> ✓ verified: found word for word in the paper's own text


**Representation Learning on Graphs with Jumping Knowledge Networks** (2018) \cite{xu2018representation}

What it did: Attention learns to weight importance across multiple temporal layers  *(the tool's reading)*

An LSTM-attention layer aggregator learns node-specific scores representing each layer's feature importance. This extends learned weighting from single representations to learned selection across multiple learned representations.  *(the tool's reading)*

> “An attention mechanism identifies the most useful neighborhood ranges for each node v by computing an attention score s(l) v for each layer l”
>
> ✓ verified: found word for word in the paper's own text


**NAIS: Neural Attentive Item Similarity Model for Recommendation** (2018) \cite{he2018nais}

What it did: Attention learns individualized item importance in user histories  *(the tool's reading)*

Neural networks learn to assign different weights to different items in a user's interaction history. Items now contribute unequally through learned attention rather than uniform aggregation.  *(the tool's reading)*

> “The original idea of attention is that different parts of a model can contribute (i.e., attend) differently for the final prediction […].”
>
> ✓ verified: found word for word in the paper's own text


**Session-Based Recommendation with Graph Neural Networks** (2018) \cite{wu2018session}

What it did: Attention learns weighted composition of session graph embeddings  *(the tool's reading)*

Soft-attention computes learned weights over item node vectors in session graphs. Weighted aggregation combines individual item representations into global session preference through learned importance scores.  *(the tool's reading)*

> “Consider information in these embedding may have different levels of priority, we further adopt the soft-attention mechanism to better represent the global session preference:”
>
> ✓ verified: found word for word in the paper's own text


**A survey of the recent architectures of deep convolutional neural networks** (2019) \cite{khan2019survey}

> “Designing of new blocks is one of the growing areas of research in CNN, where generic blocks are used to assign attention to spatial and feature-map (channel) information (Wang et al. 2017a; Roy et al. 2018; Woo et al. 2018).”
>
> ✓ verified: found word for word in the paper's own text


**CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features** (2019) \cite{yun2019cutmix}

> “They have proved to be effective for guiding the model to attend on less discriminative parts of objects ( leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities.”
>
> ✓ verified: found word for word in the paper's own text


**KGAT** (2019) \cite{wang2019kgat}

What it did: Attention learns neighbor and relation weights in knowledge graphs  *(the tool's reading)*

KGAT employs neural attention to learn different weights for different neighbors and their relations in knowledge graph propagation. This extends learned weighting to relational graph structures.  *(the tool's reading)*

> “employs the neural attention mechanism […] to learn the weight of each neighbor during a propagation, such that the attention weights of cascaded propagations can reveal the importance of a high-order connectivity.”
>
> ✓ verified: found word for word in the paper's own text


**VisualBERT: A Simple and Performant Baseline for Vision and Language** (2019) \cite{li2019visualbert}

> “VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention.”
>
> ✓ verified: found word for word in the paper's own text


**Knowledge Distillation: A Survey** (2020) \cite{gou2020knowledge}

> “To be specific, […] derived an “attention map" from the original feature maps to express knowledge.”
>
> ✓ verified: found word for word in the paper's own text


**Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision** (2021) \cite{jia2021scaling}

> “Recently more advanced models emerge with cross-modal attention layers […] and show superior performance in image-text matching tasks. However, they are orders of magnitudes slower and hence impractical for image-text retrieval systems in the real world.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis of this thread: what it enabled, what it left unsolved.]_


## Where threads crossed

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Soft Alignment in Sequence-to-Sequence Models** → **Self-Attention & Transformers** (2017): Scaled dot-product unifies alignment
- **Recurrent Glimpse-Based Visual Attention** → **Self-Attention & Transformers** (2017): Vision enters Transformer era
- **Attention as Learned Weighting & Gating** → **Self-Attention & Transformers** (2018): Gating integrated into self-attention
- **Soft Alignment in Sequence-to-Sequence Models** → **Attention as Learned Weighting & Gating** (2015): Weights generalize beyond alignment
- **Self-Attention & Transformers** → **Saliency & Biological Vision** (2020): Interpretability meets neuroscience

_[Your paragraph tying these crossings into a narrative.]_