# Self-Attention & Transformers

> One thread of 5 from the "attention" map, covering the 34 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Soft Alignment in Sequence-to-Sequence Models, Recurrent Glimpse-Based Visual Attention, Attention as Learned Weighting & Gating, Saliency & Biological Vision.

Inaugurated by the Transformer, this thread redefines attention as scaled dot-product self-attention over positions within a single sequence (or modality), computing query-key-value relationships. Eliminates recurrence and decoder-encoder asymmetry, enabling fully parallel training and inference. Rapidly becomes the foundational architecture for language models, vision models, and multimodal systems, with subsequent work optimizing efficiency, interpretability, and structural variants.


**A Structured Self-attentive Sentence Embedding** (2017) \cite{lin2017structured}

What it did: Self-attention computes weights via MLP without external inputs  *(the tool's reading)*

Paper [30] introduces self-attention as an MLP-based weighting mechanism operating on LSTM hidden states, enabling attention purely from internal representations. This establishes attention operating without separate key-value inputs.  *(the tool's reading)*

> “Our aim is to encode a variable length sentence into a fixed size embedding. We achieve that by choosing a linear combination of the n LSTM hidden vectors in H. Computing the linear combination requires the self-attention mechanism. The attention mechanism takes the whole LSTM hidden states H as input, and outputs a vector of weights 𝐚:”
>
> ✓ verified: found word for word in the paper's own text


**Attention Is All You Need** (2017) \cite{vaswani2017attention}

What it did: Scaled dot-product attention replaces RNNs entirely in Transformers  *(the tool's reading)*

Paper [34] defines attention as query-key-value mapping with scaled dot-product computation and removes RNNs/convolutions completely. The Transformer becomes the first model relying entirely on self-attention for representation.  *(the tool's reading)*

> “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.”
>
> ✓ verified: found word for word in the paper's own text


**Graph Attention Networks** (2017) \cite{velivckovic2017graph}

What it did: Masked self-attention extends to graph-structured data  *(the tool's reading)*

Paper [38] applies masked self-attention to graph neighborhoods instead of sequences, computing learned importance weights. This generalizes Transformer attention beyond sequential data.  *(the tool's reading)*

> “One of the benefits of attention mechanisms is that they allow for dealing with variable sized inputs, focusing on the most relevant parts of the input to make decisions.”
>
> ✓ verified: found word for word in the paper's own text


**Image Transformer** (2018) \cite{parmar2018image}

What it did: Self-attention scales to image generation via sequence modeling  *(the tool's reading)*

Paper [41] applies scaled dot-product self-attention to image generation, treating pixels as sequence positions. This extends Transformer attention to visual domains with tractable likelihood.  *(the tool's reading)*

> “Each self-attention layer computes a d-dimensional representation for each position, that is, each channel of each pixel. To recompute the representation for a given position, it first compares the position's current representation to other positions' representations, obtaining an attention distribution over the other positions. This distribution is then used to weight the contribution of the other positions' representations to the next representation for the position at hand.”
>
> ✓ verified: found word for word in the paper's own text


**Self-Attention with Relative Position Representations** (2018) \cite{shaw2018self}

What it did: Relative position representations augment scaled dot-product attention  *(the tool's reading)*

Paper [42] extends Transformer self-attention by incorporating relative position representations alongside dot-product computation. This adds structural awareness to standard attention weights.  *(the tool's reading)*

> “Each attention head operates on an input sequence, x = (x_1, …, x_n) of n elements where x_i ∈ℝ^d_x, and computes a new sequence z = (z_1, …, z_n) of the same length where z_i ∈ℝ^d_z.”
>
> ✓ verified: found word for word in the paper's own text


**QANet: Combining Local Convolution with Global Self-Attention for\n Reading Comprehension** (2018) \cite{yu2018qanet}

What it did: Self-attention replaces max pooling in reading comprehension  *(the tool's reading)*

Paper [45] combines convolutions exclusively with self-attention for reading comprehension, using dot-product query-key similarity. This demonstrates self-attention as pooling replacement across modalities.  *(the tool's reading)*

> “For the self-attention-layer, we adopt the multi-head attention mechanism defined in […] which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product.”
>
> ✓ verified: found word for word in the paper's own text


**Self-Attention Generative Adversarial Networks** (2018) \cite{zhang2018self}

What it did: Self-attention enters GANs for spatial image feature weighting  *(the tool's reading)*

Paper [46] introduces self-attention into GAN generators and discriminators, computing spatial feature responses as weighted sums. This marks first self-attention use in adversarial training.  *(the tool's reading)*

> “The self-attention module calculates response at a position as a weighted sum of the features at all positions, where the weights – or attention vectors – are calculated with only a small computational cost.”
>
> ✓ verified: found word for word in the paper's own text


**OCNet: Object Context Network for Scene Parsing** (2018) \cite{yuan2018ocnet}

What it did: Sparse self-attention approximates dense relation matrices efficiently  *(the tool's reading)*

Paper [48] proposes interlaced sparse self-attention to approximate dense position-weighted aggregation matrices. This reduces quadratic computational cost while preserving self-attention semantics.  *(the tool's reading)*

> “The self-attention scheme […] calculates the context at one position as a aggregation of all positions in a sentence (at the encoder stage).”
>
> ✓ verified: found word for word in the paper's own text


**Deep Audio-Visual Speech Recognition** (2018) \cite{afouras2018deep}

What it did: Dual self-attention extends Transformer to audio-visual modeling  *(the tool's reading)*

Paper [49] extends Transformer self-attention to dual encoder-decoder setup for multimodal speech recognition. This maintains standard scaled dot-product mechanism across new modalities.  *(the tool's reading)*

> “the architecture incorporates a novel dual attention mechanism that can operate over visual input only, audio input only, or both.”
>
> ✓ verified: found word for word in the paper's own text


**Generating Long Sequences with Sparse Transformers** (2019) \cite{child2019generating}

What it did: Sparse factorizations reduce attention complexity from O(n²) to O(n√n)  *(the tool's reading)*

Paper [54] introduces sparse factorizations of full attention matrices via connectivity patterns. This dramatically reduces quadratic scaling without sacrificing performance on long sequences.  *(the tool's reading)*

> “A self-attention layer maps a matrix of input embeddings X to an output matrix and is parameterized by a connectivity pattern S = {S_1,..., S_n}, where S_i denotes the set of indices of the input vectors to which the ith output vector attends.”
>
> ✓ verified: found word for word in the paper's own text


**Unified Language Model Pre-training for Natural Language Understanding\n and Generation** (2019) \cite{dong2019unified}

What it did: Masked self-attention enables unified language model pre-training  *(the tool's reading)*

Paper [55] uses Transformer self-attention with specific masking patterns to control context conditioning. This applies standard mechanisms to unified bidirectional pre-training.  *(the tool's reading)*

> “In each Transformer block, multiple self-attention heads are used to aggregate the output vectors of the previous layer.”
>
> ✓ verified: found word for word in the paper's own text


**Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned** (2019) \cite{voita2019analyzing}

What it did: Individual attention heads specialize; pruning them preserves performance  *(the tool's reading)*

Paper [58] analyzes multi-head self-attention confidence scores and introduces head pruning via stochastic gates. This reveals that individual heads specialize without affecting overall performance.  *(the tool's reading)*

> “The multi-head attention mechanism obtains h (i.e. one per head) different representations of (Q, K, V), computes scaled dot-product attention for each representation, concatenates the results, and projects the concatenation through a feed-forward layer.”
>
> ✓ verified: found word for word in the paper's own text


**Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting** (2019) \cite{li2019enhancing}

What it did: Convolutional self-attention incorporates local context into dot-product  *(the tool's reading)*

Paper [59] modifies Transformer attention for time series by using causal convolutions to produce queries and keys. This replaces point-wise dot-product with locally-contextual attention matching.  *(the tool's reading)*

> “After these linear projections, the scaled dot-product attention computes a sequence of vector outputs: 𝐎_h = Attention(𝐐_h, 𝐊_h, 𝐕_h)= softmax(𝐐_h𝐊_h^T/√(d_k)·𝐌)𝐕_h.”
>
> ✓ verified: found word for word in the paper's own text


**Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism** (2019) \cite{shoeybi2019megatron}

What it did: Multi-head self-attention parallelization enables billion-parameter training  *(the tool's reading)*

Paper [61] exploits parallelism across multi-head self-attention computation for massive language models. This addresses engineering of Transformer attention at unprecedented scale.  *(the tool's reading)*

> “As shown in Figure […]b, for the self attention block we exploit inherent parallelism in the multihead attention operation, partitioning the GEMMs associated with key (K), query (Q), and value (V) in a column parallel fashion such that the matrix multiply corresponding to each attention head is done locally on one GPU.”
>
> ✓ verified: found word for word in the paper's own text


**Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer** (2019) \cite{raffel2019exploring}

> “Self-attention is a variant of attention […] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”
>
> ✓ verified: found word for word in the paper's own text


**Axial Attention in Multidimensional Transformers** (2019) \cite{ho2019axial}

What it did: Axial attention generalizes to multidimensional tensors with savings  *(the tool's reading)*

Paper [63] generalizes standard self-attention to axial variants aligned with multiple tensor dimensions. This saves O(N^(d-1)/d) computation over full self-attention.  *(the tool's reading)*

> “Self-attention is a neural network operation that is able to transform a sequence y_1, …, y_N into a sequence y'_1, …, y'_N, where each y'_i depends on all y_i by way of a single vectorizable computation […].”
>
> ✓ verified: found word for word in the paper's own text


**Reformer: The Efficient Transformer** (2020) \cite{kitaev2020reformer}

What it did: Locality-sensitive hashing approximates attention in O(L log L) time  *(the tool's reading)*

Paper [64] replaces dot-product attention with LSH-based approximation reducing complexity from O(L²) to O(L log L). This enables efficient attention on very long sequences.  *(the tool's reading)*

> “The standard attention used in the Transformer is the scaled dot-product attention […]. The input consists of queries and keys of dimension d_k, and values of dimension d_v. The dot products of the query with all keys are computed, scaled by √(d_k), and a softmax function is applied to obtain the weights on the values.”
>
> ✓ verified: found word for word in the paper's own text


**MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers** (2020) \cite{wang2020minilm}

What it did: Value vector interactions become explicit self-attention knowledge  *(the tool's reading)*

Paper [65] extends self-attention distillation by including scaled dot-products between values as knowledge. This adds new information beyond attention distribution matrices.  *(the tool's reading)*

> “The attention mechanism […] has been a highly successful neural network component for NLP tasks, which is also crucial for pre-trained LMs.”
>
> ✓ verified: found word for word in the paper's own text


**Longformer: The Long-Document Transformer** (2020) \cite{beltagy2020longformer}

What it did: Sparsified attention patterns scale self-attention linearly with length  *(the tool's reading)*

Paper [66] proposes sparse attention patterns replacing quadratic self-attention, scaling linearly with sequence length. This becomes drop-in replacement for standard Transformer attention.  *(the tool's reading)*

> “To address this challenge, we sparsify the full self-attention matrix according to an “attention pattern” specifying pairs of input locations attending to one another.”
>
> ✓ verified: found word for word in the paper's own text


**Synthesizer: Rethinking Self-Attention in Transformer Models** (2020) \cite{tay2020synthesizer}

What it did: Synthetic attention learns without explicit dot-product query-key  *(the tool's reading)*

Paper [67] proposes learning attention patterns directly without computing dot-product alignment matrices. This reimagines self-attention beyond content-based mechanisms.  *(the tool's reading)*

> “The fundamental role of dot product self-attention is to learn self-alignment, i.e., to determine the relative importance of a single token with respect to all other tokens in the sequence.”
>
> ✓ verified: found word for word in the paper's own text


**Language Models are Few-Shot Learners** (2020) \cite{brown2020language}

What it did: Alternating sparse and dense patterns scale attention to 175B parameters  *(the tool's reading)*

Paper [68] extends GPT-2 with alternating dense and locally-banded sparse attention patterns. This enables Transformer scaling to unprecedented 175 billion parameters.  *(the tool's reading)*

> “We use the same model and architecture as GPT-2 […], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer […].”
>
> ✓ verified: found word for word in the paper's own text


**Visual Transformers: Token-based Image Representation and Processing for Computer Vision** (2020) \cite{wu2020visual}

What it did: Spatial attention converts image features into semantic tokens  *(the tool's reading)*

Paper [69] uses content-dependent spatial attention to aggregate image features adaptively. This shifts from treating patches equally to token-based representation.  *(the tool's reading)*

> “Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context.”
>
> ✓ verified: found word for word in the paper's own text


**Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings** (2020) \cite{wang2020conv}

What it did: Low-rank structure of attention context mapping revealed theoretically  *(the tool's reading)*

Paper [70] demonstrates that Transformer self-attention context matrices are low-rank both theoretically and empirically. This justifies efficient approximations of attention.  *(the tool's reading)*

> “Here, each token's representation is updated by attending to all other tokens in the previous layer. This operation is key for retaining long-term information, giving Transformers the edge over recurrent models on long sequences. However, attending to all tokens at each layer incurs a complexity of O(n^2) with respect to sequence length.”
>
> ✓ verified: found word for word in the paper's own text


**Deformable DETR: Deformable Transformers for End-to-End Object Detection** (2020) \cite{zhu2020deformable}

What it did: Deformable attention samples sparse locations instead of all positions  *(the tool's reading)*

Paper [72] proposes deformable attention modules that attend to small sampling location subsets rather than all features. This pre-filters prominent keys while maintaining multi-head mechanism.  *(the tool's reading)*

> “Given a query element (e.g., a target word in the output sentence) and a set of key elements (e.g., source words in the input sentence), the multi-head attention module adaptively aggregates the key contents according to the attention weights that measure the compatibility of query-key pairs.”
>
> ✓ verified: found word for word in the paper's own text


**An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale** (2020) \cite{dosovitskiy2020image}

What it did: Vision Transformer applies standard self-attention directly to image patches  *(the tool's reading)*

Paper [73] applies unchanged multi-head self-attention to image patch sequences without CNN inductive biases. This demonstrates that standard Transformer attention suffices for vision.  *(the tool's reading)*

> “For each element in an input sequence 𝐳∈ℝ^N × D, we compute a weighted sum over all values 𝐯 in the sequence. The attention weights A_ij are based on the pairwise similarity between two elements of the sequence and their respective query 𝐪^i and key 𝐤^j representations.”
>
> ✓ verified: found word for word in the paper's own text


**Training data-efficient image transformers & distillation through attention** (2020) \cite{touvron2020training}

What it did: Distillation tokens enable attention-based knowledge transfer for efficiency  *(the tool's reading)*

Paper [74] frames attention as trainable associative memory and uses distillation tokens for knowledge transfer. This extends Transformer attention training beyond standard supervision.  *(the tool's reading)*

> “Multi-head Self Attention layers (MSA). The attention mechanism is based on a trainable associative memory with (key, value) vector pairs.”
>
> ✓ verified: found word for word in the paper's own text


**TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation** (2021) \cite{chen2021transunet}

What it did: Self-attention models global dependencies in medical image segmentation  *(the tool's reading)*

Paper [75] applies Transformer self-attention to tokenized image patches for segmentation combined with U-Net. This demonstrates attention capturing long-range spatial dependencies in medical imaging.  *(the tool's reading)*

> “Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms, but can result in limited localization abilities due to insufficient low-level details.”
>
> ✓ verified: found word for word in the paper's own text


**Transformer in Transformer** (2021) \cite{han2021transformer}

What it did: Nested self-attention between and within patches improves vision  *(the tool's reading)*

Paper [77] extends standard multi-head self-attention with hierarchical structure: local patch attention plus patch-level attention. This adds nested organization to vision transformers.  *(the tool's reading)*

> “The scaled dot-product attention is applied on Q,K,V: Attention(Q,K,V) = softmax(QK^T/√(d_k))V.”
>
> ✓ verified: found word for word in the paper's own text


**CvT: Introducing Convolutions to Vision Transformers** (2021) \cite{wu2021cvt}

What it did: Convolutional projections replace linear projections before self-attention  *(the tool's reading)*

Paper [78] modifies Transformer attention by using depth-wise separable convolutions instead of position-wise linear projections. This incorporates local spatial context before attention computation.  *(the tool's reading)*

> “The goal of the proposed Convolutional Projection layer is to achieve additional modeling of local spatial context, and to provide efficiency benefits by permitting the undersampling of K and V matrices.”
>
> ✓ verified: found word for word in the paper's own text


**ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training** (2021) \cite{touvron2021resmlp}

What it did: Simple linear layers replace data-dependent self-attention token mixing  *(the tool's reading)*

Paper [79] demonstrates that fixed linear layers remove query-key weighting while maintaining competitive accuracy. This questions necessity of data-dependent attention mechanisms.  *(the tool's reading)*

> “Whereas in self-attention the weights to aggregate information from other patches are data dependent through queries and keys, in ResMLPthe weights are not data dependent and only based on absolute positions of patches.”
>
> ✓ verified: found word for word in the paper's own text


**SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers** (2021) \cite{xie2021segformer}

What it did: Hierarchical local-to-global attention simplifies semantic segmentation  *(the tool's reading)*

Paper [80] combines local attention (convolution-like) with global non-local attention in hierarchical encoder. This creates efficient multi-scale self-attention design.  *(the tool's reading)*

> “Attention(Q, K, V) = Softmax(QK^𝖳/√(d_head))V.”
>
> ✓ verified: found word for word in the paper's own text


**CoAtNet: Marrying Convolution and Attention for All Data Sizes** (2021) \cite{dai2021coatnet}

What it did: Depthwise convolution and self-attention unify via relative attention  *(the tool's reading)*

Paper [81] unifies convolution and self-attention by decomposing attention into static kernels plus input-adaptive scoring. This reveals structural similarity between mechanisms.  *(the tool's reading)*

> “In comparison, self-attention allows the receptive field to be the entire spatial locations and computes the weights based on the re-normalized pairwise similarity between the pair (x_i, x_j):”
>
> ✓ verified: found word for word in the paper's own text


**PVT v2: Improved baselines with pyramid vision transformer** (2021) \cite{wang2021pvt}

What it did: Average pooling creates linear-complexity self-attention for vision  *(the tool's reading)*

Paper [82] replaces convolution with average pooling before self-attention, reducing complexity from quadratic to linear with spatial resolution. This enables efficient attention in vision transformers.  *(the tool's reading)*

> “Different from SRA […] which uses convolutions for spatial reduction, linear SRA uses average pooling to reduce the spatial dimension (i.e., h× w) to a fixed size (i.e., P× P) before the attention operation.”
>
> ✓ verified: found word for word in the paper's own text


**MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision\n Transformer** (2021) \cite{mehta2021mobilevit}

What it did: Local self-attention blocks combine CNN inductive bias with transformer  *(the tool's reading)*

Paper [83] introduces MobileViT blocks replacing local convolution processing with self-attention for non-local dependencies. This merges CNN spatial inductive bias with Transformer global modeling.  *(the tool's reading)*

> “The computational cost of self-attention in vision transformers is O(N^2d).”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Soft Alignment in Sequence-to-Sequence Models** → **Self-Attention & Transformers** (2017): Scaled dot-product unifies alignment
- **Recurrent Glimpse-Based Visual Attention** → **Self-Attention & Transformers** (2017): Vision enters Transformer era
- **Attention as Learned Weighting & Gating** → **Self-Attention & Transformers** (2018): Gating integrated into self-attention
- **Self-Attention & Transformers** → **Saliency & Biological Vision** (2020): Interpretability meets neuroscience