# Attention as Learned Weighting & Gating

> One thread of 5 from the "attention" map, covering the 20 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Self-Attention & Transformers, Soft Alignment in Sequence-to-Sequence Models.

Treats attention as a general-purpose weighted pooling or gating mechanism applied to diverse structures: memory banks, feature maps, graph neighborhoods, or temporal sequences. Attention computes scalar or vector scores for each element (via neural networks or similarity functions) and aggregates via weighted averaging. This thread encompasses memory networks, spatial attention in CNNs, graph attention, and attention-based pooling for multiple instance learning.


**Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models** (2014) \cite{kiros2014unifying}

What it did: Attention conceived as joint alignment mechanism for multimodal tasks  *(the tool's reading)*

The paper envisions attention as a future mechanism that would jointly learn to align caption parts with image regions. This establishes attention as a learned weighting problem across modalities.  *(the tool's reading)*

> “We are developing an attention-based model that jointly learns to align parts of captions to images and use these alignments to determine where to attend next, thus dynamically modifying the vectors used for conditioning the decoder.”
>
> ✓ verified: found word for word in the paper's own text


**End-To-End Memory Networks** (2015) \cite{sukhbaatar2015end}

What it did: Attention operationalized as learned softmax weights over vectors  *(the tool's reading)*

Attention is now concretely implemented as softmax-weighted probability distributions computed via inner products between query vectors and embeddings. Multi-step extension enables sequential weighted selection across computational hops.  *(the tool's reading)*

> “Recently there has been a resurgence in models of computation using explicit storage and a notion of attention […]; manipulating such a storage offers an approach to both of these challenges.”
>
> ✓ verified: found word for word in the paper's own text


**Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks** (2015) \cite{ren2015faster}

> “We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with “attention” mechanisms, the RPN component tells the unified network where to look.”
>
> ✓ verified: found word for word in the paper's own text


**Hierarchical Question-Image Co-Attention for Visual Question Answering** (2016) \cite{lu2016hierarchical}

What it did: Learned weights extended to bidirectional co-attention across modalities  *(the tool's reading)*

Attention expands from unidirectional weighting to co-attention, where question-guided visual weights and image-guided question weights are jointly learned. Parallel and alternating strategies demonstrate flexible weight learning architectures.  *(the tool's reading)*

> “So far, all attention models for VQA in literature have focused on the problem of identifying “where to look” or visual attention. In this paper, we argue that the problem of identifying “which words to listen to” or question attention is equally important.”
>
> ✓ verified: found word for word in the paper's own text


**Pointer Sentinel Mixture Models** (2016) \cite{merity2016pointer}

What it did: Attention as learned score mechanism for content-based retrieval  *(the tool's reading)*

Inner product scoring followed by softmax creates learned weights determining when to retrieve from memory versus predict. Mixture model framework lets the pointer learn when to use attention-based selection.  *(the tool's reading)*

> “Models with soft attention or memory components have been proposed to help deal with this challenge, aiming to allow for the retrieval and use of relevant previous hidden states, in effect increasing hidden state capacity and providing a path for gradients not tied to timesteps.”
>
> ✓ verified: found word for word in the paper's own text


**Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer** (2016) \cite{zagoruyko2016paying}

What it did: Attention as learned spatial importance weighting for CNNs  *(the tool's reading)*

Attention now applies learned weights to spatial regions rather than sequence elements. Activation-based and gradient-based maps provide different mechanisms for learning which regions matter for decisions.  *(the tool's reading)*

> “To that end, here we consider attention as a set of spatial maps that essentially try to encode on which spatial areas of the input the network focuses most for taking its output decision (e.g., for classifying an image), where, furthermore, these maps can be defined w.r.t. various layers of the network so that they are able to capture both low-, mid-, and high-level representation information.”
>
> ✓ verified: found word for word in the paper's own text


**Modeling Relational Data with Graph Convolutional Networks** (2017) \cite{schlichtkrull2017modeling}

> “A potential way to overcome this limitation is to introduce an attention mechanism, i.e. to replace the normalization constant 1/c_i,r with data-dependent attention weights a_ij,r, where ∑_j,ra_ij,r=1.”
>
> ✓ verified: found word for word in the paper's own text


**Squeeze-and-Excitation Networks** (2017) \cite{hu2017squeeze}

What it did: Attention as learned channel-wise gating mechanism  *(the tool's reading)*

SE blocks learn to gate and reweight channels rather than spatial regions or sequence positions. This lightweight gating mechanism learns resource allocation across feature dimensions in a computationally efficient manner.  *(the tool's reading)*

> “Attention can be interpreted as a means of biasing the allocation of available computational resources towards the most informative components of a signal […].”
>
> ✓ verified: found word for word in the paper's own text


**Deep Contextualized Word Representations** (2018) \cite{peters2018deep}


**Attentive Statistics Pooling for Deep Speaker Embedding** (2018) \cite{okabe2018attentive}

What it did: Attention learns weighted statistics beyond means  *(the tool's reading)*

A small neural network module learns scalar importance scores per frame, normalized via softmax into weights. First application of learned attention weights to compute both weighted means and weighted standard deviations.  *(the tool's reading)*

> “An attention model works in conjunction with the original DNN and calculates a scalar score e_t for each frame-level feature”
>
> ✓ verified: found word for word in the paper's own text


**Attention U-Net: Learning Where to Look for the Pancreas** (2018) \cite{oktay2018attention}

> “Attention coefficients, α_i ∈ [0,1], identify salient image regions and prune feature responses to preserve only the activations relevant to the specific task as shown in Figure […].”
>
> ✓ verified: found word for word in the paper's own text


**Representation Learning on Graphs with Jumping Knowledge Networks** (2018) \cite{xu2018representation}

What it did: Attention learns to weight importance across multiple temporal layers  *(the tool's reading)*

An LSTM-attention layer aggregator learns node-specific scores representing each layer's feature importance. This extends learned weighting from single representations to learned selection across multiple learned representations.  *(the tool's reading)*

> “An attention mechanism identifies the most useful neighborhood ranges for each node v by computing an attention score s(l) v for each layer l”
>
> ✓ verified: found word for word in the paper's own text


**NAIS: Neural Attentive Item Similarity Model for Recommendation** (2018) \cite{he2018nais}

What it did: Attention learns individualized item importance in user histories  *(the tool's reading)*

Neural networks learn to assign different weights to different items in a user's interaction history. Items now contribute unequally through learned attention rather than uniform aggregation.  *(the tool's reading)*

> “The original idea of attention is that different parts of a model can contribute (i.e., attend) differently for the final prediction […].”
>
> ✓ verified: found word for word in the paper's own text


**Session-Based Recommendation with Graph Neural Networks** (2018) \cite{wu2018session}

What it did: Attention learns weighted composition of session graph embeddings  *(the tool's reading)*

Soft-attention computes learned weights over item node vectors in session graphs. Weighted aggregation combines individual item representations into global session preference through learned importance scores.  *(the tool's reading)*

> “Consider information in these embedding may have different levels of priority, we further adopt the soft-attention mechanism to better represent the global session preference:”
>
> ✓ verified: found word for word in the paper's own text


**A survey of the recent architectures of deep convolutional neural networks** (2019) \cite{khan2019survey}

> “Designing of new blocks is one of the growing areas of research in CNN, where generic blocks are used to assign attention to spatial and feature-map (channel) information (Wang et al. 2017a; Roy et al. 2018; Woo et al. 2018).”
>
> ✓ verified: found word for word in the paper's own text


**CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features** (2019) \cite{yun2019cutmix}

> “They have proved to be effective for guiding the model to attend on less discriminative parts of objects ( leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities.”
>
> ✓ verified: found word for word in the paper's own text


**KGAT** (2019) \cite{wang2019kgat}

What it did: Attention learns neighbor and relation weights in knowledge graphs  *(the tool's reading)*

KGAT employs neural attention to learn different weights for different neighbors and their relations in knowledge graph propagation. This extends learned weighting to relational graph structures.  *(the tool's reading)*

> “employs the neural attention mechanism […] to learn the weight of each neighbor during a propagation, such that the attention weights of cascaded propagations can reveal the importance of a high-order connectivity.”
>
> ✓ verified: found word for word in the paper's own text


**VisualBERT: A Simple and Performant Baseline for Vision and Language** (2019) \cite{li2019visualbert}

> “VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention.”
>
> ✓ verified: found word for word in the paper's own text


**Knowledge Distillation: A Survey** (2020) \cite{gou2020knowledge}

> “To be specific, […] derived an “attention map" from the original feature maps to express knowledge.”
>
> ✓ verified: found word for word in the paper's own text


**Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision** (2021) \cite{jia2021scaling}

> “Recently more advanced models emerge with cross-modal attention layers […] and show superior performance in image-text matching tasks. However, they are orders of magnitudes slower and hence impractical for image-text retrieval systems in the real world.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Attention as Learned Weighting & Gating** → **Self-Attention & Transformers** (2018): Gating integrated into self-attention
- **Soft Alignment in Sequence-to-Sequence Models** → **Attention as Learned Weighting & Gating** (2015): Weights generalize beyond alignment