Attention as Learned Weighting & Gating

The paper introduced an LSTM-based encoder to attend to sequential word information when creating joint image-text embeddings.

“For the encoder, we learn a joint image-sentence embedding where sentences are encoded using long short-term memory (LSTM) recurrent neural networks”◌ not checked against the paper’s text as it now stands

The paper used attention-like mechanisms through LSTM memory cells with gating units to selectively read and process information from sequences.

“LSTM memory cells are surrounded by gating units for the purpose of reading, writing and reseting information.”◌ not checked against the paper’s text as it now stands

The paper employed a structure-content neural language model that conditions on both forward structural context and image embeddings to guide word generation.

“our goal is to model the distribution P(w_n = i | w_1:n-1, t_n:n+k, u) from previous word context w_1:n-1 and forward structure context t_n:n+k, where k is the forward context size”◌ not checked against the paper’s text as it now stands

This paper introduced a neural network with a recurrent attention model over external memory that can be trained end-to-end.

“We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network […] but unlike the model in that work, it is trained end-to-end”◌ not checked against the paper’s text as it now stands

The paper uses softmax attention weights to compute a continuous probability distribution over memory elements for retrieval.

“In the embedding space, we compute the match between u and each memory m_i by taking the inner product followed by a softmax: p_i = Softmax(u^T m_i)”◌ not checked against the paper’s text as it now stands

The model performs multiple computational hops over memory with attention, and empirical results show that more hops improve performance.

“Importantly, more computational hops give improved performance. We give examples of the hops performed (via the values of eq. […]) over some illustrative examples in Fig. […]”◌ not checked against the paper’s text as it now stands

This paper describes how the RPN component of their unified network serves an attention-like function by directing the detection network to focus on proposed regions.

This paper characterizes the RPN module as functioning like an attention mechanism within the Faster R-CNN architecture.

“Using the recently popular terminology of neural networks with `attention' […] mechanisms, the RPN module tells the Fast R-CNN module where to look.”◌ not checked against the paper’s text as it now stands

This paper introduced a co-attention mechanism that jointly models both visual attention and question attention for VQA, whereas previous work only modeled visual attention alone.

“We propose a novel co-attention mechanism for VQA that jointly performs question-guided visual attention and image-guided question attention.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that question attention is equally important as visual attention by showing that attending to relevant words improves robustness to linguistic variations.

This paper implemented attention mechanisms at three hierarchical levels of question representation: word, phrase, and question level.

“We propose a hierarchical architecture to represent the question, and consequently construct image-question co-attention maps at 3 different levels: word level, phrase level and question level.”◌ not checked against the paper’s text as it now stands

This paper compared two strategies for co-attention - parallel and alternating - and found that alternating co-attention achieved better results on the VQA dataset.

This paper introduced a pointer sentinel mixture architecture that uses attention to decide when to employ either a pointer network or a standard softmax classifier for language modeling.

This paper modified pointer networks by adding a sentinel vector that computes attention scores, allowing the pointer component itself to determine when to use the softmax vocabulary through this gating mechanism.

This paper demonstrated that integrating the gating function directly into the pointer network's attention computation allows the model to consider both the RNN hidden state and the contents of the pointer window when making decisions.

The paper defines attention for convolutional neural networks as spatial maps that encode which areas of the input the network focuses on for making output decisions.

“here we consider attention as a set of spatial maps that essentially try to encode on which spatial areas of the input the network focuses most for taking its output decision (e.g., for classifying an image)”◌ not checked against the paper’s text as it now stands

The paper proposes activation-based and gradient-based attention transfer methods to improve student network performance by making it mimic teacher network attention maps.

“we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures”◌ not checked against the paper’s text as it now stands

The paper demonstrates that attention-based transfer provides better improvements than transferring full activation tensors.

“we find that AT, as well as full-activation transfer, greatly speeds up convergence, but AT gives much better final accuracy improvement than full-activation transfer”◌ not checked against the paper’s text as it now stands

The paper identifies attention mechanisms as a promising direction to improve R-GCN performance on datasets with high-degree hub nodes.

“A potential way to overcome this limitation is to introduce an attention mechanism, i.e. to replace the normalization constant 1/c_i,r with data-dependent attention weights a_ij,r, where ∑_j,ra_ij,r=1. We expect this to be a promising avenue for future research.”◌ not checked against the paper’s text as it now stands

This paper introduced the Squeeze-and-Excitation block as a channel attention mechanism that adaptively recalibrates channel-wise feature responses by explicitly modeling interdependencies between channels.

The paper designed SE blocks to perform feature recalibration through a gating mechanism that allows networks to selectively emphasize informative features and suppress less useful ones using global information.

“we propose a mechanism that allows the network to perform feature recalibration, through which it can learn to use global information to selectively emphasise informative features and suppress less useful ones.”◌ not checked against the paper’s text as it now stands

This work characterized attention as a means of biasing computational resources towards the most informative components, with SE blocks providing a lightweight channel-focused attention mechanism.

The paper introduced ELMo representations that use deep internal layers of a bidirectional language model to capture different types of linguistic information that can be selectively weighted for different tasks.

The paper demonstrated that attention-like mechanisms for combining layers allow models to dynamically select which types of syntactic and semantic supervision are most useful for each downstream task.

“Simultaneously exposing all of these signals is highly beneficial, allowing the learned models select the types of semi-supervision that are most useful for each end task.”◌ not checked against the paper’s text as it now stands

The paper showed that different layers of the bidirectional language model require attention-like weighting, with lower layers capturing syntax and higher layers capturing semantics, enabling task-specific combinations.

“Using intrinsic evaluations, we show that the higher-level LSTM states capture context-dependent aspects of word meaning (e.g., they can be used without modification to perform well on supervised word sense disambiguation tasks) while lower-level states model aspects of syntax (e.g., they can be used to do part-of-speech tagging).”◌ not checked against the paper’s text as it now stands

The paper proposes using an attention mechanism to weight frames differently when computing pooled speaker embeddings.

“Our method utilizes an attention mechanism to give different weights to different frames and generates not only weighted means but also weighted standard deviations.”◌ not checked against the paper’s text as it now stands

The paper introduces an attention model that computes importance scores for each frame to focus on discriminative frames.

“An attention model works in conjunction with the original DNN and calculates a scalar score e_t for each frame-level feature”◌ not checked against the paper’s text as it now stands

The paper applies attention weights to both weighted means and weighted standard deviations in the proposed pooling method.

“the authors believe that both higher-order statistics (standard deviations as utterance-level features) and attention mechanisms are effective for higher speaker discriminability. Hence, it would make sense to consider a new pooling method, attentive statistics pooling, which produces both means and standard deviations with importance weighting by means of attention”◌ not checked against the paper’s text as it now stands

The paper proposes attention gates that automatically learn to focus on target structures while suppressing irrelevant regions in medical images.

“Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that attention gates can replace external organ localization modules in segmentation frameworks without requiring multiple CNN models.

“The proposed attention gates can replace hard-attention approaches used in image classification and external organ localisation models in image segmentation frameworks.”◌ not checked against the paper’s text as it now stands

The paper shows that attention gates progressively suppress feature responses in irrelevant background regions during segmentation without cropping a region of interest between networks.

“AGs progressively suppress feature responses in irrelevant background regions without the requirement to crop a ROI between networks.”◌ not checked against the paper’s text as it now stands

The paper validates that attention gates consistently improve prediction accuracy across different datasets while maintaining computational efficiency.

This paper proposes jumping knowledge networks as an architecture that enables nodes to adaptively select different neighborhood ranges by combining representations from multiple layers at the final layer.

This paper analyzes how the influence distribution of nodes in graph neural networks is connected to random walk distributions on the graph structure.

“The following results show that the influence distributions of common aggregation schemes are closely connected to random walk distributions.”◌ not checked against the paper’s text as it now stands

This paper demonstrates that the jumping knowledge framework consistently improves performance when combined with existing models like Graph Convolutional Networks, GraphSAGE, and Graph Attention Networks.

This paper explores three attention-based mechanisms (concatenation, max-pooling, and LSTM-attention) to adaptively aggregate layer representations for each node.

The paper proposes using an attention network to distinguish which historical items in a user's profile are more important for making predictions in item-based collaborative filtering.

“The key to our design of NAIS is an attention network, which is capable of distinguishing which historical items in a user profile are more important for a prediction.”◌ not checked against the paper’s text as it now stands

The paper identifies that standard softmax attention fails due to large variance in user history lengths and proposes smoothing the attention denominator to address this problem.

“One of our key findings is that the standard attention mechanism fails to learn from users historical data, due to the large variance on the lengths of user histories. To address this, we adjust the attention design by smoothing user histories.”◌ not checked against the paper’s text as it now stands

The paper uses a Multi-Layer Perceptron to parameterize the attention function so that it can generalize to item pairs that have never co-occurred in training data.

“Inspired by the recent success of using neural networks to model the attention weight […], we similarly use a Multi-Layer Perception (MLP) to parameterize the attention function f.”

The paper uses a soft-attention mechanism to weight and aggregate node vectors when computing the global session embedding.

“Consider information in these embedding may have different levels of priority, we further adopt the soft-attention mechanism to better represent the global session preference: α_i = 𝐪^⊤ σ(𝐖_1 𝐯_n + 𝐖_2 𝐯_i + 𝐜), 𝐬_g = ∑_i = 1^nα_i 𝐯_i,”◌ not checked against the paper’s text as it now stands

The paper represents each session as a combination of global preference and current interest using an attention network.

“Each session is then represented as the composition of the global preference and the current interest of that session using an attention network.”◌ not checked against the paper’s text as it now stands

This survey identifies attention as one of seven main categories of recent CNN architectural innovations alongside spatial exploitation, depth, multi-path, width, feature-map exploitation, and channel boosting.

“The various CNN architectures discussed in this survey are broadly classified into seven main categories, namely; spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention-based CNNs.”◌ not checked against the paper’s text as it now stands

The paper indicates that attention-based information processing represents a shift in CNN research focus toward improved architectural design beyond parameter optimization.

“This shift resulted in many new architectural ideas such as channel boosting, spatial and feature-map wise exploitation and attention-based information processing etc.”◌ not checked against the paper’s text as it now stands

This paper proposes CutMix to guide models to attend to less discriminative parts of objects through regional dropout while maintaining training efficiency.

“Regional dropout strategies have been proposed to enhance the performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to attend on less discriminative parts of objects ( leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities.”◌ not checked against the paper’s text as it now stands

This paper demonstrates that CutMix enables models to accurately localize multiple object classes by making full use of image pixels, as shown through class activation map visualizations.

“CutMix efficiently improves upon Cutout by being able to localize the two object classes accurately.”◌ not checked against the paper’s text as it now stands

This paper shows that CutMix allows models to recognize two objects from their respective partial views in a single image through visual analysis of activation maps.

This paper demonstrates that CutMix improves model robustness and reduces over-confidence, making attention maps more stable across training.

The paper employs an attention mechanism to discriminate the importance of neighbors during embedding propagation in knowledge graphs.

“employs an attention mechanism to discriminate the importance of the neighbors”◌ not checked against the paper’s text as it now stands

The paper uses attention-based aggregation to learn the weight of each neighbor during propagation so that attention weights can reveal the importance of high-order connectivity.

“attention-based aggregation, which employs the neural attention mechanism […] to learn the weight of each neighbor during a propagation, such that the attention weights of cascaded propagations can reveal the importance of a high-order connectivity”◌ not checked against the paper’s text as it now stands

The paper introduces an attentive embedding propagation layer that uses attention mechanism to specify the contribution of each neighbor for propagating embeddings.

“we devise an attentive embedding propagation layer, which updates the representation of each node (it can be user, item, and entity) by aggregating messages from the connected neighbors. Wherein, the attention mechanism is introduced to specify the contribution of each neighbor for propagating embeddings towards the node”◌ not checked against the paper’s text as it now stands

VisualBERT uses the self-attention mechanism within Transformer layers to implicitly align elements of input text and regions in input images.

“The core of our idea is to reuse the self-attention mechanism within the Transformer to implicitly align elements of the input text and regions in the input image.”◌ not checked against the paper’s text as it now stands

VisualBERT's attention heads can ground visual concepts and track syntactic dependencies between words and image regions across different layers.

VisualBERT refines its attention-based alignments across layers, correcting initial misalignments in lower layers through higher-layer processing.

Certain attention heads in VisualBERT achieve high accuracy in grounding entities to image regions, with accuracy peaking at higher layers.

“We find that certain heads achieves high accuracy while the accuracy peaks at higher layers.”◌ not checked against the paper’s text as it now stands

The paper identifies attention maps as a form of feature-based knowledge that can be derived from original feature maps to express knowledge for distillation.

The paper reviews attention-based layer projection (ALP-KD) as a feature-based knowledge distillation method that uses attention mechanisms at hint layers.

“ALP-KD […] Attention-based layer projection Hint layer ℒ_2(.)”◌ not checked against the paper’s text as it now stands

This paper used a contrastive loss function that pushes embeddings of matched image-text pairs together while pushing non-matched pairs apart to train visual and language representations.

“Image and text encoders are learned via a contrastive loss (formulated as normalized softmax) that pushes the embeddings of matched image-text pair together while pushing those of non-matched image-text pair apart.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that the simple dual-encoder architecture without cross-attention mechanisms outperforms more sophisticated cross-attention models on image-text retrieval benchmarks.

“The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models.”◌ not checked against the paper’s text as it now stands

This paper made the softmax temperature variable a learnable parameter optimized jointly with other model parameters rather than manually tuned.

“Instead of manually sweeping for the optimal temperature value, we find that it can be effectively learned together with all the other parameters.”◌ not checked against the paper’s text as it now stands