“attention” · the papers and their links

“It is possible to construct deep LSTMNs by stacking multiple memory and hidden layers in an alternating fashion, resembling a stacked LSTM (Graves, 2013) or a multi-hop memory network (Sukhbaatar et al.”

“The attention at the (k + 1)th layer is computed as: ati,k+1 = v T tanh(Whhk+1i +Wlh k t +Wh̃h̃ k+1 t 1 ) (10) Skip-connections (Graves, 2013) can be applied to feed xt to upper layers as well.”

“We apply skip-connections (Graves, 2013) across layers (inputk+1 = inputk + outputk).”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• Introduces intra-attention (self-attention) applied within a single sequence encoder to induce undirected lexical relations among tokens, as opposed to prior attention work used mainly for inter-sequence alignment (e.g., encoder-decoder attention).
• Embeds the memory network and attention mechanism internal to the LSTM recurrence itself (replacing the single memory cell), rather than as an external memory module, strengthening interaction between memory and attention.
• Proposes 'deep attention fusion' and 'shallow attention fusion' methods to combine intra-attention with inter-attention in an encoder-decoder setting.

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

“A number of recent efforts have explored ways to capture long-term structure within sequences using RNNs or LSTM-based models (Chung et al., 2014; Graves, 2013; Koutn´ık et al., 2014; Mikolov et al., 2014; Hochreiter & Schmidhuber, 1997; Atkeson & Schaal, 1995).”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• Extends RNNsearch-style attention to multiple computational steps (hops) per output symbol
• Trains a continuous, end-to-end differentiable attention-based memory model without requiring supervision of which memories/facts are relevant
• Shows that multiple attention hops over memory yield improved performance on QA and language modeling tasks

“The attention mechanism in this respect was first used by Graves (2013) to build a neural network that generates convincing handwriting from a given text.”

“Accessing the input sequence through a context vector is related to the model that generates hand-written characters, proposed recently by Graves (2013).”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• We allow the learned soft alignment procedure to take the relative position into account and add a penalty helping the attention mechanism to choose a single, narrow mode and encourage that mode of attention to move forward.
• The learned dependency on the relative position of successive points of attention can, further, be used to speed up decoding, as the final model has to consider only a tiny section of the input sequence when generating each output symbol.
• In this paper, we propose a model, based on this neural translation model, that is more suited to speech recognition.

“At each step the model is auto-regressive [9], consuming the previously generated symbols as additional input when generating the next.”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• Proposes the Transformer, the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
• Introduces Scaled Dot-Product Attention with a scaling factor of 1/sqrt(d_k) to counteract vanishing gradients from large dot products.
• Introduces Multi-Head Attention, allowing the model to jointly attend to information from different representation subspaces at different positions.

“A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis.”

“The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the annotations only move in one direction.”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

“The use of gating architectures [3, 5], sophisticated optimization techniques [6, 7], gradient clipping [2, 8], and/or careful initialization [7, 9, 10, 11] can help mitigate this issue and has facilitated the success of RNNs in a variety of fields (see e.”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• We propose a simplified model of attention which is applicable to feed-forward neural networks
• the resulting model can solve the synthetic 'addition' and 'multiplication' long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks
• To our knowledge, such a variant of these tasks has not been studied before (variable-length sequence training)

“We favour LSTM due to its proven track record for handling long-range dependencies in real sequential data (Graves, 2013; Sutskever et al., 2014).”

“To endow the network with selective attention without sac-rificing the benefits of gradient descent training, we take in-spiration from the differentiable attention mechanisms recently used in handwriting synthesis (Graves, 2013) and Neural Turing Machines (Graves et al., 2014).”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

“From this perspective it is similar to machine translation and handwriting synthesis tasks, for which attention-based methods have been found suitable [2, 1].”

“Recently, attention-based recurrent networks have been successfully applied to a wide variety of tasks, such as handwriting synthesis [1], machine translation [2], image caption generation [3] and visual object classification [4].”

“For instance, Graves [1] used the location-based attention mechanism using a Gaussian mixture model in his handwriting synthesis model.”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

“Self-attention is a variant of attention [Graves, 2013; Bahdanau et al., 2015] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”

“Language models are typically used for compression or sequence generation [Graves, 2013].”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

“LSTMs have been used to achieve state-of-the-art performance in several tasks such as handwriting recognition [31], sequence generation [32] speech recognition [33] and machine translation [11] among others.”

• This makes it suitable for handwriting synthesis, where a human user inputs a text and the algorithm generates a handwritten version of it.
• The synthesis network is trained on the IAM database, then used to generate cursive handwriting samples, some of which cannot be distinguished from real data by the naked eye.
• This section describes an augmentation that allows a prediction network to generate data sequences conditioned on some high-level annotation sequence (a character string, in the case of handwriting synthesis).

“…image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture and its text description in the image caption generation task (Xu et al., 2015).”

“, 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture and its text description in the image caption generation task (Xu et al.”

• We allow the learned soft alignment procedure to take the relative position into account and add a penalty helping the attention mechanism to choose a single, narrow mode and encourage that mode of attention to move forward.
• The learned dependency on the relative position of successive points of attention can, further, be used to speed up decoding, as the final model has to consider only a tiny section of the input sequence when generating each output symbol.
• In this paper, we propose a model, based on this neural translation model, that is more suited to speech recognition.

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Proposes a fully data-driven, end-to-end trainable local attention-based model for abstractive sentence summarization that scales to large training data.
• Simplifies the attention-based encoder relative to prior neural MT attention models: uses an NNLM instead of a target-side LSTM, source-side windowed averaging instead of a bidirectional RNN encoder, and a weighted dot-product for alignment instead of an alignment MLP.
• Jointly trains the attention-based encoder and the generation (language) model end-to-end on the sentence summarization task, unlike prior noisy-channel approaches that estimate language model and translation model separately.

“…Networks (RNNs) based on Long Short-Term Memory (LSTM) [18], the attention based models have demonstrated outstanding performance on several challenging sequential data recognition and modeling tasks, including caption generation [19], machine translation [20], as well as action recognition [21].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We propose a new attention model that dynamically generates discriminative features in a recurrent way of "seeing" and "comparing" person images for automatically localizing the most discriminative parts of persons.
• our proposed CAN framework end-to-end learns the attention regions from raw images while the model in <cit.> generates attention regions based on the pre-extracted CNN features
• our model has a comparing ability in the generation of local attention regions as it is a three-branch architecture taking a triplet of images as input for each branch while <cit.> only has one branch structure taking one image as input

“Kiperwasser & Goldberg (2016) present a neural graph-based parser (in addition to a transition-based one) that uses the same kind of attention mechanism as Bahdanau et al. (2014) for machine translation.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Proposes a biaffine attention mechanism that replaces traditional MLP-based and bilinear attention mechanisms used in prior graph-based dependency parsers.
• Introduces the distinction between 'deep' biaffine attention (applying MLPs to reduce dimensionality before the biaffine transformation) versus 'shallow' bilinear attention (using recurrent states directly).
• Claims the biaffine attention mechanism directly models both the prior probability of a word receiving dependents and the likelihood of a specific dependent, which is conceptually simpler and more interpretable than MLP-based attention.

“In the encoder-decoder architecture, this problem can be sidestepped with an attention mechanism to learn a soft alignment between the encoding and decoding states (Bahdanau et al., 2014).”

“Meanwhile, we apply inter attention between them at every time step when the target sequence is analyzed, same as the RNNSearch (Bahdanau et al., 2014).”

“cent works leverage recurrent neural networks to understand text from scratch (Mikolov et al., 2010; Bahdanau et al., 2014).”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Introduces intra-attention (self-attention) applied within a single sequence encoder to induce undirected lexical relations among tokens, as opposed to prior attention work used mainly for inter-sequence alignment (e.g., encoder-decoder attention).
• Embeds the memory network and attention mechanism internal to the LSTM recurrence itself (replacing the single memory cell), rather than as an external memory module, strengthening interaction between memory and attention.
• Proposes 'deep attention fusion' and 'shallow attention fusion' methods to combine intra-attention with inter-attention in an encoder-decoder setting.

“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”

“Self-attention was proposed and has had great success in the field of natural language processing (NLP) [33], [38], [39], [149], [150], [151], [152].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Provides a comprehensive/systematic review of existing visual attention methods covering development history and latest research, friendly for beginners.
• Adopts a novel taxonomy grouping attention methods by the dimension they operate on (channel, spatial, temporal, branch, and hybrid combinations) rather than by application, allowing attention to be viewed as an independent research topic rather than a supplement to other tasks.
• Presents perspectives on future potential directions for attention mechanisms to inspire further research.

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• An end-to-end framework with two types of attention modules is designed based on the LSTM networks for skeleton based human action recognition.
• A spatial attention module with joint-selection gates is designed to adaptively allocate different attentions to different joints of the input skeleton within each frame. A temporal attention module with frame-selection gate is designed to allocate different attentions to different frames.
• Spatio-temporal regularizations are proposed to enable the better learning of the networks.

“Properly interpreting these coefficients (as performed by e.g. Bahdanau et al. (2015)) will require further domain knowledge about the dataset under study, and is left for future work.”

“Furthermore, analyzing the learned attentional weights may lead to benefits in interpretability, as was the case in the machine translation domain (e.g. the qualitative analysis of Bahdanau et al. (2015)).”

“The particular attentional setup utilized by us closely follows the work of Bahdanau et al. (2015)—but the framework is agnostic to the particular choice of attention mechanism.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Introduces graph attention networks (GATs), which apply masked self-attentional layers to graph-structured data, allowing nodes to attend over their neighborhoods' features and implicitly assign different weights to different neighbors without costly matrix operations or upfront knowledge of graph structure.
• The attention mechanism is efficient (parallelizable across node-neighbor pairs), applicable to nodes of varying degree, and directly usable for inductive learning on completely unseen graphs, addressing limitations of prior spectral graph convolution methods.
• Extends self-attention (as used in sequence tasks and Transformer-style architectures) to the graph domain via masked attention restricted to a node's neighborhood, combined with multi-head attention for stabilizing learning.

“Attention mechanisms have been successful components in both machine translation (Bahdanau et al., 2015; Luong module and memory module, to improve question answering.”

“They can benefit image classification (Stollenga et al., 2014), generating captions for images (Xu et al., 2015), among others mentioned below, and machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015).”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Proposes a new attention-based GRU that embeds the attention gate directly into the GRU update gate, allowing the model to be sensitive to position and ordering of facts unlike simple soft attention summation.
• Simplifies the attention gate computation z^t_i compared to the original DMN, removing several terms while maintaining performance.
• Introduces an input fusion layer using a bi-directional GRU to allow attention/context to flow between facts (sentences or image patches) in both directions, improving the quality of attended representations.

“To do so, we develop a sequence-to-sequence (seq2seq) version of ResMLP, where both encoder and decoders are based on ResMLP with across-attention between the encoder and decoder [7].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• ResMLP replaces the self-attention sublayer with a simple linear layer with no non-linearity, resulting in an architecture with only linear layers and GELU non-linearity
• Training of ResMLP is more stable than ViTs, allowing removal of batch-specific or cross-channel normalizations, which is speculated to result from replacing self-attention with linear layers
• The paper questions the importance of self-attention in existing architectures by showing a purely MLP-based model without any attention mechanism can achieve competitive accuracy on ImageNet and machine translation benchmarks

“, 2017), or a combination of recurrence and attention (Bahdanau et al., 2014; Cho et al., 2014; Luong et al., 2015; Wu et al., 2016) as basic building blocks.”

“…leverage recurrence (Sutskever et al., 2014), convolution (Gehring et al., 2017; Kalch-brenner et al., 2016), attention (Vaswani et al., 2017), or a combination of recurrence and attention (Bahdanau et al., 2014; Cho et al., 2014; Lu-ong et al., 2015; Wu et al., 2016) as basic building blocks.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements, rather than relying solely on absolute position encodings.
• Casting the extension as an instance of relation-aware self-attention that can generalize to arbitrary graph-labeled inputs by modeling input elements as a labeled, directed, fully-connected graph with edge representations feeding into both the compatibility function and the value aggregation.
• An efficient implementation that avoids O(h n^2 d_a) space complexity by sharing relative position representations across attention heads, reducing it to O(n^2 d_a), and by splitting the compatibility computation into two matrix multiplication terms to avoid broadcasting.

“[3] propose RNNSearch to learn an alignment over the input sentences.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Proposes a novel co-attention mechanism jointly performing question-guided visual attention and image-guided question attention, with two strategies: parallel and alternating co-attention
• Proposes a hierarchical architecture representing the question at word, phrase, and question levels, constructing co-attention maps at each level and recursively combining them for answer prediction
• Proposes a novel convolution-pooling strategy at the phrase level to adaptively select phrase sizes

“Recently, attention mechanisms have become an integral part of models that must capture global dependencies [2, 34, 36, 6].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• First application of self-attention to GANs for image generation, allowing both generator and discriminator to model long-range, multi-level dependencies across image regions
• Introduces a learnable scalar gating parameter (initialized to 0) that lets the network progressively shift from relying on local convolutional cues to non-local self-attention evidence
• Applies spectral normalization to the GAN generator (previously used only on the discriminator) to improve conditioning and training dynamics

“Other recently proposed forms of memory or attention take this approach, notably Graves et al. (2014) and Bahdanau et al. (2015).”

“Our model is also related to Bahdanau et al. (2015).”

“Our “memory” is analogous to their attention mechanism, both using weak supervision during training, although Bahdanau et al. (2015) is only a single sentence rather than many, as in our case.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Extends RNNsearch-style attention to multiple computational steps (hops) per output symbol
• Trains a continuous, end-to-end differentiable attention-based memory model without requiring supervision of which memories/facts are relevant
• Shows that multiple attention hops over memory yield improved performance on QA and language modeling tasks

“In (b), we observe how the absence of the learned preference for monotonicity makes the model confused by the repeated occurrence of the phonemes “cl k”, and to a lesser degree, by the repetition of “w”. sequence with elements of the output sequence (Bahdanau et al. , 2014).”

“sequence with elements of the output sequence (Bahdanau et al., 2014).”

“The proposed model (a variant of RNN Encoder–Decoder (Cho et al. , 2014; Bahdanau et al. , 2014)) instead uses the scores as an explicit alignment which is used to compute a context vector.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We allow the learned soft alignment procedure to take the relative position into account and add a penalty helping the attention mechanism to choose a single, narrow mode and encourage that mode of attention to move forward.
• The learned dependency on the relative position of successive points of attention can, further, be used to speed up decoding, as the final model has to consider only a tiny section of the input sequence when generating each output symbol.
• In this paper, we propose a model, based on this neural translation model, that is more suited to speech recognition.

“The first group (largely inspired by Bahdanau et al. (2015)) uses a dynamic attention mechanism, in which the attention weights are updated dynamically given the query and the context as well as the previous attention.”

“Attention mechanisms in previous works typically have one or more of the following characteristics.”

“That is, while we iteratively compute attention through time as in Bahdanau et al. (2015), the attention at each time step is a function of only the query and the context para-graph at the current time step and does not directly depend on the attention at the previous time step.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Our attention layer is not used to summarize the context paragraph into a fixed-size vector; instead the attention is computed for every time step and allowed to flow through to the modeling layer, reducing information loss from early summarization.
• We use a memory-less attention mechanism, where attention at each time step is a function of only the query and context at that time step and does not directly depend on the attention at the previous time step, unlike dynamic attention models.
• We use attention mechanisms in both directions, query-to-context and context-to-query, which provide complementary information to each other, in contrast to previous uni-directional attention approaches.

“speech recognition and machine translation [5], [7], [22], [23], [46].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Both models are built on top of the transformer self-attention architecture
• the architecture incorporates a novel dual attention mechanism that can operate over visual input only, audio input only, or both
• In every decoder layer, the resulting video and audio contexts are concatenated over the channel dimension and propagated to the feedforward block.

“Following the definition from [15], given a model which produces a hidden state ht at each time step, attention-based models compute a “context” vector ct as the weighted mean of the state sequence h by”

“They also provide an alternate argument for the claim made by Bahdanau et al. (2014) that attention helps models handle very long and widely variable-length sequences.”

“Attention-based RNNs have proven effective in a variety of sequence transduction tasks, including machine translation [15], image captioning [16], and speech recognition [17, 18].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We propose a simplified model of attention which is applicable to feed-forward neural networks
• the resulting model can solve the synthetic 'addition' and 'multiplication' long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks
• To our knowledge, such a variant of these tasks has not been studied before (variable-length sequence training)

“For instance, additive soft attention is used in sentence-to-sentence translation [2, 29] and more recently applied to image classification [11, 32].”

“Attention Gates: AGs are commonly used in natural image analysis, knowledge graphs, and language processing (NLP) for image captioning [1], machine translation [2, 30], and classification [11, 31, 32] tasks.”

“We use additive attention [2] to obtain the gating coefficient.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We take the attention approach proposed in <cit.> a step further by proposing grid-based gating that allows attention coefficients to be more specific to local regions.
• We propose one of the first use cases of soft-attention technique in a feed-forward CNN model applied to a medical imaging task.
• An extension to the standard U-Net model is proposed to improve model sensitivity to foreground pixels without requiring complicated heuristics.

“The attention mechanism proposed by Bahdanau et al. [18] can help the LSTM RNN learn these dependencies.”

“Bahdanau et al. [18] parametrize the alignment model a as a feedforward neural network which is jointly trained with all the other components of the model.”

“As discussed in Bahdanau et al. [18], the context vector c i depends on a sequence of annotations ( h 1 , ..., h T x ) to which an encoder maps the input sequence.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We also explore the usage of attention mechanism to improve time series classification with the Attention Long Short Term Memory Fully Convolutional Network (ALSTM-FCN).
• Utilization of the attention mechanism allows one to visualize the decision process of the LSTM cell.
• In addition, the Attention LSTM can also be used detect regions of the input sequence that contribute to the class label through the context vector of the Attention LSTM cells.

“The attention mechanism (Bahdanau et al., 2015) has been a highly successful neural network component for NLP tasks, which is also crucial for pre-trained LMs.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• introduces the scaled dot-product between values in the self-attention module as new deep self-attention knowledge, in addition to attention distributions
• proposes distilling only the self-attention module of the teacher's last Transformer layer rather than layer-to-layer, allowing flexible student layer numbers
• shows that value-relation transfer allows arbitrary student hidden dimensions without extra parameters

“Then we learn the interactions between context and question by standard attentions (Xiong et al., 2016; Seo et al., 2016; Bahdanau et al., 2015).”

“In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al. (2015); Luong et al. (2015), which have demonstrated excellent translation quality Wu et al. (2016), as the core models of our data augmentation pipeline.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We propose an efficient reading comprehension model that exclusively built upon convolutions and self-attentions... To the best of our knowledge, we are the first to do so.
• the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments
• convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words

“…to increase capacity and minimize the impact of vanishing gradients, some language and translation mod-Pointer els have also added a soft attention or memory component (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Cheng et al., 2016; Kumar et al., 2016; Xiong et al., 2016; Ahn et al., 2016).”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Introduces a pointer sentinel mixture architecture that combines a softmax-RNN component with a pointer attention component via a learned sentinel gate, rather than relying on the RNN hidden state alone to decide when to use the pointer.
• Integrates the gating function directly into the pointer attention computation (via an inner product with a sentinel vector) so that the decision to back off to the softmax vocabulary is informed by both the query and the pointer window's hidden states, addressing limitations of prior switching-network approaches.
• Achieves state-of-the-art Penn Treebank perplexity with far fewer parameters than large softmax LSTM models by leveraging this attention-based pointer sentinel mechanism.

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“, 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).”

“Their model implements the attention-based sequence to sequence architecture of Bahdanau et al. (2014) and uses GRU cells both in the encoder and decoder.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We also use attention in every decoder layer and demonstrate that each attention layer only adds a negligible amount of overhead.
• This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).
• Our convolutional architecture also allows to batch the attention computation across all elements of a sequence compared to RNNs

“Often this issue is partially alleviated by an encoder such as e.g. a BiRNN [2] or a deep convolutional network [3] that encode contextual information into every element of h .”

“We start with a model proposed in [2] for the machine translation task as the baseline.”

“We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

“Self-attention is a variant of attention [Graves, 2013; Bahdanau et al., 2015] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”

“Similar arguments have been made against using a unidirectional recurrent neural network encoder in sequence-to-sequence models [Bahdanau et al., 2015].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

“design of neural attention network [15], [16], [17], [19] did not work well for item-based CF, due to the large variances of the lengths of user histories.”

“The two choices of attention function differ in the construction of input: the first choice fconcat simply concatenates pi and qj to learn the attention weight wij [19], while second choice fprod feeds the element-wise product of pi and qj into the attention network [15].”

“Note that setting b to 1 means using softmax to normalize the attention weights, a standard setting for neural attention networks [15], [16], [19].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• This work is the first attempt that designs neural network models for item-based CF, opening up new research possibilities for future developments of neural recommender systems.
• The key to our design of NAIS is an attention network, which is capable of distinguishing which historical items in a user profile are more important for a prediction.
• One of our key findings is that the standard attention mechanism fails to learn from users historical data, due to the large variance on the lengths of user histories.

“Attention mechanisms (Bahdanau et al., 2014) make these models more performant and scalable, allowing them to look back at parts of the encoded input sequence while the output is generated.”

“Recently, neural network models (Nallapati et al., 2016; Zeng et al., 2016), based on the attentional encoder-decoder model for machine translation (Bahdanau et al., 2014), were able to generate abstractive summaries with high ROUGE scores.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We introduce a key attention mechanism...: (i) we use an intra-temporal attention in the encoder that records previous attention weights for each of the input tokens while a sequential intra-attention model in the decoder takes into account which words have already been generated by the decoder.
• This mechanism is not present in existing encoder-decoder models for abstractive summarization.
• A closely-related intra-RNN attention function has been introduced by <cit.> but their implementation works by modifying the underlying LSTM function, and they do not apply it to long sequence generation problems. This is a major difference with our method, which makes no assumptions about the type of decoder RNN, thus is more simple and widely applicable to other types of recurrent networks.

“These models draw on recent developments for incorporating attention mechanisms into recurrent neural network architectures [6, 7, 8].”

“The fixed width hidden vector forms a bottleneck for this information flow that we propose to circumvent using an attention mechanism inspired by recent results in translation and image recognition [6, 7].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

“It was recently adapted for neural machine translation with recurrent neural networks, e.g. Bahdanau et al. (2014) as well as in several other NLP-related tasks.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We propose attention as a mechanism of transferring knowledge from one network to another
• We propose the use of both activation-based and gradient-based spatial attention maps
• We show experimentally that our approach provides significant improvements across a variety of datasets and deep network architectures, including both residual and non-residual networks

“Current methods include using a convolutional encoder and RNN decoder [8], RNN encoder and RNN decoder [9, 10] and LSTM encoder with LSTM decoder [11].”

“Furthermore, our method builds on analogous approaches being used in machine translation [8, 9, 10, 11].”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

“On the other hand, in (Bahdanau et al., 2015; Jean et al., 2015) and this work, s, in fact, implies a set of source hidden states which are consulted throughout the entire course of the translation process.”

“Comparison to other work – Bahdanau et al. (2015) use context vectors, similar to our ct, in building subsequent hidden states, which can also achieve the “coverage” effect.”

“…part of at and for long sentences, we ignore words near the end. goes through a deep-output and a maxout layer before making predictions.7 Lastly, Bahdanau et al. (2015) only experimented with one alignment function, the concat product; whereas we show later that the other alternatives are…”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

“Attention models can be traced back to the machine translation models of (Bahdanau et al., 2014) and (Luong et al., 2015), where attention is employed to learn soft word alignments between language pairs.”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• We propose Synthetic Attention, a new way of learning to attend without explicitly attending (i.e., without dot product attention or content-based attention).
• We propose Synthesizer, a new model that leverages Synthetic Attention. The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks, including machine translation and language modeling.
• we show that (1) random learnable alignment matrices perform competitively and (2) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks.

“The attention mechanism is widely used in deep learning for image captioning (Xu et al., 2015) or text analysis (Bahdanau et al., 2014; Lin et al., 2017).”

“The two-layered neural network corre-1 According to the European Union General Data Protection Regulation (taking effect 2018), a user should have the right to obtain an explanation of the decision reached. sponds to the attention mechanism (Bahdanau et al., 2014; Raffel & Ellis, 2015).”

“sponds to the attention mechanism (Bahdanau et al., 2014; Raffel & Ellis, 2015).”

• Proposes a novel encoder-decoder extension that jointly learns to align and translate, using an attention/alignment mechanism instead of encoding the whole source sentence into a single fixed-length vector.
• The alignment model computes a soft alignment directly (not treated as a latent variable), allowing gradients to backpropagate through the alignment and translation model jointly, unlike traditional SMT alignment.
• Distinguishes its bidirectional, non-monotonic attention mechanism from prior monotonic attention-like approaches (e.g., Graves' handwriting synthesis alignment) by computing weights over all source positions rather than enforcing monotonic movement.

• Proposes a fully trainable, adaptive MIL pooling operator based on attention, replacing pre-defined non-trainable operators like max and mean.
• Introduces a gated attention mechanism combining tanh and sigmoid non-linearities to overcome the near-linearity of tanh and better capture complex relations among instances.
• Applies attention without sequential dependency among instances (unlike typical sequence-based attention), questioning whether attention can work in this independent-instance setting.

“, 2014) and object recognition (Ba et al., 2014; Mnih et al., 2014), we investigate models that can attend to salient part of an image while generating its caption.”

“…by recent advances in caption generation and inspired by recent success in employing attention in machine translation (Bahdanau et al., 2014) and object recognition (Ba et al., 2014; Mnih et al., 2014), we investigate models that can attend to salient part of an image while generating its caption.”

“As pointed out and used in Ba et al. (2014) and Mnih et al. (2014), this is formulation is equivalent to the REINFORCE learning rule (Williams, 1992), where the reward for the attention choosing a sequence of actions is a real value proportional to the log likelihood of the target sentence under…”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

“Hard attention [21], e.g. iterative region proposal and cropping, is often non-differentiable and relies on reinforcement learning for parameter updates, which makes model training more difficult.”

“Lastly, we would like to note that AG parameters can be trained with the standard back-propagation updates without a need for sampling based update methods used in hard-attention [21].”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• We take the attention approach proposed in <cit.> a step further by proposing grid-based gating that allows attention coefficients to be more specific to local regions.
• We propose one of the first use cases of soft-attention technique in a feed-forward CNN model applied to a medical imaging task.
• An extension to the standard U-Net model is proposed to improve model sensitivity to foreground pixels without requiring complicated heuristics.

“The network is similar to the recently introduced Recurrent Attention Model (RAM) (Mnih et al., 2014), except that our attention method is differentiable; we therefore refer to it as “Differ-entiable RAM”.”

“…structure can be better captured by a sequence of partial glimpses, or foveations, than by a single sweep through the entire image (Larochelle & Hinton, 2010; Denil et al., 2012; Tang et al., 2013; Ranzato, 2014; Zheng et al., 2014; Mnih et al., 2014; Ba et al., 2014; Ser-manet et al., 2014).”

“To test the classification efficacy of the DRAW attention mechanism (as opposed to its ability to aid in image generation), we evaluate its performance on the 100 × 100 clut-tered translated MNIST task (Mnih et al., 2014).”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

“These models draw on recent developments for incorporating attention mechanisms into recurrent neural network architectures [6, 7, 8].”

“The fixed width hidden vector forms a bottleneck for this information flow that we propose to circumvent using an attention mechanism inspired by recent results in translation and image recognition [6, 7].”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

“It was also exploited in computer-vision-related tasks such as image captioning Xu et al. (2015), visual question answering Yang et al. (2015), as well as in weakly-supervised object localization Oquab et al. (2015) and classification Mnih et al. (2014), to mention a few characteristic examples.”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• We propose attention as a mechanism of transferring knowledge from one network to another
• We propose the use of both activation-based and gradient-based spatial attention maps
• We show experimentally that our approach provides significant improvements across a variety of datasets and deep network architectures, including both residual and non-residual networks

“Attention can be viewed, broadly, as a tool to bias the allocation of available processing resources towards the most informative components of an input signal [17, 18, 22, 29, 32].”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• SE blocks intrinsically introduce dynamics conditioned on the input, which can be regarded as a self-attention function on channels whose relationships are not confined to the local receptive field the convolutional filters are responsive to.
• By contrast, our proposed SE block comprises a lightweight gating mechanism which focuses on enhancing the representational power of the network by modelling channel-wise relationships in a computationally efficient manner, unlike prior trunk-and-mask attention mechanisms based on hourglass modules.

“…allowing models to learn alignments between different modalities, e.g., between image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture…”

“, between image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al.”

• Develops a novel framework for attention-based task-driven visual processing with neural networks, formulated as a control problem general enough for static images, videos, or dynamic interactive environments
• Uses an RNN to integrate visual information over time and decide how to act, which is more general than prior deep learning attention attempts that rely on greedy action selection
• Provides an end-to-end optimization procedure combining backpropagation with policy gradient (REINFORCE) to train the non-differentiable sequential attention/action decision process

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• Our model inherits the simplest VSE form, but still outperforms all previous cross-attention models in image-text matching benchmarks.
• ALIGN outperforms the previous SOTA method by over 7% in most zero-shot and fine-tuned R@1 metrics in Flickr30K and MSCOCO, even when compared with more sophisticated cross-attention models.

“We use the non-anonymized version of the CNN/DailyMail dataset [37] and Gigaword [36] for model fine-tuning and evaluation.”

• Proposes a fully data-driven, end-to-end trainable local attention-based model for abstractive sentence summarization that scales to large training data.
• Simplifies the attention-based encoder relative to prior neural MT attention models: uses an NNLM instead of a target-side LSTM, source-side windowed averaging instead of a bidirectional RNN encoder, and a weighted dot-product for alignment instead of an alignment MLP.
• Jointly trains the attention-based encoder and the generation (language) model end-to-end on the sentence summarization task, unlike prior noisy-channel approaches that estimate language model and translation model separately.

• The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on.
• We use different mask matrices M to control what context a token can attend to when computing its contextualized representation.
• UniLM can be configured, using different self-attention masks (Section), to aggregate context for different types of language models, and thus can be used for both NLU and NLG tasks.

“We also evaluate on a Gigaword test set of 2000 pairs which is identical to the one used by Rush et al. (2015) and we report F1 ROUGE similar to prior work.”

“We train on the Gigaword corpus (Graff et al., 2003) and pre-process it identically to Rush et al. (2015) resulting in 3.8M training examples and 190K for validation.”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al., 2015; Nallapati et al., 2016; Shen et al., 2016) amongst others.”

• Proposes a fully data-driven, end-to-end trainable local attention-based model for abstractive sentence summarization that scales to large training data.
• Simplifies the attention-based encoder relative to prior neural MT attention models: uses an NNLM instead of a target-side LSTM, source-side windowed averaging instead of a bidirectional RNN encoder, and a weighted dot-product for alignment instead of an alignment MLP.
• Jointly trains the attention-based encoder and the generation (language) model end-to-end on the sentence summarization task, unlike prior noisy-channel approaches that estimate language model and translation model separately.

• We also use attention in every decoder layer and demonstrate that each attention layer only adds a negligible amount of overhead.
• This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).
• Our convolutional architecture also allows to batch the attention computation across all elements of a sequence compared to RNNs

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al., 2015; Nallapati et al., 2016; Shen et al., 2016) amongst others.”

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

• We also use attention in every decoder layer and demonstrate that each attention layer only adds a negligible amount of overhead.
• This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).
• Our convolutional architecture also allows to batch the attention computation across all elements of a sequence compared to RNNs

“terminology of neural networks with ‘attention’ [31] mechanisms, the RPN module tells the Fast R-CNNmodule where to look.”

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

“Recently, attention mechanisms have become an integral part of models that must capture global dependencies [2, 34, 36, 6].”

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

• First application of self-attention to GANs for image generation, allowing both generator and discriminator to model long-range, multi-level dependencies across image regions
• Introduces a learnable scalar gating parameter (initialized to 0) that lets the network progressively shift from relying on local convolutional cues to non-local self-attention evidence
• Applies spectral normalization to the GAN generator (previously used only on the discriminator) to improve conditioning and training dynamics

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

“These models draw on recent developments for incorporating attention mechanisms into recurrent neural network architectures [6, 7, 8].”

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

“It is important to note that pt is a real nummber; whereas s is an integer within the window centered at pt.10 Comparison to (Gregor et al., 2015) – have proposed a selective attention mechanism, very similar to our local attention, for the image generation task.”

“…log p (yj|y<j , s) (1) A natural choice to model such a decomposition in the decoder is to use a recurrent neural network (RNN) architecture, which most of the re- 2There is a recent work by Gregor et al. (2015), which is very similar to our local attention and applied to the image generation task.”

“Comparison to (Gregor et al., 2015) – have proposed a selective attention mechanism, very similar to our local attention, for the image generation task.”

• The attention model in DRAW is fully differentiable, making it possible to train with standard backpropagation, unlike reinforcement-learning-based attention approaches.
• Unlike prior differentiable attention mechanisms (handwriting synthesis, Neural Turing Machines), DRAW uses an explicitly two-dimensional form of attention via a grid of 2D Gaussian filters with dynamically determined location, zoom, and precision.
• The 2D differentiable attention mechanism improves both generative image modelling (state-of-the-art binarized MNIST results) and classification (outperforming the non-differentiable Recurrent Attention Model on cluttered MNIST).

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

“Attention mechanisms have been successful components in both machine translation (Bahdanau et al., 2015; Luong module and memory module, to improve question answering.”

“They can benefit image classification (Stollenga et al., 2014), generating captions for images (Xu et al., 2015), among others mentioned below, and machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015).”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• Proposes a new attention-based GRU that embeds the attention gate directly into the GRU update gate, allowing the model to be sensitive to position and ordering of facts unlike simple soft attention summation.
• Simplifies the attention gate computation z^t_i compared to the original DMN, removing several terms while maintaining performance.
• Introduces an input fusion layer using a bi-directional GRU to allow attention/context to flow between facts (sentences or image patches) in both directions, improving the quality of attended representations.

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• Extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements, rather than relying solely on absolute position encodings.
• Casting the extension as an instance of relation-aware self-attention that can generalize to arbitrary graph-labeled inputs by modeling input elements as a labeled, directed, fully-connected graph with edge representations feeding into both the compatibility function and the value aggregation.
• An efficient implementation that avoids O(h n^2 d_a) space complexity by sharing relative position representations across attention heads, reducing it to O(n^2 d_a), and by splitting the compatibility computation into two matrix multiplication terms to avoid broadcasting.

“…alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent alignments produced by the attention mechanism are unstable when the network is deep.”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• The attention mechanism connects the bottom layer of the decoder to the top layer of the encoder to improve parallelism and decrease training time, unlike aligning top-to-top which would remove parallelism in the decoder network.

“Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• Proposes the Transformer, the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
• Introduces Scaled Dot-Product Attention with a scaling factor of 1/sqrt(d_k) to counteract vanishing gradients from large dot products.
• Introduces Multi-Head Attention, allowing the model to jointly attend to information from different representation subspaces at different positions.

“Although this is computationally more expensive, it has experimentally shown to achieve higher accuracy than multiplicative attention [19].”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• We take the attention approach proposed in <cit.> a step further by proposing grid-based gating that allows attention coefficients to be more specific to local regions.
• We propose one of the first use cases of soft-attention technique in a feed-forward CNN model applied to a medical imaging task.
• An extension to the standard U-Net model is proposed to improve model sensitivity to foreground pixels without requiring complicated heuristics.

“In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al. (2015); Luong et al. (2015), which have demonstrated excellent translation quality Wu et al. (2016), as the core models of our data augmentation pipeline.”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• We propose an efficient reading comprehension model that exclusively built upon convolutions and self-attentions... To the best of our knowledge, we are the first to do so.
• the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments
• convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“On WMT’14 English to German translation we compare to the following prior work: Luong et al. (2015) is based on a four layer LSTM attention model, ByteNet (Kalchbrenner et al., 2016) propose a convolutional model based on characters without attention, with 30 layers in the encoder and 30 layers in…”

“, 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• We also use attention in every decoder layer and demonstrate that each attention layer only adds a negligible amount of overhead.
• This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016).
• Our convolutional architecture also allows to batch the attention computation across all elements of a sequence compared to RNNs

“Attention models can be traced back to the machine translation models of (Bahdanau et al., 2014) and (Luong et al., 2015), where attention is employed to learn soft word alignments between language pairs.”

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

• We propose Synthetic Attention, a new way of learning to attend without explicitly attending (i.e., without dot product attention or content-based attention).
• We propose Synthesizer, a new model that leverages Synthetic Attention. The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks, including machine translation and language modeling.
• we show that (1) random learnable alignment matrices perform competitively and (2) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks.

• Extends RNNsearch-style attention to multiple computational steps (hops) per output symbol
• Trains a continuous, end-to-end differentiable attention-based memory model without requiring supervision of which memories/facts are relevant
• Shows that multiple attention hops over memory yield improved performance on QA and language modeling tasks

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• We point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT).
• The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs.
• There are two self-attention layers in our TNT block, i.e., an inner self-attention and an outer self-attention for modeling relationship among visual words and sentences respectively.

“Designing deep architectures for joint tasks is popular and effective, which has been used for several vision tasks [38]–[41].”

“All of the methods in [39]–[41] solve a common problem with”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

“However, the pre-training with CutMix improves the performance of both SSD and Faster-RCNN.”

“Transferring to Pascal VOC object detection: Two pop- ular detection models, SSD [23] and Faster RCNN [29], are considered.”

“Detection Image Captioning SSD [23] (mAP) Faster-RCNN [29] (mAP) NIC [42] (BLEU-1) NIC [42] (BLEU-4) ResNet-50 (Baseline) 23.68 76.7 (+0.0) 75.6 (+0.0) 61.4 (+0.0) 22.9 (+0.0) Mixup-trained 22.58 76.6 (-0.1) 73.9 (-1.7) 61.6 (+0.2) 23.2 (+0.3) Cutout-trained 22.93 76.8 (+0.1) 75.0 (-0.6) 63.0 (+1.6) 24.0 (+1.1) CutMix-trained 21.40 77.6 (+0.9) 76.7 (+1.1) 64…”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• CutMix guides a classifier to attend to broader sets of cues to make decisions, improving weakly supervised object localization performance compared to Mixup and Cutout.
• Unlike Cutout, which removes information, CutMix retains full pixel information while still encouraging attention to non-discriminative object parts, as shown via CAM visualizations.

“Indeed, there has been some very recent work in this space [60], [61], [62] that shows promising”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• This survey classifies recent CNN architectural innovations into seven categories, one of which is attention-based CNNs, providing a novel taxonomy for understanding how attention mechanisms fit within broader CNN architectural evolution.

“We use Faster R-CNN [33] as the detection method and follow the basic implementation in [10].”

“Convolutional neural networks (CNNs) have proven to be effective models for tackling a variety of visual tasks [21, 27, 33, 45].”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• SE blocks intrinsically introduce dynamics conditioned on the input, which can be regarded as a self-attention function on channels whose relationships are not confined to the local receptive field the convolutional filters are responsive to.
• By contrast, our proposed SE block comprises a lightweight gating mechanism which focuses on enhancing the representational power of the network by modelling channel-wise relationships in a computationally efficient manner, unlike prior trunk-and-mask attention mechanisms based on hourglass modules.

“For example, on the COCO (Lin et al., 2014) benchmark, DETR needs 500 epochs to converge, which is around 10 to 20 times slower than Faster R-CNN (Ren et al., 2015).”

“, 2014) benchmark, DETR needs 500 epochs to converge, which is around 10 to 20 times slower than Faster R-CNN (Ren et al., 2015).”

“Our proposed Deformable DETR has on par FLOPs with Faster R-CNN + FPN and DETR-DC5.”

• The paper uses 'attention' terminology only as an analogy borrowed from existing neural network attention mechanisms to describe the RPN's role, not as a novel attention mechanism itself.
• The RPN module is described as serving as the 'attention' of the unified Faster R-CNN network, directing the detection module where to look, using nearly cost-free, shared convolutional features.

• We propose the deformable attention module, which attends to a small set of sampling locations as a pre-filter for prominent key elements out of all the feature map pixels.
• the deformable attention module only attends to a small set of key sampling points around a reference point, regardless of the spatial size of the feature maps
• The proposed (multi-scale) deformable attention module can also be perceived as an efficient variant of Transformer attention, where a pre-filtering mechanism is introduced by the deformable sampling locations.

“Interestingly, the proposed MIL pooling corresponds to a version of the attention mechanism (Lin et al., 2017; Raffel & Ellis, 2015).”

“The two-layered neural network corre-1 According to the European Union General Data Protection Regulation (taking effect 2018), a user should have the right to obtain an explanation of the decision reached. sponds to the attention mechanism (Bahdanau et al., 2014; Raffel & Ellis, 2015).”

• We propose a simplified model of attention which is applicable to feed-forward neural networks
• the resulting model can solve the synthetic 'addition' and 'multiplication' long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks
• To our knowledge, such a variant of these tasks has not been studied before (variable-length sequence training)

• Proposes a fully trainable, adaptive MIL pooling operator based on attention, replacing pre-defined non-trainable operators like max and mean.
• Introduces a gated attention mechanism combining tanh and sigmoid non-linearities to overcome the near-linearity of tanh and better capture complex relations among instances.
• Applies attention without sequential dependency among instances (unlike typical sequence-based attention), questioning whether attention can work in this independent-instance setting.

“3, introduced in [27], [19] and [21].”

“Recently, LSTMs have shown good performance in the domain of speech recognition [23] and image description [19].”

“…Networks (RNNs) based on Long Short-Term Memory (LSTM) [18], the attention based models have demonstrated outstanding performance on several challenging sequential data recognition and modeling tasks, including caption generation [19], machine translation [20], as well as action recognition [21].”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• We propose a new attention model that dynamically generates discriminative features in a recurrent way of "seeing" and "comparing" person images for automatically localizing the most discriminative parts of persons.
• our proposed CAN framework end-to-end learns the attention regions from raw images while the model in <cit.> generates attention regions based on the pre-extracted CNN features
• our model has a comparing ability in the generation of local attention regions as it is a three-branch architecture taking a triplet of images as input for each branch while <cit.> only has one branch structure taking one image as input

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• Provides a comprehensive/systematic review of existing visual attention methods covering development history and latest research, friendly for beginners.
• Adopts a novel taxonomy grouping attention methods by the dimension they operate on (channel, spatial, temporal, branch, and hybrid combinations) rather than by application, allowing attention to be viewed as an independent research topic rather than a supplement to other tasks.
• Presents perspectives on future potential directions for attention mechanisms to inspire further research.

“has attracted increasing interest in various fields, such as machine translation (Bahdanau, Cho, and Bengio 2014), image caption generation (Xu et al. 2015), and image recognition (Ba, Mnih, and Kavukcuoglu 2014).”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• An end-to-end framework with two types of attention modules is designed based on the LSTM networks for skeleton based human action recognition.
• A spatial attention module with joint-selection gates is designed to adaptively allocate different attentions to different joints of the input skeleton within each frame. A temporal attention module with frame-selection gate is designed to allocate different attentions to different frames.
• Spatio-temporal regularizations are proposed to enable the better learning of the networks.

“They can benefit image classification (Stollenga et al., 2014), generating captions for images (Xu et al., 2015), among others mentioned below, and machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015).”

“Shortly thereafter, recurrent neural networks were used to generate often novel sentences based on images (Karpathy & Fei-Fei, 2015; Chen & Zitnick, 2014; Fang et al., 2015; Xu et al., 2015).”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• Proposes a new attention-based GRU that embeds the attention gate directly into the GRU update gate, allowing the model to be sensitive to position and ordering of facts unlike simple soft attention summation.
• Simplifies the attention gate computation z^t_i compared to the original DMN, removing several terms while maintaining performance.
• Introduces an input fusion layer using a bi-directional GRU to allow attention/context to flow between facts (sentences or image patches) in both directions, improving the quality of attended representations.

“Recently, attention mechanisms have become an integral part of models that must capture global dependencies [2, 34, 36, 6].”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• First application of self-attention to GANs for image generation, allowing both generator and discriminator to model long-range, multi-level dependencies across image regions
• Introduces a learnable scalar gating parameter (initialized to 0) that lets the network progressively shift from relying on local convolutional cues to non-local self-attention evidence
• Applies spectral normalization to the GAN generator (previously used only on the discriminator) to improve conditioning and training dynamics

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• Extends RNNsearch-style attention to multiple computational steps (hops) per output symbol
• Trains a continuous, end-to-end differentiable attention-based memory model without requiring supervision of which memories/facts are relevant
• Shows that multiple attention hops over memory yield improved performance on QA and language modeling tasks

“…more direct access to the entire state sequence h. Attention-based RNNs have proven effective in a variety of sequence transduction tasks, including machine translation (Bahdanau et al., 2014), image captioning (Xu et al., 2015), and speech recognition (Chan et al., 2015; Bahdanau et al., 2015).”

“Very long sequences can also make training computationally inefficient due to the fact that RNNs must be evaluated sequentially and cannot be fully parallelized.”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• We propose a simplified model of attention which is applicable to feed-forward neural networks
• the resulting model can solve the synthetic 'addition' and 'multiplication' long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks
• To our knowledge, such a variant of these tasks has not been studied before (variable-length sequence training)

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue.
• we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
• the contribution of this work is three-fold. For one, we present a novel purely neural speech recognition architecture based on an attention mechanism... Moreover, we propose a generic method of adding location awareness to the attention mechanism. Finally, we introduce a modification of the attention mechanism to avoid concentrating the attention on a single frame.

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• This survey classifies recent CNN architectural innovations into seven categories, one of which is attention-based CNNs, providing a novel taxonomy for understanding how attention mechanisms fit within broader CNN architectural evolution.

“It was also exploited in computer-vision-related tasks such as image captioning Xu et al. (2015), visual question answering Yang et al. (2015), as well as in weakly-supervised object localization Oquab et al. (2015) and classification Mnih et al. (2014), to mention a few characteristic examples.”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• We propose attention as a mechanism of transferring knowledge from one network to another
• We propose the use of both activation-based and gradient-based spatial attention maps
• We show experimentally that our approach provides significant improvements across a variety of datasets and deep network architectures, including both residual and non-residual networks

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• SE blocks intrinsically introduce dynamics conditioned on the input, which can be regarded as a self-attention function on channels whose relationships are not confined to the local receptive field the convolutional filters are responsive to.
• By contrast, our proposed SE block comprises a lightweight gating mechanism which focuses on enhancing the representational power of the network by modelling channel-wise relationships in a computationally efficient manner, unlike prior trunk-and-mask attention mechanisms based on hourglass modules.

“The latter can be viewed as an interesting blend between the hard and soft attention models proposed in (Xu et al., 2015): it is computationally less expensive than the global model or the soft attention; at the same time, unlike the hard attention, the local attention is differentiable, making it…”

“Xu et al. (2015) propose a doubly attentional approach with an additional constraint added to the training objective to make sure the model pays equal attention to all parts of the image during the caption generation process.”

“, 2014), or between visual features of a picture and its text description in the image caption generation task (Xu et al., 2015).”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.
• The local attention can be viewed as an interesting blend between the hard and soft attention models: computationally less expensive than global attention or soft attention, yet differentiable almost everywhere unlike hard attention.
• The global attention approach is simpler architecturally than Bahdanau et al.'s model, using only top-layer hidden states and a simpler computation path.

“The attention mechanism is widely used in deep learning for image captioning (Xu et al., 2015) or text analysis (Bahdanau et al., 2014; Lin et al., 2017).”

“MIL and attention The attention mechanism is widely used in deep learning for image captioning (Xu et al., 2015) or text analysis (Bahdanau et al.”

• Introduces two attention-based image caption generators under a common framework: a soft deterministic attention mechanism trainable by standard back-propagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound or via REINFORCE.
• Shows how attention enables visualization/interpretation of 'where' and 'what' the model attends to when generating each word.
• Does not explicitly use object detectors but instead learns latent alignments from scratch, allowing the model to attend to abstract, non-object salient regions.

• Proposes a fully trainable, adaptive MIL pooling operator based on attention, replacing pre-defined non-trainable operators like max and mean.
• Introduces a gated attention mechanism combining tanh and sigmoid non-linearities to overcome the near-linearity of tanh and better capture complex relations among instances.
• Applies attention without sequential dependency among instances (unlike typical sequence-based attention), questioning whether attention can work in this independent-instance setting.

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

• Introduces intra-attention (self-attention) applied within a single sequence encoder to induce undirected lexical relations among tokens, as opposed to prior attention work used mainly for inter-sequence alignment (e.g., encoder-decoder attention).
• Embeds the memory network and attention mechanism internal to the LSTM recurrence itself (replacing the single memory cell), rather than as an external memory module, strengthening interaction between memory and attention.
• Proposes 'deep attention fusion' and 'shallow attention fusion' methods to combine intra-attention with inter-attention in an encoder-decoder setting.

“In [8], the authors propose an attention model to circumvent the bottleneck caused by fixed width hidden vector in text reading and comprehension.”

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

• Proposes a novel co-attention mechanism jointly performing question-guided visual attention and image-guided question attention, with two strategies: parallel and alternating co-attention
• Proposes a hierarchical architecture representing the question at word, phrase, and question levels, constructing co-attention maps at each level and recursively combining them for answer prediction
• Proposes a novel convolution-pooling strategy at the phrase level to adaptively select phrase sizes

“We also evaluate our model on the task of cloze-style reading comprehension using the CNN and Daily Mail datasets (Hermann et al., 2015).”

“Massive cloze test datasets (CNN/DailyMail by Hermann et al. (2015) and Childrens Book Test by Hill et al. (2016)), enabled the application of deep neural architectures to this task.”

“Hermann et al. (2015) have recently compiled a massive Cloze-style comprehension dataset, consisting of 300k/4k/3k and 879k/65k/53k (train/dev/test) examples from CNN and DailyMail news articles, respectively.”

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

• Our attention layer is not used to summarize the context paragraph into a fixed-size vector; instead the attention is computed for every time step and allowed to flow through to the modeling layer, reducing information loss from early summarization.
• We use a memory-less attention mechanism, where attention at each time step is a function of only the query and context at that time step and does not directly depend on the attention at the previous time step, unlike dynamic attention models.
• We use attention mechanisms in both directions, query-to-context and context-to-query, which provide complementary information to each other, in contrast to previous uni-directional attention approaches.

“, 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al.”

“Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.”

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

• We propose an efficient reading comprehension model that exclusively built upon convolutions and self-attentions... To the best of our knowledge, we are the first to do so.
• the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments
• convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words

“The CNN/Daily Mail [Hermann et al., 2015] dataset was introduced as a question-answering task but was adapted for text summarization by Nallapati et al.”

“Furthermore, while CNN/Daily Mail is posed as an abstractive summarization benchmark, purely extractive approaches have been shown to work well [Liu, 2019].”

“For CNN/Daily Mail, we use the standard test set distributed with the dataset.”

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

“Nallapati et al. (2016) also applied their abstrac-tive summarization model on the CNN/Daily Mail dataset (Hermann et al., 2015), which contains input sequences of up to 800 tokens and multi-sentence summaries of up to 100 tokens.”

“Neural encoder-decoder models are widely used in NLP applications such as machine translation (Sutskever et al., 2014), summarization (Chopra et al., 2016; Nallapati et al., 2016), and question answering (Hermann et al., 2015).”

“We evaluate our model on a modified version of the CNN/Daily Mail dataset (Hermann et al., 2015), following the same pre-processing steps described in Nallapati et al. (2016).”

• This paper proposes a finer grained token-level attention mechanism (as opposed to sentence-level attention in Memory Networks) applied to reading comprehension.
• Introduces the Impatient Reader, which recurrently accumulates attention-based document representations as each query token is read, allowing the model to reread the document.
• Demonstrates via heatmap visualization that the attention mechanism enables lexical generalization and coreference resolution for machine reading, and argues attention is the key contributor to improved performance over non-attentive LSTM models.

• We introduce a key attention mechanism...: (i) we use an intra-temporal attention in the encoder that records previous attention weights for each of the input tokens while a sequential intra-attention model in the decoder takes into account which words have already been generated by the decoder.
• This mechanism is not present in existing encoder-decoder models for abstractive summarization.
• A closely-related intra-RNN attention function has been introduced by <cit.> but their implementation works by modifying the underlying LSTM function, and they do not apply it to long sequence generation problems. This is a major difference with our method, which makes no assumptions about the type of decoder RNN, thus is more simple and widely applicable to other types of recurrent networks.