Soft Alignment in Sequence-to-Sequence Models

The paper demonstrates that LSTM can use its memory to generate complex, realistic sequences containing long-range structure.

“The main goal of this paper is to demonstrate that LSTM can use its memory to generate complex, realistic sequences containing long-range structure.”◌ not checked against the paper’s text as it now stands

The paper shows that LSTM architecture with purpose-built memory cells is better at finding and exploiting long-range dependencies in data compared to standard RNNs with sigmoid hidden layer functions.

“However we have found that the Long Short-Term Memory (LSTM) architecture […], which uses purpose-built memory cells to store information, is better at finding and exploiting long range dependencies in the data.”◌ not checked against the paper’s text as it now stands

The paper extended the prediction network to condition its outputs on a sequence with unknown alignment, making it suitable for handwriting synthesis.

The paper introduced an attention mechanism that allows the decoder to selectively focus on relevant parts of the source sentence when generating each target word.

“Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to.”◌ not checked against the paper’s text as it now stands

The paper showed that attention weights can be visualized to reveal interpretable alignments between source and target words.

“The proposed approach provides an intuitive way to inspect the between the words in a generated translation and those in a source sentence. This is done by visualizing the annotation weights α_ij from Eq. […], as in Fig. […].”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the soft attention mechanism enables the model to handle non-monotonic word alignments and phrases of different lengths without explicit NULL mappings.

The paper showed that attention allows the model to better translate long sentences by avoiding the need to compress all information into a fixed-length vector.

The paper introduced an attention mechanism for speech recognition that allows the decoder to select relevant input frames dynamically based on the decoder's hidden state.

“At each step of decoding, the model first scores all input frames against its hidden state to soft-select relevant input frames.”◌ not checked against the paper’s text as it now stands

The paper designed an attention mechanism with a learned gating function that encourages monotonic alignment and prevents the model from matching repeated phonemes at non-adjacent locations.

“The gating procedure prevents this behavior by confining the search to locations that are near the inputs relevant to the previously generated symbol.”◌ not checked against the paper’s text as it now stands

The paper implemented attention weights as a normalized weighted sum of input annotations, where each annotation is selected with a learned certainty value.

“The context is a weighted sum of annotations: c_o = ∑_i=1^I α_o,i h_i, where α_o,i is a normalized weight for each annotation h_i. This effectively means that the decoder selects each annotation h_i with a certainty α_o,i.”◌ not checked against the paper’s text as it now stands

The paper introduced two variants of attention mechanisms for image caption generation: a soft attention mechanism trainable by backpropagation and a hard stochastic attention mechanism trainable by maximizing a variational lower bound.

The paper demonstrated that attention allows the model to dynamically focus on salient features of an image rather than compressing the entire image into a static representation.

“Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed.”◌ not checked against the paper’s text as it now stands

The paper showed through visualization that the model learns to fix its gaze on salient objects while generating corresponding words, enabling interpretation of where and what the model attended to.

“We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence.”◌ not checked against the paper’s text as it now stands

The paper validated attention through a doubly stochastic regularization technique that encourages the model to pay attention to different parts of the image over the course of generation.

The paper developed attention-based deep neural networks for reading comprehension that can focus on relevant parts of documents to answer questions.

“This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.”◌ not checked against the paper’s text as it now stands

The paper implemented an attention mechanism that allows models to identify which tokens in a document are most important for answering a query.

“These weights are interpreted as the degree to which the network attends to a particular token in the document when answering the query”◌ not checked against the paper’s text as it now stands

The paper proposed an Impatient Reader model with attention that can reread the document as each query token is processed.

“The Attentive Reader is able to focus on the passages of a context document that are most likely to inform the answer to the query. We can go further by equipping the model with the ability to reread from the document as each query token is read.”◌ not checked against the paper’s text as it now stands

The paper showed that attention-based models outperformed pure LSTM models on the reading comprehension task.

This paper extended attention mechanisms with location-awareness features specifically designed for speech recognition tasks.

“We extend the attention-mechanism with features needed for speech recognition.”◌ not checked against the paper’s text as it now stands

This paper proposed a hybrid attention mechanism that combines both content-based and location-based information through convolutional features to make attention more robust to long inputs.

“we propose to modify the attention mechanism such that it explicitly takes into account both (a) the location of the focus from the previous step, as in […] and (b) the features of the input sequence, as in […]. This is achieved by adding as inputs to the attention mechanism auxiliary convolutional features which are extracted by convolving the attention weights from the previous step with trainable filters.”◌ not checked against the paper’s text as it now stands

This paper introduced a smoothing modification to the attention mechanism using sigmoid normalization to prevent the model from concentrating too much on single frames.

“we replace the unbounded exponential function of the softmax function in Eq. […] with the bounded logistic sigmoid σ such that a_i,j=σ(e_i,j)/∑_j=1^Lσ(e_i,j).. This has the effect of smoothing the focus found by the attention mechanism.”◌ not checked against the paper’s text as it now stands

The paper introduced global attention, which attends to all source words by computing alignment weights between the current target state and each source state.

The paper proposed local attention as a more efficient alternative that focuses on only a subset of source positions rather than all positions.

“To address this deficiency, we propose a local attentional mechanism that chooses to focus only on a small subset of the source positions per target word.”◌ not checked against the paper’s text as it now stands

The paper developed the input-feeding approach where attention vectors are fed as inputs to subsequent time steps to inform the model about past alignment decisions.

The paper explored multiple alignment functions for computing attention weights, including content-based functions and location-based functions.

The paper introduced an attention-based encoder that learns a soft alignment between input and summary generation to help inform the abstractive summarization process.

“Our encoder is modeled off of the attention-based encoder of […] in that it learns a latent soft alignment over the input text to help inform the summary”◌ not checked against the paper’s text as it now stands

The paper demonstrated that attention mechanisms enable the model to weight different parts of the input based on generation context rather than using a uniform distribution.

“Informally we can think of this model as simply replacing the uniform distribution in bag-of-words with a learned soft alignment, 𝐏, between the input and the summary.”◌ not checked against the paper’s text as it now stands

The paper showed that the attention-based encoder can selectively attend to relevant input regions when constructing representations for summary generation.

This paper proposed a simplified attention mechanism that can be applied to feed-forward neural networks rather than only recurrent networks.

“We propose a simplified model of attention which is applicable to feed-forward neural networks”◌ not checked against the paper’s text as it now stands

This paper demonstrated that attention enables models to refer to specific points in a sequence when computing output.

“Our experiments explicitly demonstrate that including an attention mechanism can allow a model to refer to specific points in a sequence when computing its output.”◌ not checked against the paper’s text as it now stands

This paper showed that attention provides a clear benefit over unweighted averaging for handling long-term dependencies in sequences.

This paper provided evidence that attention helps models handle very long and widely variable-length sequences.

“They also provide an alternate argument for the claim made by that attention helps models handle very long and widely variable-length sequences.”

This paper uses attention as a mechanism for the model to learn which past tokens in memory relate to the current token being processed.

“The model processes text incrementally while learning which past tokens in the memory and to what extent they relate to the current token being processed.”◌ not checked against the paper’s text as it now stands

This paper employs attention within the LSTM recurrence to induce relations between tokens in a soft and differentiable manner.

“A key idea behind the LSTMN is to use attention for inducing relations between tokens. These relations are soft and differentiable, and components of a larger representation learning network.”◌ not checked against the paper’s text as it now stands

This paper extends the LSTM architecture with a memory network and uses neural attention to enable adaptive memory addressing during recurrence.

“This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens.”◌ not checked against the paper’s text as it now stands

This paper introduced an attention mechanism that allows the decoder to focus on different regions of the source sentence during decoding.

“The encoder and decoder are connected through an attention module which allows the decoder to focus on different regions of the source sentence during the course of decoding.”◌ not checked against the paper’s text as it now stands

This paper designed an attention mechanism where the bottom decoder layer connects to the top encoder layer to maximize parallelism during decoding.

“For the attention portion of the model, we chose to align the bottom decoder output to the top encoder output to maximize parallelism when running the decoder network.”◌ not checked against the paper’s text as it now stands

This paper used attention mechanism as part of the solution to improve handling of rare words in neural machine translation.

The paper introduced a bi-directional attention mechanism that computes attention in both query-to-context and context-to-query directions to provide complementary information.

“Third, we use attention mechanisms in both directions, query-to-context and context-to-query, which provide complimentary information to each other.”◌ not checked against the paper’s text as it now stands

The paper proposed an attention flow mechanism that allows attention vectors to flow through to subsequent layers rather than summarizing context into fixed-size vectors early.

“Instead, the attention vector at each time step, along with the embeddings from previous layers, are allowed to flow through to the subsequent modeling layer. This reduces the information loss caused by early summarization.”◌ not checked against the paper’s text as it now stands

The paper developed a memory-less attention mechanism where attention at each time step depends only on the current query and context, not on previous attention states.

“Second, we use a memory-less attention mechanism. That is, while we iteratively compute attention through time as in […], the attention at each time step is a function of only the query and the context paragraph at the current time step and does not directly depend on the attention at the previous time step.”◌ not checked against the paper’s text as it now stands

The paper introduced biaffine attention as an improvement over MLP-based and bilinear attention mechanisms for dependency parsing.

“we use biaffine attention instead of bilinear or traditional MLP-based attention”◌ not checked against the paper’s text as it now stands

The paper demonstrated that deep biaffine attention outperforms shallow bilinear attention and MLP-based attention in both accuracy and speed.

“What we see is that the deep bilinear model outperforms the others with respect to both speed and accuracy.”◌ not checked against the paper’s text as it now stands

The paper showed that applying dimension-reducing MLPs before the biaffine attention helps by removing superfluous information and reducing overfitting.

The paper modeled the biaffine attention mechanism to directly capture both the prior probability and the likelihood of specific dependencies in the parsing task.

“this has the conceptual advantage of directly modeling both the prior probability of a word j receiving any dependents in the term 𝐫_j^⊤𝐮^(2) and the likelihood of j receiving a specific dependent i in the term 𝐫^⊤_jU^(1)𝐫_i”

This paper proposed an end-to-end spatial and temporal attention model that learns to selectively focus on discriminative joints of skeleton within each frame and pays different levels of attention to outputs of different frames.

“we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames.”◌ not checked against the paper’s text as it now stands

This paper designed a spatial attention module with joint-selection gates that automatically allocates different attentions to different joints of the input skeleton within each frame.

“A spatial attention module with joint-selection gates is designed to adaptively allocate different attentions to different joints of the input skeleton within each frame.”◌ not checked against the paper’s text as it now stands

This paper designed a temporal attention module with frame-selection gates that allocates different attentions to different frames instead of simply skipping frames.

“A temporal attention module with frame-selection gate is designed to allocate different attentions to different frames.”

The paper equipped each decoder layer with a separate attention module that only adds negligible overhead compared to recurrent approaches.

“we equip each decoder layer with a separate attention module”◌ not checked against the paper’s text as it now stands

The paper introduced multi-step attention where attention is computed at each decoder layer, allowing the model to refocus on different source elements at different depths.

The paper's convolutional architecture allows batching attention computation across all sequence elements, unlike RNNs which compute attention sequentially.

“Our convolutional architecture also allows to batch the attention computation across all elements of a sequence compared to RNNs (Figure 1, middle).”◌ not checked against the paper’s text as it now stands

The paper introduced a novel intra-attention mechanism that attends over both the input and continuously generated output separately to address repetition problems in abstractive summarization.

“We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately”◌ not checked against the paper’s text as it now stands

The paper developed an intra-decoder attention mechanism that allows the decoder to look back at previous decoding steps to avoid generating repeated phrases.

“To prevent that, we can incorporate more information about the previously decoded sequence into the decoder. Looking back at previous decoding steps will allow our model to make more structured predictions and avoid repeating the same information”◌ not checked against the paper’s text as it now stands

The paper demonstrated that intra-attention improves performance, with the model achieving higher ROUGE scores when intra-attention was included compared to without it.

“ML, no intra-attention 37.86 14.69 34.99 ML, with intra-attention 38.30 14.81 35.49”◌ not checked against the paper’s text as it now stands

The paper proposes using an attention mechanism to improve time series classification and allow visualization of the LSTM cell's decision process.

“We also explore the usage of attention mechanism to improve time series classification with the Attention Long Short Term Memory Fully Convolutional Network (ALSTM-FCN). Utilization of the attention mechanism allows one to visualize the decision process of the LSTM cell.”◌ not checked against the paper’s text as it now stands

The paper explains that the attention mechanism computes a weighted context vector by scoring how well input positions match output positions using an alignment model.