Self-Attention & Transformers

This paper introduced a self-attention mechanism for sentence embeddings that replaces max pooling or averaging in sequential models.

“We propose a self-attention mechanism for these sequential models to replace the max pooling or averaging step.”◌ not checked against the paper’s text as it now stands

The paper's self-attention mechanism enables attention to be applied when there is only a single sentence as input without extra information.

“This enables attention to be used in those cases when there are no extra inputs.”◌ not checked against the paper’s text as it now stands

The paper's attention mechanism produces multiple weighted representations by computing multiple hops of attention across different aspects of a sentence.

“Thus, to represent the overall semantics of the sentence, we need multiple 𝐦's that focus on different parts of the sentence. Thus we need to perform multiple hops of attention.”◌ not checked against the paper’s text as it now stands

The paper proposed a penalization term to encourage diversity among the attention weight vectors across different hops.

“The embedding matrix M can suffer from redundancy problems if the attention mechanism always provides similar summation weights for all the r hops. Thus we need a penalization term to encourage the diversity of summation weight vectors across different hops of attention.”◌ not checked against the paper’s text as it now stands

The paper introduced the Transformer architecture based entirely on attention mechanisms rather than recurrence or convolution.

“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”◌ not checked against the paper’s text as it now stands

The paper designed scaled dot-product attention as a specific attention function implementation.

“We call our particular attention "Scaled Dot-Product Attention" (Figure […]). The input consists of queries and keys of dimension d_k, and values of dimension d_v. We compute the dot products of the query with all keys, divide each by √(d_k), and apply a softmax function to obtain the weights on the values.”◌ not checked against the paper’s text as it now stands

The paper developed multi-head attention to allow the model to attend to information from different representation subspaces.

“Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.”◌ not checked against the paper’s text as it now stands

This paper introduced graph attention networks that leverage masked self-attentional layers to compute node representations by attending over neighborhoods.

“We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations.”◌ not checked against the paper’s text as it now stands

This paper enabled implicit specification of different weights to different nodes in a neighborhood through attention mechanisms without requiring costly matrix operations.

“By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that attention mechanisms allow the model to assign different importances to nodes in the same neighborhood, improving model capacity over prior graph convolutional approaches.

“As opposed to GCNs, our model allows for (implicitly) assigning different importances to nodes of a same neighborhood, enabling a leap in model capacity.”

This paper applied self-attention as the core mechanism for image generation, replacing recurrent and convolutional networks.

“Following recent work on modeling text, however, we propose eschewing recurrent and convolutional networks in favor of the Image Transformer, a model based entirely on a self-attention mechanism.”◌ not checked against the paper’s text as it now stands

This paper introduced local self-attention to make attention computationally tractable for large images by restricting attention to local neighborhoods.

“By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks.”◌ not checked against the paper’s text as it now stands

This paper developed two distinct factorizations of local self-attention (1D and 2D) for image generation with different spatial conditioning patterns.

“In both 1D and 2D local attention, we mask attention weights in the query and memory blocks such that positions that have not yet been generated are ignored.”◌ not checked against the paper’s text as it now stands

This paper extended the self-attention mechanism to incorporate relative position representations between sequence elements as an alternative to absolute position encodings.

“In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that relative position representations improve machine translation quality, achieving 1.3 BLEU improvement over absolute position representations on English-to-German translation.

“On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively.”◌ not checked against the paper’s text as it now stands

This paper framed relative position representations in attention as a special case of relation-aware self-attention that can generalize to arbitrary graph-labeled inputs.

“We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.”◌ not checked against the paper’s text as it now stands

The paper uses self-attention as a global interaction mechanism in place of recurrent networks for reading comprehension.

“Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions.”◌ not checked against the paper’s text as it now stands

The paper employs multi-head attention with 8 heads to compute weighted sums across all positions based on query-key similarity.

“For the self-attention-layer, we adopt the multi-head attention mechanism defined in […] which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product. The number of heads is 8 throughout all the layers.”◌ not checked against the paper’s text as it now stands

The paper uses context-query attention to construct query-aware context vectors for each position in the paragraph.

“The additional context-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers.”◌ not checked against the paper’s text as it now stands

This paper introduced self-attention mechanisms into GANs to enable modeling of long-range dependencies across image regions.

“In this work, we propose Self-Attention Generative Adversarial Networks (SAGANs), which introduce a self-attention mechanism into convolutional GANs. The self-attention module is complementary to convolutions and helps with modeling long range, multi-level dependencies across image regions.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that self-attention allows generators to coordinate fine details across distant image locations rather than just local regions.

“In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other.”◌ not checked against the paper’s text as it now stands

The paper showed that attention mechanisms work better than alternative architectural components like residual blocks for improving GAN performance.

“Compared with residual blocks with the same number of parameters, the self-attention blocks also achieve better results. For example, the training is not stable when we replace the self-attention block with the residual block in 8×8 feature maps, which leads to a significant decrease in performance (, FID increases from 22.98 to 42.13).”

This paper proposes using self-attention to predict a dense relation matrix that serves as a surrogate for capturing object context in semantic segmentation.

“We use the conventional self-attention scheme […] to predict the dense relation matrix, which requires 𝒪(N^2) computation complexity.”◌ not checked against the paper’s text as it now stands

This paper introduces an interlaced sparse self-attention scheme that approximates the dense relation matrix using two sparse relation matrices to improve computational efficiency.

“we propose a new interlaced sparse self-attention scheme that significantly improves the efficiency while maintaining the performance via two sparse relation matrices to approximate the dense relation matrix.”◌ not checked against the paper’s text as it now stands

This paper applies self-attention mechanisms through global and local relation modules that together enable modeling relations between any two pixels in the image.

“We can propagate the information from all input positions to each output position with the combination of the global relation module and the local relation module.”◌ not checked against the paper’s text as it now stands

The paper builds both CTC and sequence-to-sequence models on top of the transformer self-attention architecture for lip reading.

“Both models are built on top of the transformer self-attention architecture”◌ not checked against the paper’s text as it now stands

The paper incorporates a novel dual attention mechanism in the sequence-to-sequence model that can operate over visual input only, audio input only, or both.

“the architecture incorporates a novel dual attention mechanism that can operate over visual input only, audio input only, or both”◌ not checked against the paper’s text as it now stands

The paper uses separate attention heads in the decoder to attend to video and audio embeddings independently before concatenating them.

This paper introduced sparse factorizations of the attention matrix to reduce computational complexity from quadratic to O(n√n) with sequence length.

“we introduce sparse factorizations of the attention matrix which reduce this to O(n √(n)).”◌ not checked against the paper’s text as it now stands

This paper demonstrated that factorized attention patterns can learn specialized sparse structures across different layers, enabling the network to adapt to different domains.

“Layers are able to learn a variety of specialized sparse structures, which may explain their ability to adapt to different domains.”◌ not checked against the paper’s text as it now stands

This paper showed that sparse attention patterns achieved competitive or better performance than full attention while being more computationally efficient.

“in addition to running significantly faster than full attention, sparse patterns also converged to lower error”◌ not checked against the paper’s text as it now stands

This paper proposed efficient GPU kernels that compute sparse attention operations by slicing and computing products in blocks to accelerate training.

The paper uses self-attention masks to control what context tokens can attend to when computing contextualized representations for different language modeling objectives.

“In order to control the access to the context of the word token to be predicted, we employ different masks for self-attention. In other words, we use masking to control how much context the token should attend to when computing its contextualized representation.”◌ not checked against the paper’s text as it now stands

The paper employs different mask matrices in the self-attention mechanism to determine which pairs of tokens can attend to each other for different language modeling tasks.

“the mask matrix 𝐌∈ℝ^|x| × |x| determines whether a pair of tokens can be attended to each other. We use different mask matrices 𝐌 to control what context a token can attend to when computing its contextualized representation”◌ not checked against the paper’s text as it now stands

The paper uses triangular mask matrices in self-attention to implement unidirectional left-to-right language modeling where tokens only attend to leftward context.

The paper introduced a method to identify important attention heads by using layer-wise relevance propagation to evaluate how much individual heads contribute to model predictions.

The paper characterized three interpretable roles that attention heads play: positional heads attending to adjacent tokens, syntactic heads attending to tokens in specific dependency relations, and heads focusing on rare words.

The paper developed a pruning method using stochastic gates and Hard Concrete distributions to remove redundant heads while maintaining translation quality.

The paper demonstrated that specialized heads with interpretable functions are pruned last, while the majority of heads can be removed with minimal performance loss.

The paper proposes convolutional self-attention to make the attention mechanism aware of local context in time series, improving the model's ability to distinguish between anomalies and patterns.

The paper identifies that canonical Transformer's point-wise dot-product self-attention is insensitive to local context, which makes the model prone to anomalies in time series.

“in the self-attention layers of canonical Transformer, the similarities between queries and keys are computed based on their point-wise values without fully leveraging local context like shape, as shown in Figure (a) and (b). Query-key matching agnostic of local context may confuse the self-attention module in terms of whether the observed value is an anomaly, change point or part of patterns”◌ not checked against the paper’s text as it now stands

The paper demonstrates through synthetic experiments that Transformer with proposed attention mechanisms can capture long-term dependencies better than LSTM-based approaches.

The paper proposes sparse attention that reduces memory complexity from quadratic to logarithmic per layer while maintaining information flow through multiple layers.

The paper exploits the structure of multi-head attention in transformers to partition attention head computations across GPUs for model parallelism.

“for the self attention block we exploit inherent parallelism in the multihead attention operation, partitioning the GEMMs associated with key (K), query (Q), and value (V) in a column parallel fashion such that the matrix multiply corresponding to each attention head is done locally on one GPU.”◌ not checked against the paper’s text as it now stands

The paper shows that splitting attention head operations across GPUs allows each GPU to compute its own attention heads without requiring immediate communication between GPUs.

The paper demonstrates that careful placement of layer normalization in attention-based models like BERT is critical for achieving improved performance as model size grows.

“We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased accuracies as the model grows.”◌ not checked against the paper’s text as it now stands

This paper uses self-attention as the primary building block of the Transformer architecture for their text-to-text models.

“The primary building block of the Transformer is self-attention […]. Self-attention is a variant of attention […] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”◌ not checked against the paper’s text as it now stands

This paper implements attention mechanisms with multiple independent heads whose outputs are concatenated for further processing.

This paper introduced axial attention as a generalization of self-attention that applies attention along a single axis of a multidimensional tensor rather than flattening it.

This paper demonstrated that axial attention achieves significant computational savings compared to standard self-attention by reducing complexity from O(N^2) to O(N^(d-1)/d) for d-dimensional tensors.

“for a d-dimensional tensor with shape N = N^1/d×⋯× N^1/d, axial attention saves a O(N^(d-1)/d) factor of resources over standard self-attention.”◌ not checked against the paper’s text as it now stands

This paper showed that stacking multiple axial attention layers along different axes allows the model to achieve full global receptive field while maintaining the computational benefits.

“Of course, a single layer of axial attention along some axis k does not have the full receptive field since it covers a single axis, but we will see in […] that stacking two axial attention layers allows the model to obtain a global receptive field.”◌ not checked against the paper’s text as it now stands

This paper enabled semi-parallel computation of context in autoregressive models by structuring axial attention so that the majority of computation can proceed in parallel during decoding without independence assumptions.

This paper replaces standard dot-product attention with locality-sensitive hashing to reduce computational complexity from quadratic to logarithmic in sequence length.

“we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O(L^2) to O(Llog L), where L is the length of the sequence.”◌ not checked against the paper’s text as it now stands

This paper proposes LSH attention that restricts the attention targets for each query to only those keys in the same hash bucket rather than all keys.

This paper introduces multi-round LSH attention using multiple hash functions in parallel to reduce the probability that similar items fall into different buckets.

“With hashing, there is always a small probability that similar items nevertheless fall in different buckets. This probability can be reduced by doing multiple rounds of hashing with n_rounds distinct hash functions {h^(1), h^(2), …}”◌ not checked against the paper’s text as it now stands

This paper modifies the causal masking for shared-QK attention to prevent a token from attending to itself, which would otherwise dominate the attention computation.

The paper proposes distilling the self-attention distributions of the teacher's last Transformer layer into the student model using KL-divergence.

“Specifically, we minimize the KL-divergence between the self-attention distributions of the teacher and student: ℒ_AT = 1/A_h|x|∑_a=1^A_h∑_t=1^|x|D_KL(𝐀^T_L,a,t∥𝐀^S_M,a,t)”◌ not checked against the paper’s text as it now stands

The paper introduces a novel self-attention value-relation transfer that distills the scaled dot-product between values in addition to traditional attention distributions.

“In addition to the attention distributions, we propose using the relation between values in the self-attention module to guide the training of the student. The value relation is computed via the multi-head scaled dot-product between values.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that distilling only the last Transformer layer's self-attention allows flexible student architectures without requiring layer-to-layer mapping.

“Different from previous works which transfer teacher's knowledge layer-to-layer, we only use the attention maps of the teacher's last Transformer layer. Distilling attention knowledge of the last Transformer layer allows more flexibility for the number of layers of our student models, avoids the effort of finding the best layer mapping.”

The paper introduces an attention mechanism that scales linearly with sequence length instead of quadratically, solving the computational limitation of standard self-attention.

The paper combines local windowed attention with task-motivated global attention to create a flexible sparse attention pattern.

“'s attention mechanism is a combination of a windowed local-context self-attention and an end task motivated global attention that encodes inductive bias about the task.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that its attention mechanism can serve as a drop-in replacement for standard self-attention in pretrained transformer models.

“'s attention mechanism can act as a drop-in replacement for the self-attention mechanism in pretrained Transformers, and leads to gains across a suite of document NLP tasks.”◌ not checked against the paper’s text as it now stands

The paper shows through ablation studies that both local and global attention types are essential for effective performance.

This paper proposes Synthesizer, which learns synthetic attention weights without token-token interactions as an alternative to dot product self-attention.

“we propose Synthesizer, a model that learns synthetic attention weights without token-token interactions.”◌ not checked against the paper’s text as it now stands

This paper demonstrates that random alignment matrices can perform competitively with standard Transformers, showing that token-token dependencies are not necessary for good performance.

“we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all.”◌ not checked against the paper’s text as it now stands

This paper shows that attention matrices can be synthesized through various parameterized functions that do not rely on pairwise dot products.

“This paper proposes Synthesizer, a new model that learns to synthesize the self-alignment matrix instead of manually computing pairwise dot products.”◌ not checked against the paper’s text as it now stands

The paper introduced the concept of 'in-context learning' to describe how language models use demonstrations in their context window to perform tasks without weight updates.

The paper demonstrated that larger models show improved ability to learn tasks from contextual information compared to smaller models.

The paper showed that few-shot performance increases more rapidly with model size than zero-shot performance, indicating larger models are more proficient at in-context learning.

“While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient at in-context learning.”◌ not checked against the paper’s text as it now stands

This paper introduced Visual Transformers that use self-attention to judiciously attend to different image parts based on context rather than treating all pixels equally.

“Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that transformers can relate spatially-distant concepts through self-attention in token-space, overcoming a key limitation of convolutions.

“3) relating spatially-distant concepts through self-attention in token-space.”◌ not checked against the paper’s text as it now stands

This paper showed that transformers with input-dependent weights can support visual tokens with variable meaning, enabling use of far fewer tokens than graph convolution approaches.

“Different from graph convolution, in a transformer, weights between tokens are input-dependent and computed as a key-query product: (𝐓_in𝐊) (𝐓_in𝐐)^T ∈ℝ^L× L. This allows us to use as few as 16 visual tokens, in contrast to hundreds of analogous nodes for graph-convolution approaches”◌ not checked against the paper’s text as it now stands

This paper proposed a linear self-attention mechanism that reduces the computational complexity of self-attention from O(n^2) to O(n) in both time and space.

“we introduce a novel mechanism that reduces self-attention to an O(n) operation in both space- and time-complexity: we decompose the original scaled dot-product attention into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.”◌ not checked against the paper’s text as it now stands

This paper demonstrated theoretically that the self-attention context mapping matrix can be approximated by a low-rank matrix.

“we show both theoretically and empirically that the stochastic matrix formed by self-attention can be approximated by a low-rank matrix.”◌ not checked against the paper’s text as it now stands

This paper introduced linear projection matrices to approximate attention with reduced dimensional projections of keys and values instead of the original full-dimensional versions.

“The main idea of our proposed linear self-attention (Figure […]) is to add two linear projection matrices E_i, F_i∈ℝ^n× k when computing key and value. We first project the original (n× d)-dimensional key and value layers KW_i^K and VW_i^V into (k × d)-dimensional projected key and value layers.”

The paper introduced deformable attention modules that attend to only a small set of key sampling points around a reference, addressing the limitation of standard Transformer attention modules in processing image feature maps.

“the deformable attention module only attends to a small set of key sampling points around a reference point, regardless of the spatial size of the feature maps”◌ not checked against the paper’s text as it now stands

The paper reduced the computational complexity of attention from quadratic to linear with respect to spatial size by using the deformable attention module in the encoder.

“When it is applied in DETR encoder, where N_q = HW, the complexity becomes O(HWC^2), which is of linear complexity with the spatial size.”◌ not checked against the paper’s text as it now stands

The paper extended deformable attention to handle multi-scale features naturally without requiring feature pyramid networks.

“Our proposed multi-scale deformable attention module can naturally aggregate multi-scale feature maps via attention mechanism, without the help of these feature pyramid networks.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that pure transformers can be applied directly to sequences of image patches for image classification without relying on convolutional networks.

“We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.”◌ not checked against the paper’s text as it now stands

The paper shows that multi-headed self-attention layers in Vision Transformer operate globally across all patches rather than being locally constrained like CNN layers.

“In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that large-scale pre-training enables the attention-based Vision Transformer to achieve competitive or superior performance compared to convolutional networks, suggesting that learned attention patterns can replace hand-crafted inductive biases.

“However, the picture changes if the models are trained on larger datasets (14M-300M images). We find that large scale training trumps inductive bias.”◌ not checked against the paper’s text as it now stands

This paper introduced a distillation token that allows transformers to learn from a teacher model through self-attention mechanisms.

“We add a new token, the distillation token, to the initial embeddings (patches and class token). Our distillation token is used similarly as the class token: it interacts with other embeddings through self-attention, and is output by the network after the last layer.”◌ not checked against the paper’s text as it now stands

This paper showed that the distillation token and class token interact through attention while maintaining distinct learned representations.

This paper demonstrated that the distillation token approach provides significant improvement over vanilla distillation methods for transformers.

“In contrast, our distillation strategy provides a significant improvement over a vanilla distillation baseline, as validated by our experiments in Section […].”◌ not checked against the paper’s text as it now stands

This paper leveraged multi-head self-attention mechanisms as the core component of vision transformers for processing image patches as token sequences.

The paper introduced Transformers with global self-attention mechanisms as encoders for medical image segmentation to overcome CNN limitations in modeling long-range dependencies.

“Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms”◌ not checked against the paper’s text as it now stands

The paper designed a hybrid CNN-Transformer encoder with a cascaded upsampler to combine global context from Transformers with high-resolution CNN features for precise localization.

“TransUNet employs a hybrid CNN-Transformer architecture to leverage both detailed high-resolution spatial information from CNN features and the global context encoded by Transformers.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that incorporating U-Net-like skip-connections with Transformer attention improves segmentation performance by recovering low-level spatial details.

“the self-attentive feature encoded by Transformers is then upsampled to be combined with different high-resolution CNN features skipped from the encoding path, for enabling precise localization”◌ not checked against the paper’s text as it now stands

This paper introduced nested attention mechanisms by embedding transformer blocks within patches to compute attention between visual words in addition to sentence-level attention.

The paper proposed using separate multi-head self-attention for local word-level features and global sentence-level features to better capture hierarchical attention patterns.

This work demonstrated that adding word-level attention within patches improves the effectiveness of visual transformers while maintaining computational efficiency.

“With a small increase of computation and memory cost, our TNT block can efficiently model the local structure information and achieve a much better trade-off between accuracy and complexity”◌ not checked against the paper’s text as it now stands

This paper introduced convolutional projections to replace linear projections in the attention mechanism to better model local spatial context.

“the linear projection prior to every self-attention block in the Transformer module is replaced with our proposed convolutional projection, which employs a s × s depth-wise separable convolution […] operation on an 2D-reshaped token map. This allows the model to further capture local spatial context and reduce semantic ambiguity in the attention mechanism.”◌ not checked against the paper’s text as it now stands

This paper replaced position-wise linear projections for multi-head self-attention with depth-wise separable convolutions to enable efficient subsampling of key and value matrices.

“Instead, we propose to replace the original position-wise linear projection for Multi-Head Self-Attention (MHSA) with depth-wise separable convolutions, forming the Convolutional Projection layer.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that convolutional projections in every transformer block eliminate the need for positional embeddings while maintaining attention performance.

“The introduction of Convolutional Projections for every Transformer block, combined with the Convolutional Token Embedding, gives us the ability to model local spatial relationships through the network. This built-in property allows dropping the position embedding from the network without hurting performance”

This paper replaced the self-attention mechanism used in Vision Transformers with a simple linear layer for patch interactions.

This paper demonstrated that removing attention layers makes training more stable and eliminates the need for batch-dependent normalization.

This paper showed that linear layers can serve as an interpretable alternative to attention for understanding patch interactions in vision models.

“Finally, another advantage of using a linear layer is that we can still visualize the interactions between patch embeddings, revealing filters that are similar to convolutions on the lower layers, and longer range in the last layers.”◌ not checked against the paper’s text as it now stands

SegFormer uses attention mechanisms in its hierarchical Transformer encoder, with lower layers producing local attention patterns and higher layers producing non-local attention patterns.

“SegFormer's encoder naturally produces local attentions which resemble convolutions at lower stages, while able to output highly non-local attentions that effectively capture contexts at Stage-4.”◌ not checked against the paper’s text as it now stands

SegFormer reduces the computational complexity of self-attention through sequence reduction, lowering it from O(N²) to O(N²/R).

SegFormer's MLP decoder combines both local and global attention from different encoder layers to create powerful representations.

“By aggregating the information from different layers, the MLP decoder combines both local and global attention.”◌ not checked against the paper’s text as it now stands

SegFormer benefits from the non-local attention properties of Transformers to achieve a larger receptive field without requiring complex context modules.

This paper demonstrated that depthwise convolution and self-attention can be unified through relative attention by expressing both as per-dimension weighted sums over receptive fields.

“both depthwise convolution and self-attention can be expressed as a per-dimension weighted sum of values in a pre-defined receptive field.”◌ not checked against the paper’s text as it now stands

This paper showed that combining static convolutional kernels with adaptive attention weights via relative attention allows a model to gain both translation equivariance and input-adaptive weighting properties.

“the attention weight A_i,j is decided jointly by the w_i-j of translation equivariance and the input-adaptive x_i^⊤ x_j, which can enjoy both effects depending on their relative magnitudes.”◌ not checked against the paper’s text as it now stands

This paper systematically studied how to stack convolution and attention layers vertically by showing that convolutional stages should appear before attention stages for better generalization and capacity.

“the overall trend is that the more convolution stages the model has, the smaller the generalization gap is.”◌ not checked against the paper’s text as it now stands

The paper proposes linear spatial reduction attention to reduce the computational complexity of the attention operation from quadratic to linear.

The paper demonstrates that linear SRA significantly reduces computation overhead by 22% while maintaining comparable accuracy.

“As reported in #3 and #4 in Tab. […], compared to SRA […], our LSRA significantly reduces the computation overhead (GFLOPs) of the model by 22%, while keeping a comparable top-1 accuracy on ImageNet (82.1% 82.0%), and only 1 point lower AP on COCO (43.6 44.6).”◌ not checked against the paper’s text as it now stands

The paper shows that attention mechanism improvements enable the model to achieve state-of-the-art performance across multiple vision tasks with reduced computational complexity.

“With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation.”◌ not checked against the paper’s text as it now stands

The paper introduces the MobileViT block that replaces local processing in convolutions with global processing using transformers to enable learning of global representations while maintaining spatial inductive bias.

The paper demonstrates that attention-based transformers can be applied within patches to encode inter-patch relationships, preserving spatial structure unlike standard Vision Transformers.

The paper shows that multi-head self-attention is more computationally efficient in MobileViT compared to standard Vision Transformers due to its different organization of patches.

The paper applies self-attention mechanisms to model long-range dependencies across patches while maintaining an effective receptive field equal to the entire image dimensions.