Visual & Image Embeddings

The paper introduced DeCAF, a feature representation extracted from the activations of a deep convolutional network trained on ImageNet that can be applied to novel visual recognition tasks.

“Our model can either be considered as a deep architecture for transfer learning based on a supervised pre-training phase, or simply as a new visual feature DeCAF defined by the convolutional network weights learned on a set of pre-defined object recognition tasks.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that deep convolutional activation features show superior semantic clustering compared to conventional features like GIST and LLC.

“Furthermore, other features such as GIST or LLC fail to capture the semantic difference in the image (although they show interesting clustering structure).”◌ not checked against the paper’s text as it now stands

The paper showed that DeCAF features generalize well across different datasets and tasks, maintaining semantic clustering even on datasets different from the training set.

“Even there, the features show very good clustering of semantic classes (e.g., indoor vs. outdoor). This suggests DeCAF is a good feature for general object recognition tasks.”◌ not checked against the paper’s text as it now stands

The paper released DeCAF as an open-source implementation with pre-trained network parameters to enable researchers to extract deep convolutional features without retraining.

“We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.”◌ not checked against the paper’s text as it now stands

This paper presented Caffe as a framework that enables efficient extraction of semantic features from images using pre-trained deep neural networks.

“In addition to end-to-end training, Caffe can also be used to extract semantic features from images using a pre-trained network.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that Caffe can create learned embeddings of images that show clear semantic separation between object categories.

“Figure […] shows a two-dimensional embedding of all the ImageNet validation images, colored by a coarse category that they come from. The nice separation testifies to a successful semantic embedding.”◌ not checked against the paper’s text as it now stands

This paper showed that Caffe-based feature embeddings are useful beyond object classification, such as for finding images by visual style attributes.

This paper demonstrated that convolutional network representations learned through deep architectures generalize well to other datasets beyond ImageNet.

“We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results.”◌ not checked against the paper’s text as it now stands

This paper showed that deep features from their networks can be effectively used as embeddings for image recognition tasks when combined with simple classifiers.

“they achieve excellent performance even when used as a part of a relatively simple pipelines (deep features classified by a linear SVM without fine-tuning).”◌ not checked against the paper’s text as it now stands

This paper produced embeddings by using stacks of small 3×3 convolutional filters which increased the non-linearity and discriminativeness of the learned representations.

“First, we incorporate three non-linear rectification layers instead of a single one, which makes the decision function more discriminative.”◌ not checked against the paper’s text as it now stands

This paper proposes a compatibility function between input and output embeddings for zero-shot image classification.

“Given image and class embeddings, we learn a compatibility function such that matching embeddings are assigned a higher score than mismatching ones; zero-shot classification of an image proceeds by finding the label yielding the highest joint compatibility score.”◌ not checked against the paper’s text as it now stands

This paper evaluates multiple types of output embeddings including attributes, word embeddings from text corpora, and hierarchical embeddings from WordNet.

“We use state-of-the-art image features and focus on different supervised attributes and unsupervised output embeddings either derived from hierarchies or learned from unlabeled text corpora.”◌ not checked against the paper’s text as it now stands

This paper demonstrates that unsupervised output embeddings learned from text can achieve competitive or better results than supervised attribute-based methods.

“Most encouragingly, we demonstrate that purely unsupervised output embeddings (learned from Wikipedia and improved with fine-grained text) achieve compelling results, even outperforming the previous supervised state-of-the-art.”

The paper introduced a multimodal joint embedding space that unifies images and text representations learned through an encoder-decoder pipeline.

“we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the learned embedding space captures multimodal regularities expressible through vector space arithmetic, such as image analogies.

“we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars”◌ not checked against the paper’s text as it now stands

The paper learned distributed representations by projecting image features from a deep convolutional network into the embedding space of LSTM hidden states using a pairwise ranking loss.

“we learn a joint image-sentence embedding where sentences are encoded using long short-term memory (LSTM) recurrent neural networks. Image features from a deep convolutional network are projected into the embedding space of the LSTM hidden states. A pairwise ranking loss is minimized in order to learn to rank images and their descriptions”

FaceNet directly learns a mapping from face images to a compact Euclidean space where distances correspond to face similarity, enabling easy implementation of recognition, verification, and clustering tasks.

“we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity. Once this space has been produced, tasks such as face recognition, verification and clustering can be easily implemented using standard techniques with FaceNet embeddings as feature vectors.”◌ not checked against the paper’s text as it now stands

FaceNet optimizes the embedding itself directly using a triplet-based loss function rather than using an intermediate bottleneck layer, achieving greater representational efficiency.

“Our method uses a deep convolutional network trained to directly optimize the embedding itself, rather than an intermediate bottleneck layer as in previous deep learning approaches.”◌ not checked against the paper’s text as it now stands

FaceNet introduced the concept of harmonic embeddings and harmonic triplet loss to allow different face embeddings produced by different networks to be compatible and directly compared.

“We also introduce the concept of harmonic embeddings, and a harmonic triplet loss, which describe different versions of face embeddings (produced by different networks) that are compatible to each other and allow for direct comparison between each other.”

This paper introduced a trainable NetVLAD layer that integrates VLAD pooling into CNN architectures for place recognition.

“we propose to mimic VLAD in a CNN framework and design a trainable generalized VLAD layer, NetVLAD. The result is a powerful image representation trainable end-to-end on the target task (in our case place recognition).”◌ not checked against the paper’s text as it now stands

This paper made VLAD differentiable for backpropagation by replacing hard cluster assignment with soft assignment using exponential weighting.

“To make this operation differentiable, we replace it with soft assignment of descriptors to multiple clusters a̅_k( x_i) = e^-α x_i- c_k^2/∑_k'e^-α x_i- c_k'^2”◌ not checked against the paper’s text as it now stands

This paper developed an end-to-end learning procedure for place recognition embeddings using weakly supervised training data from Google Street View Time Machine.

“we develop a learning procedure for place recognition that learns parameters of the architecture in an end-to-end manner tailored for the place recognition task from the weakly labelled Time Machine imagery.”◌ not checked against the paper’s text as it now stands

PlaNet learns feature representations from images that can be used for image retrieval tasks.

“Moreover, we show that the features learned by PlaNet can be used for image retrieval and achieve state-of-the-art results on the INRIA Holidays dataset.”◌ not checked against the paper’s text as it now stands

PlaNet uses a CNN to learn embeddings that represent geographical information about images through its classification approach.

The paper developed a character-level text encoder to learn discriminative text feature representations for conditioning image generation.

“we develop a simple and effective approach for text-based image synthesis using a character-level text encoder and class-conditional GAN.”◌ not checked against the paper’s text as it now stands

The paper used a hybrid character-level convolutional-recurrent neural network to encode text descriptions into embeddings for the GAN.

“we always used a hybrid character-level convolutional-recurrent network.”◌ not checked against the paper’s text as it now stands

The paper projected text embeddings to lower dimensions and depth concatenated them with image feature maps in the generator and discriminator networks.

“The description embedding φ(t) is first compressed using a fully-connected layer to a small dimension (in practice we used 128) followed by leaky-ReLU and then concatenated to the noise vector z.”◌ not checked against the paper’s text as it now stands

The paper introduced text manifold interpolation to generate additional synthetic text embeddings by interpolating between pairs of training caption embeddings.

This paper proposed a siamese network that combines identification and verification losses to learn discriminative CNN embeddings for person re-identification.

“we propose a siamese network that has two losses: identification loss and verification loss. This network simultaneously learns a discriminative CNN embedding and a similarity metric, thus improving pedestrian retrieval accuracy.”◌ not checked against the paper’s text as it now stands

The paper learned embeddings by jointly optimizing cross-entropy loss for identity prediction and cross-entropy loss for similarity verification on image pairs.

“The proposed network is trained to minimize the three cross-entropy losses jointly.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that combining identification and verification losses outperforms using either loss individually across multiple network architectures.

“the fine-tuned CNN model with two kinds of losses outperforms the one trained individually. This result has been confirmed on the three different network structures.”◌ not checked against the paper’s text as it now stands

The paper introduced depthwise separable convolutions as the core building block for MobileNets to achieve substantial reduction in computational cost compared to standard convolutions.

“The standard convolution operation has the effect of filtering features based on the convolutional kernels and combining features in order to produce a new representation. The filtering and combination steps can be split into two steps via the use of factorized convolutions called depthwise separable convolutions for substantial reduction in computational cost.”◌ not checked against the paper’s text as it now stands

The paper developed width multiplier and resolution multiplier hyperparameters that allow efficient control of model size and computational cost by scaling network capacity.

“We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that MobileNets can produce effective learned representations across diverse applications including object detection, fine-grained classification, face attributes and large scale geo-localization.

“We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.”

The paper introduced a novel loss function that incorporates hard negatives to improve visual-semantic embeddings for cross-modal retrieval.

“Our main contribution is to incorporate hard negatives in the loss function. This was inspired by the use of hard negative mining in classification tasks […], and by the use of hard negatives for improving image embeddings for face recognition […]. Minimizing a loss function using hard negative mining is equivalent to minimizing a modified non-transparent loss function with uniform sampling. We extend the idea with the explicit introduction of hard negatives in the loss for multi-modal embeddings, without any additional cost of mining.”◌ not checked against the paper’s text as it now stands

The paper defined a Max of Hinges loss function that emphasizes the hardest negatives to improve embedding learning.

“To emphasize hard negatives we define our loss as ℓ_MH(i, c) = max_c^'[α + s(i,c^') - s(i,c)]_+ + max_i^'[α + s(i^',c) -s(i,c)]_+. Like Eq. […], this loss comprises two terms, one with i and one with c as queries. Unlike Eq. […], this loss is specified in terms of the hardest negatives, c^' and i^'. We refer to the loss in Eq. […] as Max of Hinges loss, and the loss in Eq. […] as Sum of Hinges loss.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the new loss function combined with fine-tuning and augmented data yields significant improvements in visual-semantic embedding performance on benchmark datasets.

This paper extended BigGAN with an encoder module to perform representation learning, creating BigBiGAN.

“Our approach,, builds upon the state-of-the-art BigGAN model, extending it to representation learning by adding an encoder and modifying the discriminator.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the learned encoder provides effective visual representation embeddings for downstream tasks on ImageNet.

“We show that BigBiGAN (BiGAN with BigGAN generator) matches the state of the art in unsupervised representation learning on ImageNet.”◌ not checked against the paper’s text as it now stands

The paper showed that a powerful generator is important for learning good representation embeddings, as varying generator capacity directly affected classification performance.

“These results confirm that a powerful image generator is indeed important for learning good representations via the encoder.”◌ not checked against the paper’s text as it now stands

The paper improved representation embeddings by adding unary loss terms to guide the optimization of the data and latent embeddings independently.

This paper introduced momentum contrast (MoCo) which encodes dictionary keys using a slowly evolving encoder updated via momentum averaging to maintain consistency.

“The keys are encoded by a slowly progressing encoder, driven by a momentum update with the query encoder.”◌ not checked against the paper’s text as it now stands

This paper maintains embeddings in a queue-based dictionary structure that decouples dictionary size from mini-batch size.

“We maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The queue decouples the dictionary size from the mini-batch size, allowing it to be large.”◌ not checked against the paper’s text as it now stands

This paper normalizes embeddings by L2-norm before use in contrastive learning.

“This output vector is normalized by its L2-norm. This is the representation of the query or key.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that embeddings learned through MoCo transfer effectively to downstream detection and segmentation tasks, sometimes surpassing supervised pre-training.

This paper introduced a learnable nonlinear projection head that substantially improves the quality of learned embeddings in contrastive learning.

“Introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that normalized embeddings are important for contrastive learning performance.

“Representation learning with contrastive cross entropy loss benefits from normalized embeddings and an appropriately adjusted temperature parameter.”◌ not checked against the paper’s text as it now stands

This paper showed that the representation before the projection head is much better than the representation after projection for downstream tasks.

“Furthermore, even when nonlinear projection is used, the layer before the projection head, $ h$, is still much better (>10\%) than the layer after, $ z=g( h)$”◌ not checked against the paper’s text as it now stands

The paper uses embeddings of images encoded into a representation space for contrastive learning.

“Images are encoded into a representation space, in which pairwise affinities are computed.”◌ not checked against the paper’s text as it now stands

The paper treats embeddings as query and key representations in the InfoNCE loss function for contrastive learning.

“Here q is a query representation, k^+ is a representation of the positive (similar) key sample, and {k^-} are representations of the negative (dissimilar) key samples.”◌ not checked against the paper’s text as it now stands

The paper improves embedding quality by replacing the output fc projection head with an MLP head during unsupervised training.

“we replace the fc head in MoCo with a 2-layer MLP head (hidden layer 2048-d, with ReLU). Note this only influences the unsupervised training stage; the linear classification or transferring stage does not use this MLP head.”◌ not checked against the paper’s text as it now stands

This paper proposes that embeddings should encode semantic structures discovered by clustering into the learned embedding space.

“it encodes semantic structures discovered by clustering into the learned embedding space.”◌ not checked against the paper’s text as it now stands

This paper introduces prototypes as representatives in the embedding space that are closer to semantically similar instances than to other prototypes.

This paper shows that prototypes learned through their method contain more information about image classes than instance-based embeddings.

“compared to instance features, the prototypes have a larger MI with the class labels due to the effect of clustering.”◌ not checked against the paper’s text as it now stands

This paper develops a contrastive loss that encourages representations to be closer to their assigned prototypes by dynamically estimating concentration levels per prototype.

“We propose ProtoNCE, a new contrastive loss which improves the widely used InfoNCE by dynamically estimating the concentration for the feature distribution around each prototype.”

This paper demonstrates that learned embeddings in contrastive learning should minimize mutual information between views while preserving task-relevant information.

“we use theoretical and empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact.”◌ not checked against the paper’s text as it now stands

This paper shows that embeddings produced by contrastive learning frameworks learn representations that focus on the shared information between views.

“the framework results in representations that focus on the shared information between views”◌ not checked against the paper’s text as it now stands

This paper establishes that optimal embeddings correspond to minimal sufficient statistics that contain only task-relevant information with minimal complexity.

“the optimal representation 𝐳^* encoded from 𝐱 is the minimal sufficient statistic with respect to 𝐲. This says a model built on top of 𝐳^* has all the information necessary to predict 𝐲 as accurately as if it were to access 𝐱. Furthermore, 𝐳^* maintains the smallest complexity, i.e., containing no other information besides that about 𝐲, which makes it more generalizable”◌ not checked against the paper’s text as it now stands

The paper proposes an objective function that learns embeddings by measuring and minimizing the cross-correlation matrix between outputs of two networks to make it close to the identity matrix.

“We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible.”◌ not checked against the paper’s text as it now stands

The method uses an embedding space with both an invariance term that makes embedding vectors of distorted versions similar and a redundancy reduction term that decorrelates vector components.

“This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that embeddings produced by the method benefit from very high-dimensional output vectors compared to other self-supervised methods.

“Intriguingly, strongly benefits from the use of very high-dimensional embeddings.”◌ not checked against the paper’s text as it now stands

The paper produces visual embeddings through discriminative self-supervised learning that combines DINO and iBOT losses to learn features at both image and patch levels.

“We learn our features with a discriminative self-supervised method that can be seen as a combination of DINO and iBOT losses with the centering of SwAV”◌ not checked against the paper’s text as it now stands

The paper demonstrates that self-supervised embeddings trained on curated data can produce general-purpose visual features that work across image distributions and tasks without fine-tuning.

“This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources.”◌ not checked against the paper’s text as it now stands

The paper creates a data pipeline that maps images to embeddings for deduplication and retrieval-based curation of the pretraining dataset.

“Images from curated and uncurated data sources are first mapped to embeddings. Uncurated images are then deduplicated before being matched to curated images.”◌ not checked against the paper’s text as it now stands