Self-Supervised & Contrastive Embeddings

This paper proposes Contrastive Predictive Coding, a method that learns embeddings by predicting future information in latent space using a contrastive loss.

“In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models.”◌ not checked against the paper’s text as it now stands

The paper introduces an architecture that compresses high-dimensional data into compact latent embeddings through an encoder before making predictions.

“first, we compress high-dimensional data into a much more compact latent embedding space in which conditional predictions are easier to model.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that CPC learns useful embeddings across multiple domains by showing strong performance on speech, images, text and reinforcement learning tasks.

“We apply the resulting model, Contrastive Predictive Coding (CPC) to widely different data modalities, images, speech, natural language and reinforcement learning, and show that the same mechanism learns interesting high-level information on each of these domains, outperforming other approaches.”◌ not checked against the paper’s text as it now stands

The paper shows that embeddings learned by CPC capture meaningful information, such as speaker identity and phonetic content in speech, which can be extracted with linear classifiers.

“Beyond phone classification, Table […] (bottom) shows the accuracy of performing speaker identity (out of 251) with a linear classifier from the same representation (we do not average utterances over time). Interestingly, CPCs capture both speaker identity and speech contents, as demonstrated by the good accuracies attained with a simple linear classifier”◌ not checked against the paper’s text as it now stands

This paper develops a theoretical framework for analyzing contrastive learning by introducing latent classes to formalize semantic similarity in unlabeled data.

The paper proves that representations learned through contrastive learning achieve good linear classification performance if the function class contains a function with low unsupervised loss and low within-class variance.

“We prove that if the function class is rich enough to capture the similarity of data using inner products of representations that also have small variance within classes, then the representations learned using contrastive learning will have good performance on classification tasks.”◌ not checked against the paper’s text as it now stands

The paper shows that learned representations can reduce the amount of labeled data needed for downstream classification tasks through generalization bounds.

“Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks.”◌ not checked against the paper’s text as it now stands

The paper provides theoretical analysis decomposing the unsupervised loss into components that separately account for class differentiation and within-class concentration, revealing when contrastive learning can succeed.

The paper improved CPC by designing an encoder architecture that learns embeddings of image patches which can be used for downstream classification tasks.

The paper demonstrated that CPC embeddings enable linear classification on ImageNet with state-of-the-art accuracy by learning predictable representations.

“We revisit CPC in terms of its architecture and training methodology, and arrive at a new implementation with a dramatically-improved ability to linearly separate image classes (from 48.7% to 71.5% Top-1 ImageNet classification accuracy, a 23% absolute improvement), setting a new state-of-the-art.”◌ not checked against the paper’s text as it now stands

The paper showed that CPC embeddings can be applied to full images for downstream recognition tasks, enabling data-efficient classification with limited labeled data.

“Note that while pre-training required that the encoder be applied to patches, for downstream recognition tasks we can apply it directly to the entire image.”◌ not checked against the paper’s text as it now stands

The paper enhanced embedding quality through architectural improvements including increased model capacity, layer normalization, and multi-directional spatial predictions instead of single-direction predictions.

The paper proposes learning embeddings by maximizing mutual information between features extracted from multiple views of a shared context.

“We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context.”◌ not checked against the paper’s text as it now stands

The paper extends embedding methods to predict features across multiple scales simultaneously, rather than between a single global and local scale.

“Second, we maximize mutual information between multiple feature scales simultaneously, rather than between a single global and local scale.”◌ not checked against the paper’s text as it now stands

The paper introduces mixture-based embeddings that capture multiple component representations for each input feature.

“We now extend our model to use mixture-based features. For each antecedent feature f_1, we compute a set of mixture features {f_1^1,..., f_1^k}, where k is the number of mixture components.”◌ not checked against the paper’s text as it now stands

This paper learns embeddings by training a feature encoding function such that views of the same scene map to nearby points in representation space while views of different scenes map to far apart points.

“we learn a feature embedding such that views of the same scene map to nearby points (measured with Euclidean distance in representation space) while views of different scenes map to far apart points”◌ not checked against the paper’s text as it now stands

This paper implements embeddings by extracting latent representations through encoders and computing their cosine similarity as a score.

“To extract compact latent representations of v_1 and v_2, we employ two encoders f_θ_1(·) and f_θ_2(·) with parameters θ_1 and θ_2 respectively. The latent representions are extracted as z_1=f_θ_1(v_1), z_2=f_θ_2(v_2). We compute their cosine similarity as score”◌ not checked against the paper’s text as it now stands

This paper shows that the learned embeddings can be concatenated across views to form the full representation of a scene.

“The encodings for each view may be concatenated to form the full representation of a scene”◌ not checked against the paper’s text as it now stands

MoCo enables learning visual embeddings by treating contrastive learning as a dictionary look-up task where encoded queries are matched against a dictionary of encoded keys.

“Contrastive learning […], and its recent developments, can be thought of as training an encoder for a dictionary look-up task”◌ not checked against the paper’s text as it now stands

MoCo builds embeddings using a queue-based dictionary that allows large embedding dictionaries decoupled from mini-batch size.

“At the core of our approach is maintaining the dictionary as a queue of data samples. This allows us to reuse the encoded keys from the immediate preceding mini-batches. The introduction of a queue decouples the dictionary size from the mini-batch size.”◌ not checked against the paper’s text as it now stands

MoCo maintains embedding consistency through a momentum-updated encoder that slowly evolves to keep key representations consistent despite dictionary evolution.

“a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, is proposed to maintain consistency.”◌ not checked against the paper’s text as it now stands

BYOL learns image representations by training an online network to predict the target network's representation of different augmentations of the same image.

BYOL uses a projection stage to embed representations into a smaller space before computing the prediction loss.

BYOL incorporates learned embeddings through both an encoder and a predictor network that operate on projected representations.

The paper introduced a learnable nonlinear transformation between the representation and the contrastive loss that substantially improves the quality of learned embeddings.

“Introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the representation before the nonlinear projection head contains significantly more useful information for downstream tasks than the embedding after the projection head.

The paper showed that L2 normalized embeddings with temperature scaling in the contrastive loss substantially improve representation quality compared to unnormalized embeddings.

“Representation learning with contrastive cross entropy loss benefits from normalized embeddings and an appropriately adjusted temperature parameter.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that embeddings learned through contrastive learning can be effectively encoded into a representation space where pairwise affinities are computed.

“Images are encoded into a representation space, in which pairwise affinities are computed.”◌ not checked against the paper’s text as it now stands

The paper shows that learned embeddings from the query and key representations are sufficient for computing contrastive loss, using an InfoNCE formulation where q is a query representation and k+ and k- are positive and negative key representations.

“Here q is a query representation, k^+ is a representation of the positive (similar) key sample, and {k^-} are representations of the negative (dissimilar) key samples.”◌ not checked against the paper’s text as it now stands

The paper improves embeddings by replacing the fc projection head with an MLP head that better transforms representations for contrastive learning.

“replacing the output fc projection head […] with an MLP head”◌ not checked against the paper’s text as it now stands

The paper demonstrates that embeddings produced by the momentum encoder maintain better representation consistency between current and earlier encoded keys in the queue.

The paper proposes an embedding function that encodes semantic structures discovered by clustering into the learned embedding space.

“it encodes semantic structures discovered by clustering into the learned embedding space.”◌ not checked against the paper’s text as it now stands

The paper uses prototypes as representative embeddings for groups of semantically similar instances to improve the embedding space.

The paper learns embeddings through a contrastive loss that enforces similarity between sample embeddings and their corresponding prototypes.

“construct a contrastive loss which enforces the embedding of a sample to be more similar to its corresponding prototypes compared to other prototypes.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that learned prototypes contain more information about image classes than instance-based embeddings alone.

“compared to instance features, the prototypes have a larger MI with the class labels due to the effect of clustering.”

The paper shows that contrastive learning creates representations that focus on the shared information between views, and that the choice of views controls what information the representation captures.

The paper demonstrates that reducing mutual information between views while preserving task-relevant information improves downstream task performance, revealing a reverse-U shaped relationship between mutual information and accuracy.

The paper proposes that optimal view selection should minimize mutual information between views subject to preserving all task-relevant information, formalizing this as the InfoMin principle.

The paper shows that data augmentation can be used to reduce mutual information between views, and that stronger augmentation leads to decreased mutual information and improved downstream classification accuracy.

“We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy.”◌ not checked against the paper’s text as it now stands

This paper introduced a method that computes visual feature embeddings by mapping image representations to prototype vectors online.

“We then compute a code 𝐪_nt from this feature by mapping 𝐳_nt to a set of K trainable prototypes vectors, {𝐜_1,…,𝐜_K}.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that embeddings learned through their method achieve competitive performance on visual understanding tasks without requiring explicit pairwise feature comparisons.

This paper showed that their embedding approach enables the learned representations to outperform supervised pretraining on transfer tasks.

“we improve the performance of self-supervised by +4.2% on ImageNet with a standard ResNet and outperforms supervised ImageNet pretraining on multiple downstream tasks.”◌ not checked against the paper’s text as it now stands

The paper proposes an embedding method that measures cross-correlation between embeddings from two networks to create invariant representations while reducing redundancy.

The method produces embeddings through a projector network that outputs very high-dimensional vectors, which uniquely benefits the approach compared to other SSL methods.

“Intriguingly, strongly benefits from the use of very high-dimensional embeddings.”◌ not checked against the paper’s text as it now stands

The paper's embedding loss function naturally avoids collapse by construction through its invariance and redundancy reduction terms applied to the cross-correlation matrix.

“distinguishes itself from other methods by its innovative loss function ℒ_ℬ𝒯: ℒ_ℬ𝒯≜∑_i (1-𝒞_ii)^2_invariance term + λ∑_i∑_j ≠ i𝒞_ij^2_redundancy reduction term where λ is a positive constant trading off the importance of the first and second terms of the loss”◌ not checked against the paper’s text as it now stands

The paper demonstrates that embeddings learned by this method transfer effectively to downstream tasks including image classification, object detection, and instance segmentation across multiple datasets.