# Self-Supervised & Contrastive Embeddings

> One thread of 9 from the "embedding" map, covering the 13 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Latent Representation & Generative Spaces, Word & Text Embeddings, Visual & Image Embeddings.

Emerged from contrastive predictive coding learning compact distributed representations without labels (2018-2019), evolved into contrastive learning frameworks treating embedding as learned features maximized under similarity constraints (2019-2020), matured through momentum contrast, clustering, and redundancy reduction approaches (2020-2021).


**Representation Learning with Contrastive Predictive Coding** (2018) \cite{vandenoord2018representation}

What it did: Introduce contrastive learning in compact latent space  *(the tool's reading)*

CPC pioneered learning compact distributed latent vector representations through prediction and contrastive loss. It proposes "learning representations by predicting the future in a compact latent embedding space using autoregressive models combined with a contrastive (NCE-based) loss, rather than reconstructing raw high-dimensional inputs."  *(the tool's reading)*

> “In this paper we propose the following: first, we compress high-dimensional data into a much more compact latent embedding space in which conditional predictions are easier to model.”
>
> ✓ verified: found word for word in the paper's own text


**A Theoretical Analysis of Contrastive Unsupervised Representation Learning** (2019) \cite{arora2019theoretical}

> “In natural language processing (NLP), low-dimensional representations of text – called text embeddings – have been computed with unlabeled data […]. Often the embedding function is trained by using the embedding of a piece of text to predict the surrounding text […].”
>
> ✓ verified: found word for word in the paper's own text


**Data-Efficient Image Recognition with Contrastive Predictive Coding** (2019) \cite{henaff2019data}

> “Having trained an encoder network f_θ, a context network g_ϕ, and a set of linear predictors {W_k} using the CPC objective, we use the encoder to form a representation z=f_θ(x) of new observations x, and discard the rest.”
>
> ✓ verified: found word for word in the paper's own text


**Learning Representations by Maximizing Mutual Information Across Views** (2019) \cite{bachman2019learning}

What it did: Extend contrastive embeddings to multiple scales  *(the tool's reading)*

This work extended contrastive learning across multiple views and scales, advancing the embedding framework. It "Extends local DIM by predicting across independently-augmented views, at multiple scales simultaneously, and with a more powerful encoder, using embedding functions ϕ to map features into a space where linear (dot-product) matching approximates belief-shift computations."  *(the tool's reading)*

> “Parameters ndf and nrkhs determine encoder feature dimension and output dimension for the embedding functions ϕ_n(f_n). The embeddings ϕ_7(f_7) and ϕ_5(f_5) are computed by applying a small MLP via convolution.”
>
> ✓ verified: found word for word in the paper's own text


**Contrastive Multiview Coding** (2019) \cite{tian2019contrastive}

What it did: Generalize contrastive embeddings to multiple views  *(the tool's reading)*

CMC generalized contrastive embedding learning beyond temporal sequences to multiple arbitrary views of the same data. It "Extends contrastive multiview embedding learning to more than two views (core-view and full-graph paradigms), showing representation quality improves as number of views increases."  *(the tool's reading)*

> “Given a set of sensory views, a deep representation is learnt by bringing views of the same scene together in embedding space, while pushing views of different scenes apart.”
>
> ✓ verified: found word for word in the paper's own text


**Momentum Contrast for Unsupervised Visual Representation Learning** (2019) \cite{he2019momentum}

What it did: Enable scalable contrastive embeddings via momentum  *(the tool's reading)*

MoCo introduced a momentum-encoded dynamic dictionary that enabled larger-scale contrastive embedding learning. It presents "MoCo builds a dynamic dictionary with a queue and a moving-averaged (momentum) encoder to produce large and consistent sets of encoded representations (embeddings) for contrastive learning."  *(the tool's reading)*

> “In general, the query representation is q=f_q(x^q) where f_q is an encoder network and x^q is a query sample (likewise, k=f_k(x^k)).”
>
> ✓ verified: found word for word in the paper's own text


**Bootstrap your own latent: A new approach to self-supervised Learning** (2020) \cite{grill2020bootstrap}

> “Generative approaches to representation learning build a distribution over data and latent embedding and use the learned embeddings as image representations.”
>
> ✓ verified: found word for word in the paper's own text


**A Simple Framework for Contrastive Learning of Visual Representations** (2020) \cite{chen2020simple}

What it did: Improve embeddings via normalization and projection  *(the tool's reading)*

SimCLR advanced embedding quality through L2 normalization and a learnable projection head that separates the embedding space from the representation space. It shows "normalizing embeddings (L2 normalization/cosine similarity) together with an appropriately tuned temperature parameter substantially improves representation quality compared to using un-normalized embeddings."  *(the tool's reading)*

> “Representation learning with contrastive cross entropy loss benefits from normalized embeddings and an appropriately adjusted temperature parameter.”
>
> ✓ verified: found word for word in the paper's own text


**Improved Baselines with Momentum Contrastive Learning** (2020) \cite{chen2020improved}

> “Images are encoded into a representation space, in which pairwise affinities are computed.”
>
> ✓ verified: found word for word in the paper's own text


**Prototypical Contrastive Learning of Unsupervised Representations** (2020) \cite{li2020prototypical}

What it did: Encode semantic structure via prototype embeddings  *(the tool's reading)*

PCL introduced prototypical embeddings that encode semantic clustering structure beyond instance-level contrastive learning. "PCL implicitly encodes the semantic structure of data into the embedding space, unlike instance-wise contrastive learning which only preserves local smoothness."  *(the tool's reading)*

> “unsupervised visual representation learning aims to learn an embedding function f_θ (realized via a DNN) that maps X to V={v_1, v_2,...,v_n} with v_i=f_θ(x_i), such that v_i best describes x_i.”
>
> ✓ verified: found word for word in the paper's own text


**What Makes for Good Views for Contrastive Learning?** (2020) \cite{tian2020what}

> “(a) Schematic of multiview contrastive representation learning, where an image is split into two views, and passed through two encoders to learn an embedding where the views are close relative to views from other images.”
>
> ✓ verified: found word for word in the paper's own text


**Unsupervised Learning of Visual Features by Contrasting Cluster Assignments** (2020) \cite{caron2020unsupervised}

> “The feature is then projected to the unit sphere, i.e., 𝐳_nt = f_θ(𝐱_nt) / f_θ(𝐱_nt)_2.”
>
> ✓ verified: found word for word in the paper's own text


**Barlow Twins: Self-Supervised Learning via Redundancy Reduction** (2021) \cite{zbontar2021barlow}

What it did: Enable high-dimensional embeddings via redundancy reduction  *(the tool's reading)*

Barlow Twins advanced embedding learning by using redundancy reduction instead of normalization, allowing high-dimensional embeddings. "Barlow Twins benefits strongly from very high-dimensional embeddings, unlike other SSL methods (e.g. SimCLR, BYOL) whose performance saturates or degrades with increased embedding dimensionality."  *(the tool's reading)*

> “We call the output of the encoder the 'representations' and the output of the projector the 'embeddings'. The representations are used for downstream tasks and the embeddings are fed to the loss function of Barlow Twins.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Latent Representation & Generative Spaces** → **Self-Supervised & Contrastive Embeddings** (2018): Unsupervised latent learning
- **Word & Text Embeddings** → **Self-Supervised & Contrastive Embeddings** (2019): Contextual pre-training
- **Visual & Image Embeddings** → **Self-Supervised & Contrastive Embeddings** (2019): Unsupervised vision features