# Visual & Image Embeddings

> One thread of 9 from the "embedding" map, covering the 20 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Multimodal & Cross-Modal Embeddings, Self-Supervised & Contrastive Embeddings.

Emerged from treating deep CNN activations as visual feature vectors (2013-2014), evolved toward explicitly learned embedding spaces optimized for specific tasks like face recognition and image retrieval (2015-2016), eventually scaling to billion-scale similarity search and self-supervised visual representation learning (2017-2023).


**DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition** (2013) \cite{donahue2013decaf}

What it did: Embedding as visualization tool for feature validation  *(the tool's reading)*

DeCAF uses t-SNE to visualize high-dimensional CNN feature vectors, establishing embedding as a dimensionality-reduction technique for analyzing learned representations. The visualization confirms that deep network activations capture semantic structure.  *(the tool's reading)*

> “we run the t-SNE algorithm […] to find a 2-dimensional embedding of the high-dimensional feature space, and plot them as points colored depending on their semantic category in a particular hierarchy.”
>
> ✓ verified: found word for word in the paper's own text


**Caffe: Convolutional Architecture for Fast Feature Embedding** (2014) \cite{jia2014caffe}

What it did: Embedding as learned semantic clustering of images  *(the tool's reading)*

Caffe extends the visualization purpose to a functional one: deep networks extract semantic features that map images into low-dimensional spaces where similar images cluster. This shifts embedding from post-hoc analysis to an intrinsic property of trained networks.  *(the tool's reading)*

> “Figure […] shows a two-dimensional embedding of all the ImageNet validation images, colored by a coarse category that they come from. The nice separation testifies to a successful semantic embedding.”
>
> ✓ verified: found word for word in the paper's own text


**Very Deep Convolutional Networks for Large-Scale Image Recognition** (2014) \cite{simonyan2014very}

What it did: Deep embeddings generalize across datasets better than shallow ones  *(the tool's reading)*

VGGNet demonstrates that penultimate-layer activations from very deep networks (16-19 layers) serve as fixed-length feature vectors superior to shallower alternatives. This establishes depth as a key factor in embedding quality.  *(the tool's reading)*

> “we remove the last fully-connected layer (which performs 1000-way ILSVRC classification), and use 4096-D activations of the penultimate layer as image features, which are aggregated across multiple locations and scales.”
>
> ✓ verified: found word for word in the paper's own text


**Evaluation of output embeddings for fine-grained image classification** (2014) \cite{akata2014evaluation}

What it did: Class embeddings enable zero-shot learning without human labels  *(the tool's reading)*

This work decouples image embeddings from class embeddings, showing that unsupervised text-derived class representations (Word2Vec, GloVe) rival supervised ones. Combining multiple embedding types outperforms prior art, introducing modular embedding composition.  *(the tool's reading)*

> “Given a specific input embedding, we derive a prediction by maximizing the compatibility F over SJEas follows: f(x;w) = max_y ∈𝒴 F(x,y; w). The parameter vector w can be written as a D × E matrix W with D being the input embedding dimension and E being the output embedding dimension. This leads to the bi-linear form of the compatibility function: F(x,y;W) = θ(x)^⊤ W φ(y). Here, the input embedding is denoted by θ(x) and the output embedding by φ(y).”
>
> ✓ verified: found word for word in the paper's own text


**Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models** (2014) \cite{kiros2014unifying}

What it did: Embeddings unify vision and language in shared space  *(the tool's reading)*

This paper creates multimodal embeddings by jointly projecting image features and LSTM-encoded sentences into a common space, enabling image-text alignment and vector arithmetic. Embeddings now capture cross-modal semantic relationships.  *(the tool's reading)*

> “we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space.”
>
> ✓ verified: found word for word in the paper's own text


**FaceNet: A unified embedding for face recognition and clustering** (2015) \cite{schroff2015facenet}

What it did: Triplet loss directly optimizes embeddings for metric learning  *(the tool's reading)*

FaceNet introduces triplet loss to optimize embeddings end-to-end rather than via classification bottlenecks, achieving compact 128-dimensional spaces. This shifts embedding design from a byproduct of classification to a primary optimization objective.  *(the tool's reading)*

> “In this paper we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity.”
>
> ✓ verified: found word for word in the paper's own text


**NetVLAD: CNN architecture for weakly supervised place recognition** (2015) \cite{arandjelovic2015netvlad}

What it did: Task-specific embeddings learned end-to-end via triplet ranking  *(the tool's reading)*

NetVLAD extends triplet-loss embedding learning by making VLAD pooling differentiable and trainable within CNNs. Embeddings now incorporate explicit place-matching structure rather than generic semantic clustering.  *(the tool's reading)*

> “This is generally done by designing a function f which acts as the “image representation extractor”, such that given an image I_i it produces a fixed size vector f(I_i).”
>
> ✓ verified: found word for word in the paper's own text


**PlaNet - Photo Geolocation with Convolutional Neural Networks** (2016) \cite{weyand2016planet}

What it did: Classification-derived embeddings useful but suboptimal for retrieval  *(the tool's reading)*

PlaNet shows that embeddings learned for location classification transfer to image retrieval, yet notes Euclidean distance between such embeddings lacks inherent meaning. This reveals a gap between classification-derived and retrieval-optimized embeddings.  *(the tool's reading)*

> “We extract image embeddings from the final layer below the SoftMax layer (a 2048-dim. vector) and rank images by the Euclidean distance between their embedding vectors.”
>
> ✓ verified: found word for word in the paper's own text


**Generative Adversarial Text to Image Synthesis** (2016) \cite{reed2016generative}

What it did: Text embeddings condition image generation end-to-end  *(the tool's reading)*

This work integrates text embeddings into both generator and discriminator networks for synthesis, using embeddings to capture discriminative content while noise captures style. Embeddings now bridge text and image generation at the pixel level.  *(the tool's reading)*

> “To obtain a visually-discriminative vector representation of text descriptions, we follow the approach of […] by using deep convolutional and recurrent text encoders that learn a correspondence function with images.”
>
> ✓ verified: found word for word in the paper's own text


**A Discriminatively Learned CNN Embedding for Person Reidentification** (2016) \cite{zheng2016discriminatively}

What it did: Combined identification and verification losses improve embedding quality  *(the tool's reading)*

This work combines identification and verification losses simultaneously to learn more discriminative embeddings than either loss alone, using cross-entropy instead of contrastive loss. Dual objectives create richer embedding activation patterns.  *(the tool's reading)*

> “Our network learns a discriminative embedding and a similarity measurement at the same time, thus making full usage of the re-ID annotations.”
>
> ✓ verified: found word for word in the paper's own text


**MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications** (2017) \cite{howard2017mobilenets}

What it did: Knowledge distillation compresses embeddings into mobile models  *(the tool's reading)*

MobileNets transfers FaceNet's triplet-loss embeddings into smaller models via distillation, adapting rich embeddings for resource-constrained devices. Embeddings now optimize for both accuracy and model efficiency.  *(the tool's reading)*

> “It builds face embeddings based on the triplet loss.”
>
> ✓ verified: found word for word in the paper's own text


**VSE++: Improving Visual-Semantic Embeddings with Hard Negatives** (2017) \cite{faghri2017vse}

What it did: Hard negative mining improves vision-semantic embeddings dramatically  *(the tool's reading)*

VSE++ introduces Max of Hinges loss emphasizing hard negatives within mini-batches for image-caption embeddings, achieving 8-11% retrieval gains over prior work. Mining strategy becomes integral to embedding quality in multimodal spaces.  *(the tool's reading)*

> “Such embeddings entail mappings from two (or more) domains into a common vector space in which semantically associated inputs (e.g., text and images) are mapped to similar locations. The embedding space thus represents the underlying domain structure, where location and often direction are semantically meaningful.”
>
> ✓ verified: found word for word in the paper's own text


**Large Scale Adversarial Representation Learning** (2019) \cite{donahue2019large}

What it did: Adversarial representation learning scales unsupervised embeddings  *(the tool's reading)*

BigBiGAN matches supervised embeddings by training encoders and generators adversarially, showing that unsupervised encoder outputs serve as competitive downstream representations. Embeddings now achieve state-of-the-art via self-supervised adversarial training.  *(the tool's reading)*

> “The encoder ℰ models the inverse conditional distribution P(𝐳|𝐱), predicting latents 𝐳 given data 𝐱 sampled from the data distribution P_𝐱.”
>
> ✓ verified: found word for word in the paper's own text


**Momentum Contrast for Unsupervised Visual Representation Learning** (2019) \cite{he2019momentum}

What it did: Dictionary lookup with momentum encoder scales contrastive embeddings  *(the tool's reading)*

MoCo frames embedding learning as dictionary lookup with a dynamic queue and momentum encoder, producing large consistent embedding sets without end-to-end backprop. Normalization by L2-norm enables dot-product similarity in fixed-dimensional spaces.  *(the tool's reading)*

> “In general, the query representation is q=f_q(x^q) where f_q is an encoder network and x^q is a query sample (likewise, k=f_k(x^k)).”
>
> ✓ verified: found word for word in the paper's own text


**A Simple Framework for Contrastive Learning of Visual Representations** (2020) \cite{chen2020simple}

What it did: Projection head and normalization standardize contrastive embeddings  *(the tool's reading)*

SimCLR introduces learnable nonlinear projection heads mapping representations into normalized embedding spaces, where L2 normalization and temperature tuning substantially improve quality. Projected embeddings lose task-specific information while the pre-projection representation excels downstream.  *(the tool's reading)*

> “Representation learning with contrastive cross entropy loss benefits from normalized embeddings and an appropriately adjusted temperature parameter.”
>
> ✓ verified: found word for word in the paper's own text


**Improved Baselines with Momentum Contrastive Learning** (2020) \cite{chen2020improved}

What it did: MLP projection and augmentation orthogonally improve momentum contrastive embeddings  *(the tool's reading)*

MoCo v2 adopts SimCLR's projection head and data augmentation, showing these improvements are orthogonal to MoCo's momentum mechanism. Strong embeddings emerge without requiring large batches, unlike SimCLR.  *(the tool's reading)*

> “Images are encoded into a representation space, in which pairwise affinities are computed.”
>
> ✓ verified: found word for word in the paper's own text


**Prototypical Contrastive Learning of Unsupervised Representations** (2020) \cite{li2020prototypical}

What it did: Prototype-based contrastive loss encodes semantic structure into embeddings  *(the tool's reading)*

PCL extends instance-wise contrastive learning by assigning multiple prototypes per image and using prototype-specific concentration estimation. Embeddings now implicitly encode hierarchical semantic structure beyond local smoothness.  *(the tool's reading)*

> “unsupervised visual representation learning aims to learn an embedding function f_θ (realized via a DNN) that maps X to V={v_1, v_2,...,v_n} with v_i=f_θ(x_i), such that v_i best describes x_i.”
>
> ✓ verified: found word for word in the paper's own text


**What Makes for Good Views for Contrastive Learning?** (2020) \cite{tian2020what}

> “(a) Schematic of multiview contrastive representation learning, where an image is split into two views, and passed through two encoders to learn an embedding where the views are close relative to views from other images.”
>
> ✓ verified: found word for word in the paper's own text


**Barlow Twins: Self-Supervised Learning via Redundancy Reduction** (2021) \cite{zbontar2021barlow}

What it did: Redundancy reduction enables high-dimensional embeddings without collapse  *(the tool's reading)*

Barlow Twins benefits from very high-dimensional embeddings via cross-correlation-based redundancy reduction, normalizing along the batch rather than feature dimension. This avoids collapsed embeddings without momentum encoders or large batches required by other methods.  *(the tool's reading)*

> “We call the output of the encoder the 'representations' and the output of the projector the 'embeddings'. The representations are used for downstream tasks and the embeddings are fed to the loss function of Barlow Twins.”
>
> ✓ verified: found word for word in the paper's own text


**DINOv2: Learning Robust Visual Features without Supervision** (2023) \cite{oquab2023dinov}

What it did: Self-supervised embeddings enable automatic metadata-free data curation  *(the tool's reading)*

DINOv2 applies self-supervised embeddings to large-scale data curation, using embedding similarity for deduplication and retrieval without text metadata. Hardware-optimized embedding dimensions accelerate practical deployment of vision embeddings.  *(the tool's reading)*

> “In order to do this, we first compute an image embedding using a self-supervised ViT-H/16 network pretrained on ImageNet-22k, and use cosine-similarity as a distance measure between images.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Visual & Image Embeddings** → **Multimodal & Cross-Modal Embeddings** (2014): Joint vision-language spaces
- **Visual & Image Embeddings** → **Self-Supervised & Contrastive Embeddings** (2019): Unsupervised vision features