# Multimodal & Cross-Modal Embeddings

> One thread of 9 from the "embedding" map, covering the 11 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Word & Text Embeddings, Visual & Image Embeddings.

Emerged from zero-shot learning using word embeddings as semantic bridges (2013), developed into explicit visual-semantic joint embedding spaces for image-caption alignment (2014-2016), expanded to encompass text-video, speech-visual, and heterogeneous cross-modal fusion (2017-2019).


**Zero-Shot Learning Through Cross-Modal Transfer** (2013) \cite{socher2013zero}

What it did: Establish cross-modal semantic space via word embeddings  *(the tool's reading)*

This paper pioneered mapping images into a semantic space defined by word embeddings, grounding word vectors through visual modality. It demonstrated that 'Images are mapped into a semantic space of words that is learned by a neural network model, so word vectors get implicitly grounded by the visual modality, allowing prototypical instances for various words.'  *(the tool's reading)*

> “Multimodal embeddings relate information from multiple sources such as sound and video […] or images and text.”
>
> ✓ verified: found word for word in the paper's own text


**Deep Fragment Embeddings for Bidirectional Image Sentence Mapping** (2014) \cite{karpathy2014deep}

What it did: Decompose embeddings into fine-grained fragment-level alignments  *(the tool's reading)*

Building on shared embedding spaces, this work introduced fragment-level embedding by moving beyond whole-unit projections. It showed that 'Unlike previous work that embeds images and sentences as whole units, this model embeds fragments of images (objects) and fragments of sentences (dependency tree relations) into a common embedding space.'  *(the tool's reading)*

> “We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space.”
>
> ✓ verified: found word for word in the paper's own text


**Conditional Generative Adversarial Nets** (2014) \cite{mirza2014conditional}

> “Conceptual word embeddings […] can be very useful here since related concepts end up being represented by similar vectors.”
>
> ✓ verified: found word for word in the paper's own text


**Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models** (2014) \cite{kiros2014unifying}

What it did: Unify joint embeddings with language generation via encoders-decoders  *(the tool's reading)*

This paper extended fragment and image-text embeddings by integrating them with neural language models in a unified pipeline. It demonstrated that embeddings could capture meaningful structure: 'Shows that with linear encoders, the learned multimodal embedding space captures vector-space arithmetic regularities analogous to word embedding analogies (e.g., image of a blue car - 'blue' + 'red' is near images of red cars).'  *(the tool's reading)*

> “we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space.”
>
> ✓ verified: found word for word in the paper's own text


**Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)** (2014) \cite{mao2014deep}

> “The two word embedding layers embed the one-hot input into a dense word representation. It encodes both the syntactic and semantic meaning of the words. The semantically relevant words can be found by calculating the Euclidean distance between two dense word vectors in embedding layers.”
>
> ✓ verified: found word for word in the paper's own text


**Learning Transferable Features with Deep Adaptation Networks** (2015) \cite{long2015learning}

> “The mean embedding of distribution p in ℋ_k is a unique element μ_k(p) such that 𝐄_𝐱∼ pf( 𝐱) = ⟨f( 𝐱),μ _k( p )⟩ _ℋ_k for all f ∈ℋ_k.”
>
> ✓ verified: found word for word in the paper's own text


**Generative Adversarial Text to Image Synthesis** (2016) \cite{reed2016generative}

> “To obtain a visually-discriminative vector representation of text descriptions, we follow the approach of […] by using deep convolutional and recurrent text encoders that learn a correspondence function with images.”
>
> ✓ verified: found word for word in the paper's own text


**VSE++: Improving Visual-Semantic Embeddings with Hard Negatives** (2017) \cite{faghri2017vse}

What it did: Refine embeddings with hard negative mining strategies  *(the tool's reading)*

This work improved embedding training by introducing hard negative sampling to joint visual-semantic spaces. It showed that 'Introduces a novel loss function (Max of Hinges) that emphasizes hard negatives within mini-batches for training visual-semantic embeddings, without additional mining cost.'  *(the tool's reading)*

> “Such embeddings entail mappings from two (or more) domains into a common vector space in which semantically associated inputs (e.g., text and images) are mapped to similar locations. The embedding space thus represents the underlying domain structure, where location and often direction are semantically meaningful.”
>
> ✓ verified: found word for word in the paper's own text


**HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million\n Narrated Video Clips** (2019) \cite{miech2019howto}

What it did: Scale embeddings to large-scale video-text data  *(the tool's reading)*

This paper extended multimodal embeddings to video, demonstrating scalability without manual supervision. It revealed that 'Learning a text-video embedding from a large-scale dataset of narrated instructional videos without any manual caption annotation, using automatically transcribed narrations as weak supervision.'  *(the tool's reading)*

> “Given this, our goal is to learn two mapping functions: f:ℝ^d_v→ℝ^d and g:ℝ^d_c→ℝ^d that respectively embed video and caption features into a common d-dimensional space, such that the cosine similarity”
>
> ✓ verified: found word for word in the paper's own text


**VL-BERT: Pre-training of Generic Visual-Linguistic Representations** (2019) \cite{su2019bert}

What it did: Unify visual-linguistic embeddings in Transformer architecture  *(the tool's reading)*

This work extended embeddings to unified visual-linguistic representations within Transformers by combining appearance and geometry. It showed that 'Extends BERT's embedding scheme (token, segment, position) by adding a visual feature embedding derived from Fast R-CNN appearance features and geometric position encoding, enabling a single unified Transformer to jointly embed visual and linguistic elements.'  *(the tool's reading)*

> “For each input element, its embedding feature is the summation of four types of embedding, namely, token embedding, visual feature embedding, segment embedding, and sequence position embedding.”
>
> ✓ verified: found word for word in the paper's own text


**Hierarchical Text-Conditional Image Generation with CLIP Latents** (2022) \cite{ramesh2022hierarchical}

What it did: Leverage pre-trained embeddings for generative conditioning  *(the tool's reading)*

This paper used established multimodal embeddings (CLIP) as conditioning vectors for generative models, advancing from retrieval to synthesis. It demonstrated that 'Proposes explicitly generating image embeddings (via a prior) from text captions rather than directly conditioning a decoder on text embeddings, improving diversity with minimal loss in photorealism/caption similarity.'  *(the tool's reading)*

> “Given an image x, let z_i and z_t be its CLIP image and text embeddings, respectively.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Word & Text Embeddings** → **Multimodal & Cross-Modal Embeddings** (2013): Semantic bridges
- **Visual & Image Embeddings** → **Multimodal & Cross-Modal Embeddings** (2014): Joint vision-language spaces