Multimodal & Cross-Modal Embeddings

The paper mapped images into a semantic space using word embeddings learned from unsupervised text corpora to enable zero-shot learning.

“First, images are mapped into a semantic space of words that is learned by a neural network model […]. Word vectors capture distributional similarities from a large, unsupervised text corpus. By learning an image mapping into this space, the word vectors get implicitly grounded by the visual modality, allowing us to give prototypical instances for various words.”◌ not checked against the paper’s text as it now stands

The paper projected image feature vectors into 50-dimensional word embedding space to learn semantic relationships and class membership of images.

“In order to learn semantic relationships and class membership of images we project the image feature vectors into the 50-dimensional word space.”◌ not checked against the paper’s text as it now stands

The paper initialized word embeddings with pre-trained 50-dimensional vectors that capture distributional syntactic and semantic information from text.

“We initialize all word vectors with pre-trained 50-dimensional word vectors from the unsupervised model of Huang et al. […]. Using free Wikipedia text, their model learns word vectors by predicting how likely it is for each word to occur in its context.”◌ not checked against the paper’s text as it now stands

The paper proposes embedding both visual and language fragments into a common space rather than embedding entire images and sentences.

“Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space.”◌ not checked against the paper’s text as it now stands

The paper introduces a fragment alignment objective that learns to associate image and sentence fragments across modalities through latent alignment.

“In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that jointly reasoning at both global and fragment levels improves embedding-based image-sentence retrieval performance.

“reasoning on both the global level of images and sentences and the finer level of their respective fragments significantly improves performance on image-sentence retrieval tasks”◌ not checked against the paper’s text as it now stands

The paper uses word embeddings learned from a skip-gram model to represent tags as vectors in a multi-modal learning task.

“we trained a skip-gram model […] with word vector size of 200.”◌ not checked against the paper’s text as it now stands

The paper leverages conceptual word embeddings to normalize user-generated tags by representing related concepts with similar vectors.

“Conceptual word embeddings […] can be very useful here since related concepts end up being represented by similar vectors.”◌ not checked against the paper’s text as it now stands

The paper introduces a method to learn a multimodal joint embedding space that combines images and text using an encoder-decoder pipeline.

“we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space”◌ not checked against the paper’s text as it now stands

The paper demonstrates that the learned embedding space captures multimodal regularities through vector space arithmetic operations.

“we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars”◌ not checked against the paper’s text as it now stands

The paper uses LSTM to encode sentences into an embedding space where images and their descriptions can be ranked against each other.

“For the encoder, we learn a joint image-sentence embedding where sentences are encoded using long short-term memory (LSTM) recurrent neural networks […]. Image features from a deep convolutional network are projected into the embedding space of the LSTM hidden states. A pairwise ranking loss is minimized in order to learn to rank images and their descriptions.”

The paper introduced word embedding layers that learn dense word representations from training data rather than using pre-computed vectors.

“The two word embedding layers embed the one-hot input into a dense word representation. It encodes both the syntactic and semantic meaning of the words.”◌ not checked against the paper’s text as it now stands

The paper incorporated a two-layer word embedding system in the m-RNN network structure which learns the word representation more efficiently than single-layer word embedding.

“Firstly, we incorporate a two-layer word embedding system in the m-RNN network structure which learns the word representation more efficiently than the single-layer word embedding.”◌ not checked against the paper’s text as it now stands

The paper used a multimodal layer that combines word embedding, recurrent layer, and image representation into a shared semantic space through learned projection matrices.

“We map the activation of the three layers to the same multimodal feature space and add them together to obtain the activation of the multimodal layer: 𝐦(t)=g_2(𝐕_w ·𝐰(t) + 𝐕_r ·𝐫(t) + 𝐕_I ·𝐈)”◌ not checked against the paper’s text as it now stands

This paper embeds hidden representations of task-specific layers in a reproducing kernel Hilbert space to enable mean embedding matching for domain adaptation.

This paper uses an unbiased linear-time estimate of kernel embedding to enable scalable training of deep neural networks.

“we implement a linear-time unbiased estimate of the kernel mean embedding to enable scalable training, which is very desirable for deep learning”◌ not checked against the paper’s text as it now stands

This paper embeds probability distributions in reproducing kernel Hilbert spaces to bound target domain risk based on kernel distance between domains.

“ϵ _t(θ ) ⩽ϵ _s(θ ) + 2 d_k(p,q) + C, where C is a constant for the complexity of hypothesis space and the risk of an ideal hypothesis for both domains”◌ not checked against the paper’s text as it now stands

The paper learns text embeddings through a deep convolutional-recurrent text encoder trained on structured joint embedding with images.

“To obtain a visually-discriminative vector representation of text descriptions, we follow the approach of […] by using deep convolutional and recurrent text encoders that learn a correspondence function with images.”◌ not checked against the paper’s text as it now stands

The paper projects text embeddings to lower dimensions and concatenates them with image feature maps in both the generator and discriminator networks.

“The description embedding φ(t) is first compressed using a fully-connected layer to a small dimension (in practice we used 128) followed by leaky-ReLU and then concatenated to the noise vector z.”◌ not checked against the paper’s text as it now stands

The paper generates additional synthetic text embeddings by interpolating between embeddings of training set captions to regularize the generator.

“Motivated by this property, we can generate a large amount of additional text embeddings by simply interpolating between embeddings of training set captions.”◌ not checked against the paper’s text as it now stands

The paper uses a pre-trained character-level convolutional-recurrent text encoder that produces 1024-dimensional embeddings for representing text descriptions.

The paper introduced a hard negative loss function that focuses on the hardest negative samples rather than summing over all negatives.

“To emphasize hard negatives we define our loss as ℓ_MH(i, c) = max_c^'[α + s(i,c^') - s(i,c)]_+ + max_i^'[α + s(i^',c) -s(i,c)]_+.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the hard negative loss can be efficiently applied within mini-batches without additional mining cost.

“For computational efficiency, instead of finding the hardest negatives in the entire training set, we find them within each mini-batch. This has the same quadratic complexity as the complexity of the loss.”◌ not checked against the paper’s text as it now stands

The paper showed that their embedding approach achieves significant improvements in cross-modal retrieval performance, outperforming prior methods by substantial margins.

The paper mapped images and captions into a joint embedding space using linear projections followed by normalization to lie on the unit hypersphere.

“let the mappings into the joint embedding space be defined by linear projections: f(i; W_f, θ_ϕ) = W_f^T ϕ(i; θ_ϕ) g(c; W_g, θ_ψ) = W_g^T ψ(c; θ_ψ) where W_f∈^D_ϕ× D and W_g∈^D_ψ× D. We further normalize f(i; W_f, θ_ϕ), and g(c; W_g, θ_ψ), to lie on the unit hypersphere.”

The paper introduces HowTo100M, a large-scale dataset of 136 million video clips with automatically transcribed narrations for learning text-video embeddings without manual annotation.

“First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that text-video embeddings trained on HowTo100M achieve state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets.

“Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask.”◌ not checked against the paper’s text as it now stands

The paper shows that the learned embedding transfers well to other domains through fine-tuning on generic YouTube videos and movies.

“Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone.”

The paper introduces embedding features for visual-linguistic tasks by extending BERT to take both visual and linguistic embedded features as input.

“VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input.”◌ not checked against the paper’s text as it now stands

The paper designs visual feature embeddings that combine visual appearance features from Fast R-CNN with visual geometry embeddings encoding the spatial location of regions-of-interest.

“The visual feature embedding is attached to each of the input elements, which is the output of a fully connected layer taking the concatenation of visual appearance feature and visual geometry embedding as input.”◌ not checked against the paper’s text as it now stands

The paper incorporates token embeddings, segment embeddings, and sequence position embeddings alongside visual feature embeddings to create comprehensive input representations.

“For each input element, its embedding feature is the summation of four types of embedding, namely, token embedding, visual feature embedding, segment embedding, and sequence position embedding.”◌ not checked against the paper’s text as it now stands

The paper used CLIP embeddings as a continuous vector representation to condition image generation, enabling the decoder to invert these embeddings into images.

“a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding”◌ not checked against the paper’s text as it now stands

The paper demonstrated that CLIP's joint embedding space of text and images enables language-guided image manipulations by interpolating between image embeddings and text embedding differences.

“the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion”◌ not checked against the paper’s text as it now stands

The paper applied PCA to reduce the dimensionality of CLIP image embeddings from 1,024 to 319 dimensions for the autoregressive prior while preserving nearly all information.

The paper used the CLIP image embedding as a bipartite latent representation that captures semantic information while allowing the decoder to reconstruct images with high fidelity.

“Our approach allows us to encode any given image x into a bipartite latent representation (z_i, x_T) that is sufficient for the decoder to produce an accurate reconstruction. The latent z_i describes the aspects of the image that are recognized by CLIP”