Word & Text Embeddings

The paper introduced learned word embeddings as a core component of the neural network architecture for NLP tasks.

The paper demonstrated that embedding representations could be learned jointly with task objectives through backpropagation rather than using hand-crafted features.

“The features computed by the deep layers of the network are automatically trained by backpropagation to be relevant to the task.”◌ not checked against the paper’s text as it now stands

The paper showed that learned word embeddings could transfer knowledge across multiple NLP tasks when trained on large unlabeled datasets.

“Instead we reach good performance levels in most of the tasks by transferring intermediate representations discovered on large unlabeled datasets.”◌ not checked against the paper’s text as it now stands

The paper extended the embedding approach to handle multiple discrete features by concatenating separate lookup table outputs for each feature type.

“We associate to each feature a lookup table LT_W^k(·), with parameters W^k∈^d^k_wrd×| D^k| where d^k_wrd∈ is a user-specified vector size. Given a word w, a feature vector of dimension d_wrd=∑_k d^k_wrd is then obtained by concatenating all lookup table outputs”◌ not checked against the paper’s text as it now stands

This paper introduced two novel model architectures, CBOW and Skip-gram, for computing continuous vector representations of words called embeddings.

“We propose two novel model architectures for computing continuous vector representations of words from very large data sets.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that word embeddings can capture linear regularities and semantic relationships between words through algebraic operations on vectors.

“Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities. Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector("King") - vector("Man") + vector("Woman") results in a vector that is closest to the vector representation of the word Queen”◌ not checked against the paper’s text as it now stands

The paper created a comprehensive test set to measure the quality of word embeddings on both syntactic and semantic word relationships.

“We design a new comprehensive test set for measuring both syntactic and semantic regularities[The test set is available at <www.fit.vutbr.cz/ imikolov/rnnlm/word-test.v1.txt>], and show that many such regularities can be learned with high accuracy.”

This paper presented the Skip-gram model as an efficient method for learning high-quality distributed vector representations that capture syntactic and semantic word relationships.

“The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships.”◌ not checked against the paper’s text as it now stands

This paper extended word embeddings to phrase embeddings by treating phrases as individual tokens during training, enabling the model to represent idiomatic phrases that cannot be easily composed from individual words.

“Therefore, using vectors to represent the whole phrases makes the Skip-gram model considerably more expressive. Other techniques that aim to represent meaning of sentences by composing the word vectors, such as the recursive autoencoders […], would also benefit from using phrase vectors instead of the word vectors.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that the learned embeddings exhibit an additive compositionality property where vector addition of word embeddings produces meaningful results.

This paper explained the derivation of Mikolov et al.'s negative-sampling word-embedding method, which was previously described in a cryptic way in the original papers.

“We found the description of the models in these papers to be somewhat cryptic and hard to follow. While the motivations and presentation may be obvious to the neural-networks language-modeling crowd, we had to struggle quite a bit to figure out the rationale behind the equations. This note is an attempt to explain equation (4) (negative sampling)”◌ not checked against the paper’s text as it now stands

This paper derived and presented the complete mathematical objective function for negative-sampling word embeddings in a clear step-by-step manner.

“max_θ∑_(w,c)∈ Dlogσ(v_c · v_w) + ∑_(w,c) ∈ D'logσ(-v_c · v_w) which is almost equation (4) in Mikolov et al”◌ not checked against the paper’s text as it now stands

This paper showed that the negative-sampling approach constructs word embeddings by distinguishing true word-context pairs from randomly sampled negative pairs through a binary classification objective.

“One way to do so, is to present the model with some (w,c) pairs for which p(D=1|w,c;θ) must be low, i.e. pairs which are not in the data. This is achieved by generating the set D' of random (w,c) pairs, assuming they are all incorrect”

This paper introduced Paragraph Vector, an unsupervised algorithm that learns fixed-length vector embeddings for variable-length text sequences like sentences, paragraphs, and documents.

“we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that paragraph vector embeddings capture semantic relationships between words, addressing a key weakness of bag-of-words representations.

The paper created two variants of the paragraph vector embedding method: a distributed memory model (PV-DM) and a distributed bag of words model (PV-DBOW).

“In our experiments, each paragraph vector is a combination of two vectors: one learned by the standard paragraph vector with distributed memory (PV-DM) and one learned by the paragraph vector with distributed bag of words (PV-DBOW).”◌ not checked against the paper’s text as it now stands

This paper proposed learning word embeddings of dimension 100 for each word by approximating input and output matrices with lower-rank matrices in the RNN Encoder-Decoder model.

“The input matrix between each input symbol x_t and the hidden unit is approximated with two lower-rank matrices, and the output matrix is approximated similarly. We used rank-100 matrices, equivalent to learning an embedding of dimension 100 for each word.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that the RNN Encoder-Decoder learns continuous space representations of phrases that preserve semantic and syntactic structure.

“The further analysis of the model reveals that the RNN Encoder–Decoder learns a continuous space representation of a phrase that preserves both the semantic and syntactic structure of the phrase.”◌ not checked against the paper’s text as it now stands

This paper showed that neural language models project input words into embedding space, specifically using a 512-dimensional embedding space for the CSLM model.

“Each input word was projected into the embedding space ^512, and they were concatenated to form a 3072-dimensional vector.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that LSTMs learn to map variable-length input sequences into fixed-dimensional vector representations that capture sentence meaning.

“A useful property of the LSTM is that it learns to map an input sentence of variable length into a fixed-dimensional vector representation. Given that translations tend to be paraphrases of the source sentences, the translation objective encourages the LSTM to find sentence representations that capture their meaning, as sentences with similar meanings are close to each other while different sentences meanings will be far.”◌ not checked against the paper’s text as it now stands

The paper showed that the fixed-dimensional embeddings learned by the LSTM are sensitive to word order and invariant to voice.

“The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice.”◌ not checked against the paper’s text as it now stands

The paper visualized that LSTM sentence embeddings cluster phrases by meaning through 2-dimensional PCA projections.

“The figure shows a 2-dimensional PCA projection of the LSTM hidden states that are obtained after processing the phrases in the figures. The phrases are clustered by meaning, which in these examples is primarily a function of word order, which would be difficult to capture with a bag-of-words model.”

The paper uses embedding matrices to represent words in the LSTM decoder for image caption generation.

The paper initializes LSTM hidden and memory states by computing averages of annotation vectors fed through MLPs, using learned embeddings to transform image features.

The paper creates annotation vectors as D-dimensional embeddings of different spatial regions extracted from a convolutional neural network's lower layers.

“We use a convolutional neural network in order to extract a set of feature vectors which we refer to as annotation vectors. The extractor produces L vectors, each of which is a D-dimensional representation corresponding to a part of the image.”◌ not checked against the paper’s text as it now stands

This paper developed an LSTM-RNN model that maps sentences to semantic embedding vectors by sequentially processing words and using the final hidden activation as the sentence representation.

“The proposed LSTM-RNN model sequentially takes each word in a sentence, extracts its information, and embeds it into a semantic vector. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence.”◌ not checked against the paper’s text as it now stands

This paper showed that LSTM-RNN embedding automatically detects keywords and allocates them to different cells based on topic similarity.

“The model is found to automatically attenuate the unimportant words and detects the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell.”◌ not checked against the paper’s text as it now stands

This paper trained the embedding model on weakly supervised click-through data from web search engines to learn embeddings useful for document retrieval.

“In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine.”

This paper introduced skip-thought vectors, an unsupervised method for learning sentence embeddings by training an encoder-decoder model to reconstruct surrounding sentences.

“We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage.”◌ not checked against the paper’s text as it now stands

The paper developed a vocabulary expansion technique that maps word embeddings from word2vec space to the encoder's embedding space, enabling the model to encode words not seen during training.

“We solve this problem by learning a mapping that transfers word representations from one model to another. Using pre-trained word2vec representations learned with a continuous bag-of-words model, we learn a linear mapping from a word in word2vec space to a word in the encoder's vocabulary space.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that skip-thought vectors produce generic sentence embeddings that perform robustly across multiple diverse downstream tasks without fine-tuning.

“After learning skip-thoughts, freeze the model and use the encoder as a generic feature extractor for arbitrary tasks. In our experiments we consider 8 tasks: semantic-relatedness, paraphrase detection, image-sentence ranking and 5 standard classification benchmarks.”

The paper uses word embedding as an input representation to the LSTM networks for sequence tagging.

“To use this embedding, we simply replace the one hot encoding word representation with its corresponding 50-dimensional vector.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that the BI-LSTM-CRF model has reduced dependence on word embedding compared to the Conv-CRF baseline model.

“It is interesting that our best model BI-LSTM-CRF has less dependence on Senna word embedding compared to Conv-CRF model. For example, the tagging difference between BI-LSTM-CRF model for random and Senna categories are 0.12%, 0.33%, and 4.57% for POS, chunking and NER data sets respectively.”◌ not checked against the paper’s text as it now stands

The paper shows that LSTM-based models can achieve good performance even without word embeddings by learning from the data more effectively.

“In addition, our BI-LSTM-CRF model already reaches a good accuracy without the use of the Senna embedding.”◌ not checked against the paper’s text as it now stands

This paper showed that neural network models can be trained effectively on the large SNLI corpus to perform competitively on natural language inference tasks.

“it allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.”◌ not checked against the paper’s text as it now stands

This paper evaluated distributed representation models and neural network sequence models as approaches for learning sentence embeddings on the NLI task.

“distributed representation models, including a baseline model and neural network sequence models.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that an LSTM-based neural network model can learn effective sentence embeddings by achieving comparable performance to feature-rich classifiers on natural language inference.

“We find that two models achieve comparable performance: a feature-rich classifier model and a neural network model centered around a Long Short-Term Memory network (LSTM”◌ not checked against the paper’s text as it now stands

The paper applied character-level Convolutional Neural Networks to create more efficient word embeddings for language modeling.

“we propose an extension to this idea to also reduce the number of parameters of the Softmax layer. Recall from Section […] that the Softmax computes a logit as z_w=h^Te_w where h is a context vector and e_w the word embedding. Instead of building a matrix of |V|× |h| (whose rows correspond to e_w), we produce e_w with a CNN over the characters of w as e_w = CNN(chars_w) – we call this a CNN Softmax.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that CNN-based embeddings could achieve strong language modeling performance while using substantially fewer parameters than standard word embedding matrices.

“The character-level features allow for a smoother and compact parametrization of the word embeddings. Recent efforts on small scale language modeling have used CNN character embeddings for the input embeddings […].”◌ not checked against the paper’s text as it now stands

The paper showed that character-level CNN embeddings enable the model to handle out-of-vocabulary words and support parallel training across different datasets and languages.

“Aside from a big reduction in the number of parameters and incorporating morphological knowledge from words, the other benefit of this approach is that out-of-vocabulary (OOV) words can easily be scored. This may be useful for other problems such as Machine Translation where handling out-of-vocabulary words is very important […]. This approach also allows parallel training over various data sets since the model is no longer explicitly parametrized by the vocabulary size – or the language.”

This paper initializes word embeddings with pre-trained Glove vectors for the embedding layer.

“We use pre-trained 300-D Glove 840B vectors […] to initialize our word embeddings.”◌ not checked against the paper’s text as it now stands

This paper uses embedding vectors as input to bidirectional LSTM encoders for representing words and their context.

“The a_i or b_j ∈^l is an embedding of l-dimensional vector, which can be initialized with some pre-trained word embeddings and organized with parse trees.”◌ not checked against the paper’s text as it now stands

This paper updates all word embeddings during training rather than keeping them fixed.

“All vectors including word embedding are updated during training.”◌ not checked against the paper’s text as it now stands

The paper uses embeddings by transforming source sentences into lists of vectors through an encoder network, with one vector per input symbol.

“The encoder transforms a source sentence into a list of vectors, one vector per input symbol.”◌ not checked against the paper’s text as it now stands

The paper employs embeddings in the attention mechanism by computing attention context as a weighted sum of encoder vectors.

“𝐚_i = ∑_t=1^M p_t. 𝐱_t”◌ not checked against the paper’s text as it now stands

The paper utilizes embeddings in the decoder by producing hidden state vectors that are processed through a softmax layer to generate probability distributions over output symbols.

“The decoder RNN network produces a hidden state 𝐲_𝐢 for the next symbol to be predicted, which then goes through the softmax layer to generate a probability distribution over candidate output symbols.”◌ not checked against the paper’s text as it now stands

The paper created ConceptNet Numberbatch, a hybrid embedding system that combines ConceptNet with distributional word embeddings using retrofitting.

The paper demonstrated that combining ConceptNet with word embeddings produces better results than using distributional semantics alone.

“When ConceptNet is combined with word embeddings acquired from distributional semantics (such as word2vec), it provides applications with understanding that they would not acquire from distributional semantics alone, nor from narrower resources such as WordNet or DBPedia.”◌ not checked against the paper’s text as it now stands

The paper computed embeddings directly from ConceptNet using PPMI and truncated SVD to reduce dimensionality to 300 dimensions.

“As in Levy et al., we determine the pointwise mutual information of the matrix entries with context distributional smoothing, clip the negative values to yield positive pointwise mutual information (PPMI), reduce the dimensionality of the result to 300 dimensions with truncated SVD, and combine the terms and contexts symmetrically into a single matrix of word embeddings.”◌ not checked against the paper’s text as it now stands

The paper proposes using a 2-D matrix instead of a vector to represent sentence embeddings, where each row attends to different parts of the sentence.

“Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.”◌ not checked against the paper’s text as it now stands

The paper introduces a self-attention mechanism that allows extracting multiple different aspects of a sentence into separate vector representations rather than collapsing to a single representation.

“the proposed self-attention mechanism allows extracting different aspects of the sentence into multiple vector representations.”◌ not checked against the paper’s text as it now stands

The paper proposes a novel penalization term based on Frobenius norm to encourage diversity among the attention weight vectors across different hops.

“We use the dot product of A and its transpose, subtracted by an identity matrix, as a measure of redundancy. P = ( AA^T-I )_ F ^2”◌ not checked against the paper’s text as it now stands

The paper trained sentence embeddings on the Natural Language Inference task to learn universal sentence representations.

“sentence embeddings trained on various supervised tasks, and show that sentence embeddings generated from models trained on a natural language inference (NLI) task reach the best results in terms of transfer accuracy.”◌ not checked against the paper’s text as it now stands

The paper investigated multiple neural network architectures for encoding sentences into fixed-size embeddings.

“We compare 7 different architectures: standard recurrent encoders with either Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU), concatenation of last hidden states of forward and backward GRU, Bi-directional LSTMs (BiLSTM) with either mean or max pooling, self-attentive network and hierarchical convolutional networks.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that BiLSTM-Max embeddings trained on SNLI outperform unsupervised embedding methods on transfer tasks.

“Our experiments show that an encoder based on a bi-directional LSTM architecture with max pooling, trained on the Stanford Natural Language Inference (SNLI) dataset […], yields state-of-the-art sentence embeddings compared to all existing alternative unsupervised approaches like SkipThought or FastSent, while being much faster to train.”

The paper introduced learned embeddings to convert input and output tokens to vectors of fixed dimension in the Transformer model.

“Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension.”◌ not checked against the paper’s text as it now stands

The paper multiplied embedding weights by a scaling factor to balance the embeddings with other components.

“In the embedding layers, we multiply those weights by √.”◌ not checked against the paper’s text as it now stands

The paper added positional encodings with the same dimension as embeddings so they could be summed together to inject sequence order information.

“The positional encodings have the same dimension as the embeddings, so that the two can be summed.”◌ not checked against the paper’s text as it now stands

The paper used sinusoidal positional encodings as an alternative to learned positional embeddings for representing token positions.

“In this work, we use sine and cosine functions of different frequencies”

The paper introduced a new type of embedding that models both complex characteristics of word use and how these uses vary across linguistic contexts.

“We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).”◌ not checked against the paper’s text as it now stands

The paper created embeddings derived from internal states of a deep bidirectional language model rather than context-independent word type embeddings.

“Our representations differ from traditional word type embeddings in that each token is assigned a representation that is a function of the entire input sentence. We use vectors derived from a bidirectional LSTM that is trained with a coupled language model (LM) objective on a large text corpus.”◌ not checked against the paper’s text as it now stands

The paper showed that using all layers of the biLM for embeddings improves performance over using only the top layer.

“Including representations from all layers improves overall performance over just using the last layer, and including contextual representations from the last layer improves performance over the baseline.”

This paper introduced two models for encoding sentences into fixed-dimensional embedding vectors that enable transfer learning to other NLP tasks.

“We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that sentence-level embeddings substantially outperform word-level transfer learning approaches on diverse NLP tasks.

“Models that make use of sentence level transfer learning tend to perform better than models that only use word level transfer.”◌ not checked against the paper’s text as it now stands

This paper showed that sentence embeddings can achieve strong task performance with remarkably little task-specific training data through transfer learning.

“We find that our sentence embeddings can be used to obtain surprisingly good task performance with remarkably little task specific training data.”◌ not checked against the paper’s text as it now stands

The paper made the embedding models publicly available as a practical resource for the research and industrial NLP communities.

The paper demonstrates that cross-lingual language models can learn aligned embedding spaces across languages by using a shared subword vocabulary created through Byte Pair Encoding.

“In all our experiments we process all languages with the same shared vocabulary created through Byte Pair Encoding (BPE) […]. As shown in […], this greatly improves the alignment of embedding spaces across languages that share either the same alphabet or anchor tokens such as digits […] or proper nouns.”◌ not checked against the paper’s text as it now stands

The paper shows that cross-lingual word embeddings can be obtained from the lookup table of their pretrained cross-lingual language model.

“In this work, we also use a shared vocabulary but our word embeddings are obtained via the lookup table of our cross-lingual language model (XLM).”◌ not checked against the paper’s text as it now stands

The paper proposes using position embeddings in a novel way for the TLM objective by resetting the positions of target sentences to facilitate alignment of representations across languages.

“To facilitate the alignment, we also reset the positions of target sentences.”◌ not checked against the paper’s text as it now stands

The paper uses contextual embeddings from BERT to represent tokens instead of relying on exact string matching or heuristic matching for similarity computation.

“In contrast to prior word embeddings~, contextual embeddings, such as ~ and ~, can generate different vector representations for the same word in different sentences depending on the surrounding words, which form the context of the target word.”◌ not checked against the paper’s text as it now stands

The paper computes token similarity using cosine similarity between contextual embeddings rather than n-gram overlap.

“computes the similarity of two sentences as a sum of cosine similarities between their tokens' embeddings.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that contextual embeddings are effective for capturing paraphrases, addressing a limitation of n-gram-based metrics.

“In contrast to string matching (e.g., in ) or matching heuristics (e.g., in ), we compute similarity using contextualized token embeddings, which have been shown to be effective for paraphrase detection~.”◌ not checked against the paper’s text as it now stands

XLNet proposes using word embeddings as the initialization for the content stream in two-stream self-attention.

“the content stream is set to the corresponding word embedding, i.e. h_i^(0) = e(x_i).”◌ not checked against the paper’s text as it now stands

XLNet initializes query stream representations with a trainable vector rather than word embeddings.

“the first layer query stream is initialized with a trainable vector, i.e. g_i^(0) = w”◌ not checked against the paper’s text as it now stands

XLNet uses relative positional encodings from Transformer-XL to encode positions in the sequence.