# Word & Text Embeddings

> One thread of 9 from the "embedding" map, covering the 24 papers in it. The other threads are not represented here.
>
> Every quotation was copied word for word from the paper's own text, and
> checked against that text. Quotes marked *inferred* failed that check and
> must be re-checked before use. Quotes marked *not re-checked* have not been
> matched against the paper's text as it now stands, so they carry no current
> verification either. Lines labelled *the tool's reading* are
> model judgment, not quotation, and carry no verification.
>
> **This is a scaffold, not prose.** The citations, quotes and structure are
> real; the argument is yours to write.

> Connects to: Multimodal & Cross-Modal Embeddings, Self-Supervised & Contrastive Embeddings.

Began with word2vec's trainable word vectors as core NLP primitives (2011-2013), expanded to phrase and sentence embeddings capturing compositional semantics (2014-2015), then evolved toward context-aware embeddings that vary per-token based on surrounding language (2017-2019).


**Natural Language Processing (almost) from Scratch** (2011) \cite{collobert2011natural}

What it did: Introduce trainable word embeddings via lookup tables  *(the tool's reading)*

This paper establishes word embeddings as learned feature vectors obtained through lookup table layers trained by backpropagation. It demonstrates that embeddings can be learned from unlabeled data and transferred across NLP tasks, avoiding hand-designed features.  *(the tool's reading)*

> “More formally, for each word w∈ D, an internal d_wrd-dimensional feature vector representation is given by the lookup table layer LT_W(·):”
>
> ✓ verified: found word for word in the paper's own text


**Efficient Estimation of Word Representations in Vector Space** (2013) \cite{mikolov2013efficient}

What it did: Demonstrate efficient neural architectures and algebraic structure  *(the tool's reading)*

Building on lookup table embeddings, this paper introduces computationally efficient models (CBOW and Skip-gram) that learn word vectors at unprecedented scale while revealing that embeddings preserve linear regularities. The paper shows that geometric relationships enable algebraic operations like vector('King') - vector('Man') + vector('Woman') ≈ vector('Queen').  *(the tool's reading)*

> “We propose two novel model architectures for computing continuous vector representations of words from very large data sets.”
>
> ✓ verified: found word for word in the paper's own text


**Distributed Representations of Words and Phrases and their Compositionality** (2013) \cite{mikolov2013distributed}

What it did: Extend embeddings to phrases and improve training efficiency  *(the tool's reading)*

This work advances Skip-gram embeddings by introducing subsampling and Negative Sampling for faster training with higher quality. It extends embeddings from words to phrases, showing that Skip-gram vectors exhibit additive compositionality.  *(the tool's reading)*

> “Distributed representations of words in a vector space help learning algorithms to achieve better performance in natural language processing tasks by grouping similar words.”
>
> ✓ verified: found word for word in the paper's own text


**word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method** (2014) \cite{goldberg2014word}

What it did: Clarify theoretical foundations of negative sampling  *(the tool's reading)*

This paper provides rigorous mathematical derivation of the negative-sampling objective, explaining that it differs from the original softmax model and relates to the joint distribution of word and context. It clarifies that the model is non-convex when both word and context vectors are learned jointly.  *(the tool's reading)*

> “An assumption underlying the embedding process is the following: Assumption maximizing objective […] will result in good embeddings v_w ∀ w∈ V, in the sense that similar words will have similar vectors.”
>
> ✓ verified: found word for word in the paper's own text


**Distributed Representations of Sentences and Documents** (2014) \cite{le2014distributed}

What it did: Extend embeddings from words to variable-length documents  *(the tool's reading)*

This paper proposes Paragraph Vector to learn fixed-length embeddings from variable-length text units (sentences, paragraphs, documents). Unlike weighted averaging approaches, it captures word order and semantics without requiring parsing.  *(the tool's reading)*

> “After the training converges, words with similar meaning are mapped to a similar position in the vector space.”
>
> ✓ verified: found word for word in the paper's own text


**Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation** (2014) \cite{cho2014learning}

What it did: Show embeddings emerge from encoder-decoder translation models  *(the tool's reading)*

This work demonstrates that RNN Encoder-Decoder models, trained for machine translation scoring, incidentally learn continuous-space embeddings that capture semantic and syntactic structure. The learned representations cluster similar items together.  *(the tool's reading)*

> “We used rank-100 matrices, equivalent to learning an embedding of dimension 100 for each word.”
>
> ✓ verified: found word for word in the paper's own text


**Sequence to Sequence Learning with Neural Networks** (2014) \cite{sutskever2014sequence}

What it did: Apply embeddings to map variable-length sequences to fixed vectors  *(the tool's reading)*

This paper applies learned embeddings within LSTM sequence-to-sequence models to map variable-length sentences into fixed-dimensional vector representations. The learned representations prove sensitive to word order and relatively invariant to voice.  *(the tool's reading)*

> “We used deep LSTMs with 4 layers, with 1000 cells at each layer and 1000 dimensional word embeddings, with an input vocabulary of 160,000 and an output vocabulary of 80,000.”
>
> ✓ verified: found word for word in the paper's own text


**Show, Attend and Tell: Neural Image Caption Generation with Visual Attention** (2015) \cite{xu2015show}

What it did: Incorporate embeddings into attention-based multimodal models  *(the tool's reading)*

This paper uses learned word embedding matrices in attention-based image caption generation, mapping one-hot word representations into continuous vector space for LSTM decoder input. It contrasts learned embeddings with object-detector-based approaches.  *(the tool's reading)*

> “𝐄∈ℝ^m× K is an embedding matrix.”
>
> ✓ verified: found word for word in the paper's own text


**Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval** (2015) \cite{palangi2015deep}

What it did: Use LSTM-RNN to produce task-specific sentence embeddings  *(the tool's reading)*

This paper proposes LSTM-RNN for sentence embedding trained on weakly supervised click-through data, showing that different cells specialize to different topics. It demonstrates interpretable topic-allocation within the embedding through cell activation analysis.  *(the tool's reading)*

> “In text and language processing, one such problem is learning of an embedding vector for a sentence; that is, to train a model that can automatically transform a sentence to a vector that encodes the semantic meaning of the sentence.”
>
> ✓ verified: found word for word in the paper's own text


**Skip-Thought Vectors** (2015) \cite{kiros2015skip}

What it did: Create generic sentence embeddings via skip-thought  *(the tool's reading)*

This paper extends embeddings to the sentence level through skip-thought vectors, learning to predict surrounding sentences. It combines sentence embeddings with word2vec spaces and learns linear mappings to expand vocabulary for generic transfer learning.  *(the tool's reading)*

> “Let 𝒱_w2v denote the word embedding space of these word representations and let 𝒱_rnn denote the RNN word embedding space.”
>
> ✓ verified: found word for word in the paper's own text


**Bidirectional LSTM-CRF Models for Sequence Tagging** (2015) \cite{huang2015bidirectional}

> “It has been shown in […] that word embedding plays a vital role to improve sequence tagging performance. We downloaded[http://ronan.collobert.com/senna/] the embedding which has 130K vocabulary size and each word corresponds to a 50-dimensional embedding vector. To use this embedding, we simply replace the one hot encoding word representation with its corresponding 50-dimensional vector.”
>
> ✓ verified: found word for word in the paper's own text


**A large annotated corpus for learning natural language inference** (2015) \cite{bowman2015large}

What it did: Enable neural sentence embeddings via large NLI corpus  *(the tool's reading)*

This paper introduces the SNLI corpus, enabling training of neural sentence embedding models (sum-of-words, RNN, LSTM) that were previously uncompetitive on natural language inference. It provides controlled comparison of embedding architectures.  *(the tool's reading)*

> “To focus specifically on the strengths of these models at producing informative sentence representations, we use sentence embedding as an intermediate step in the NLI classification task: each model must produce a vector representation of each of the two sentences without using any context from the other sentence, and the two resulting vectors are then passed to a neural network classifier which predicts the label for the pair.”
>
> ✓ verified: found word for word in the paper's own text


**Exploring the Limits of Language Modeling** (2016) \cite{jozefowicz2016exploring}

What it did: Use character-level CNNs to generate word embeddings  *(the tool's reading)*

This paper extends embeddings by replacing fixed vocabulary lookup with character-level CNNs that generate embeddings dynamically. This approach reduces parameters, incorporates morphological knowledge, and naturally handles out-of-vocabulary words.  *(the tool's reading)*

> “The logit is generally computed as an inner product z_w = h^Te_w where h is a context vector and e_w is a “word embedding” for w.”
>
> ✓ verified: found word for word in the paper's own text


**Enhanced LSTM for Natural Language Inference** (2016) \cite{chen2016enhanced}

> “The 𝐚_i or 𝐛_j ∈ℝ^l is an embedding of l-dimensional vector, which can be initialized with some pre-trained word embeddings and organized with parse trees.”
>
> ✓ verified: found word for word in the paper's own text


**Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation** (2016) \cite{wu2016google}

> “such as embedding lookup and attention module, remain on the CPU, and all other quantized operations are off-loaded to the TPU.”
>
> ✓ verified: found word for word in the paper's own text


**ConceptNet 5.5: An Open Multilingual Graph of General Knowledge** (2016) \cite{speer2016conceptnet}

What it did: Combine distributional and knowledge-graph embeddings  *(the tool's reading)*

This paper introduces ConceptNet Numberbatch, a hybrid embedding space combining distributional embeddings (word2vec, GloVe) with ConceptNet-derived embeddings via retrofitting. Multilingual embeddings are aligned through global linear projection.  *(the tool's reading)*

> “Word embeddings represent words as dense unit vectors of real numbers, where vectors that are close together are semantically related.”
>
> ✓ verified: found word for word in the paper's own text


**A Structured Self-attentive Sentence Embedding** (2017) \cite{lin2017structured}

What it did: Extend sentence embeddings from vectors to 2-D matrices  *(the tool's reading)*

This paper proposes self-attention mechanism producing 2-D matrix sentence embeddings where each row is a weighted sum of bidirectional LSTM outputs. It introduces a novel penalization term to encourage diversity across attention hops.  *(the tool's reading)*

> “Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.”
>
> ✓ verified: found word for word in the paper's own text


**Supervised Learning of Universal Sentence Representations from Natural\n Language Inference Data** (2017) \cite{conneau2017supervised}

What it did: Train universal sentence embeddings on supervised NLI  *(the tool's reading)*

This paper shows supervised sentence embeddings trained on NLI data consistently outperform unsupervised methods, introducing BiLSTM with max-pooling for faster training. It demonstrates that NLI-trained embeddings transfer effectively across tasks.  *(the tool's reading)*

> “Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features.”
>
> ✓ verified: found word for word in the paper's own text


**Attention Is All You Need** (2017) \cite{vaswani2017attention}

What it did: Scale embeddings to purely attention-based architectures  *(the tool's reading)*

This paper introduces the Transformer, scaling embeddings within a purely attention-based sequence transduction model. It scales embedding weights by sqrt(d_model) and ties them with the pre-softmax transformation for the first time.  *(the tool's reading)*

> “Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model.”
>
> ✓ verified: found word for word in the paper's own text


**Deep Contextualized Word Representations** (2018) \cite{peters2018deep}

What it did: Learn deep contextualized embeddings from bidirectional LM  *(the tool's reading)*

This paper introduces ELMo, which treats embeddings as task-specific linear combinations of all internal biLM layers rather than just the top layer. Lower layers capture syntax while higher layers capture semantics, with task-specific weighting improving performance.  *(the tool's reading)*

> “Our representations differ from traditional word type embeddings in that each token is assigned a representation that is a function of the entire input sentence.”
>
> ✓ verified: found word for word in the paper's own text


**Universal Sentence Encoder** (2018) \cite{cer2018universal}

What it did: Produce general-purpose sentence embeddings via multi-task learning  *(the tool's reading)*

This paper introduces Universal Sentence Encoder using Transformer-based and Deep Averaging Network models trained via multi-task learning. It shows sentence-level embeddings outperform word-level embeddings for transfer learning across diverse tasks.  *(the tool's reading)*

> “The models take as input English strings and produce as output a fixed dimensional embedding representation of the string.”
>
> ✓ verified: found word for word in the paper's own text


**Cross-lingual Language Model Pretraining** (2019) \cite{lample2019cross}

What it did: Learn cross-lingual embeddings from shared vocabulary LM  *(the tool's reading)*

This paper introduces cross-lingual language modeling to learn shared cross-lingual embeddings from a unified vocabulary. It shows that XLM word embeddings achieve better cross-lingual word similarity than previous translation-based methods.  *(the tool's reading)*

> “suggest that it is possible to build universal cross-lingual encoders that can encode any sentence into a shared embedding space.”
>
> ✓ verified: found word for word in the paper's own text


**BERTScore: Evaluating Text Generation with BERT** (2019) \cite{zhang2019bertscore}

What it did: Apply contextual embeddings to evaluate text generation  *(the tool's reading)*

This paper uses contextual embeddings from BERT for semantic similarity measurement in text generation evaluation, unlike prior metrics relying on static embeddings. Token-level computation with learned importance weighting replaces sentence-level representations.  *(the tool's reading)*

> “In contrast to prior word embeddings […], contextual embeddings, such as BERT […] and ELMo […], can generate different vector representations for the same word in different sentences depending on the surrounding words, which form the context of the target word.”
>
> ✓ verified: found word for word in the paper's own text


**XLNet: Generalized Autoregressive Pretraining for Language Understanding** (2019) \cite{yang2019xlnet}

> “and e(x) denotes the embedding of x.”
>
> ✓ verified: found word for word in the paper's own text


_[Your synthesis: what it enabled, what it left unsolved.]_


## Where this thread connects

Each crossing is where one line of work fed another. These are the tool's reading of
the corpus, not quotations.

- **Word & Text Embeddings** → **Multimodal & Cross-Modal Embeddings** (2013): Semantic bridges
- **Word & Text Embeddings** → **Self-Supervised & Contrastive Embeddings** (2019): Contextual pre-training