Domain-Specific Metric Embeddings

This paper introduced an embedding-based approach by showing how matrix factorization models with latent feature vectors can be optimized for ranking tasks.

“Each row wu in W can be seen as a feature vector describing a user u and similarly each row hi of H describes an item i. Thus the prediction formula can also be written as: ˆxui = ⟨wu, hi⟩= k X f=1 wuf · hif”◌ not checked against the paper’s text as it now stands

This paper demonstrated that embeddings learned through BPR optimization produce better personalized rankings than embeddings learned with standard techniques.

“For the task of ranking, i.e. estimating whether a user prefers one item over another, a better approach is to optimize against the BPR-Opt criterion. This can be achieved by using our proposed algorithm LearnBPR.”◌ not checked against the paper’s text as it now stands

This paper formalized embeddings as latent factors that model hidden user preferences and item properties in the context of personalized ranking.

The paper uses word embeddings as input features to the neural network models, replacing one-hot encoding with dense vectors.

“To use this embedding, we simply replace the one hot encoding word representation with its corresponding 50-dimensional vector.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that the BI-LSTM-CRF model has reduced dependence on word embeddings compared to previous Conv-CRF approaches.

“It is interesting that our best model BI-LSTM-CRF has less dependence on Senna word embedding compared to Conv-CRF model.”◌ not checked against the paper’s text as it now stands

FaceNet directly learns a mapping from face images to a compact Euclidean space where distances correspond to face similarity.

“FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity.”◌ not checked against the paper’s text as it now stands

FaceNet trains a deep convolutional network to directly optimize the embedding itself rather than using an intermediate bottleneck layer.

“Our method uses a deep convolutional network trained to directly optimize the embedding itself, rather than an intermediate bottleneck layer as in previous deep learning approaches.”◌ not checked against the paper’s text as it now stands

FaceNet employs a triplet-based loss function to train embeddings where the squared L2 distances in embedding space correspond to face similarity.

“The network is trained such that the squared L2 distances in the embedding space directly correspond to face similarity: faces of the same person have small distances and faces of distinct people have large distances.”◌ not checked against the paper’s text as it now stands

This paper proposes a siamese network that learns CNN embeddings by combining identification and verification losses to create more discriminative pedestrian descriptors.

“we propose a siamese network that simultaneously computes the identification loss and verification loss. Given a pair of training images, the network predicts the identities of the two input images and whether they belong to the same identity. Our network learns a discriminative embedding and a similarity measurement at the same time, thus making full usage of the re-ID annotations.”◌ not checked against the paper’s text as it now stands

This paper demonstrates that the proposed embedding approach improves state-of-the-art performance on person re-identification benchmarks by combining the complementary strengths of verification and identification models.

“Motivated by these properties, this work proposes to combine the strengths of the two networks and leverage their complementary nature to improve the discriminative ability of the learned embeddings. The proposed model is a siamese network that predicts person identities and similarity scores at the same time.”◌ not checked against the paper’s text as it now stands

This paper shows that their CNN embedding with cross-entropy loss outperforms using contrastive loss for verification in person re-identification.

This paper introduced depthwise separable convolutions as the core building block for efficient embeddings in MobileNet models.

“MobileNets are based on a streamlined architecture that uses depthwise separable convolutions to build light weight deep neural networks.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that depthwise separable convolutions reduce computational cost by factorizing standard convolutions into separate filtering and combining steps, achieving 8-9 times less computation than standard convolutions.

“MobileNet uses 3 × 3 depthwise separable convolutions which uses between 8 to 9 times less computation than standard convolutions at only a small reduction in accuracy as seen in Section […].”◌ not checked against the paper’s text as it now stands

This paper proposed width and resolution multipliers as hyperparameters to create efficient embedded representations of varying sizes for different resource constraints.

“We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem.”◌ not checked against the paper’s text as it now stands

This paper introduces Deep Speaker, a neural system that maps utterances to a hypersphere where speaker similarity is measured by cosine similarity.

“We present Deep Speaker, a neural speaker embedding system that maps utterances to a hypersphere where speaker similarity is measured by cosine similarity.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that Deep Speaker can be trained using triplet loss based on cosine similarity to generate utterance-level speaker embeddings from acoustic features.

“We experiment with ResCNN and GRU architectures to extract the acoustic features, then mean pool to produce utterance-level speaker embeddings, and train using triplet loss based on cosine similarity.”◌ not checked against the paper’s text as it now stands

This paper shows that embeddings generated by Deep Speaker can be applied to multiple speaker recognition tasks including identification, verification, and clustering.

“The embeddings generated by Deep Speaker can be used for many tasks, including speaker identification, verification, and clustering.”◌ not checked against the paper’s text as it now stands

The paper discusses performing activation maximization in code space using generative models like GANs to find prototypes of learned concepts by decoding optimized latent codes into the input domain.

“One such model is the generative adversarial network […]. It learns a decoding function g such that the generated data distribution is as hard as possible to discriminate from the true data distribution. The decoding function g is learned in competition with a discriminant between the generated and the true distributions. The decoding function and the discriminant are typically chosen to be multilayer neural networks.”◌ not checked against the paper’s text as it now stands

The paper explains that when interpreting DNNs, abstract concepts learned in top-layer neurons must be mapped to interpretable domains such as images or text, contrasting with non-interpretable domains like word embeddings.

“Examples of domains that are interpretable are images (arrays of pixels), or texts (sequences of words). A human can look at them and read them respectively. Examples of domains that are not interpretable are abstract vector spaces (e.g. word embeddings […]), or domains composed of undocumented input features (e.g. sequences with unknown words or symbols).”◌ not checked against the paper’s text as it now stands

The paper presents layer-wise relevance propagation as a technique that propagates prediction scores backward through a DNN's feed-forward graph structure to decompose and explain decisions at the input level.

This paper proposed attentive statistics pooling as a new method for deep speaker embedding that combines attention mechanisms with higher-order statistics.

“In this paper, we propose a new pooling method, called attentive statistics pooling, that provides importance-weighted standard deviations as well as the weighted means of frame-level features, for which the importance is calculated by an attention mechanism.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that attentive statistics pooling captures long-term variations in speaker characteristics more effectively than conventional methods.

“In this way, it can capture long-term variations in speaker characteristics more effectively.”◌ not checked against the paper’s text as it now stands

This paper achieved substantial performance improvements, reducing equal error rates by 7.5% on NIST SRE 2012 and 8.1% on VoxCeleb compared to conventional statistics pooling.

“An evaluation on the NIST SRE 2012 and the VoxCeleb data sets shows that it reduces equal error rates (EERs) from the conventional method by 7.5% and 8.1%, respectively.”◌ not checked against the paper’s text as it now stands

The paper developed models that encode sentences into fixed-dimensional embedding vectors specifically designed for transfer learning to other NLP tasks.

“We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks.”◌ not checked against the paper’s text as it now stands

The paper created two variants of sentence encoding models that allow trade-offs between embedding quality and computational efficiency.

“Two variants of the encoding models allow for trade-offs between accuracy and compute resources.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that sentence-level embeddings outperform word-level embeddings for transfer learning across multiple NLP tasks.

“We find that transfer learning using sentence embeddings tends to outperform word level transfer.”◌ not checked against the paper’s text as it now stands

This paper trained a speaker encoder network on speaker verification to generate fixed-dimensional embedding vectors from reference speech.

“a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech without transcripts from thousands of speakers, to generate a fixed-dimensional embedding vector from only seconds of reference speech from a target speaker”◌ not checked against the paper’s text as it now stands

The paper demonstrated that speaker embeddings learned from speaker verification transfer to condition a text-to-speech synthesis network on speaker identity.

“the synthesis network is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task”◌ not checked against the paper’s text as it now stands

The paper used speaker embeddings computed by the encoder network to condition the Tacotron 2 synthesis network at each time step.

“An embedding vector for the target speaker is concatenated with the synthesizer encoder output at each time step.”◌ not checked against the paper’s text as it now stands

The paper showed that randomly sampled speaker embeddings can synthesize speech from novel speakers, indicating the model learned a high quality speaker representation.

The paper embeds users and items in memory matrices as learned representations of their preferences and attributes.

“The memory component consists of a user memory matrix 𝐌∈ℝ^P × d and an item memory matrix 𝐄∈ℝ^Q × d, where P and Q represents the number of users and items respectively and d denotes the size (dimensionality) of each memory cell. Each user u is embedded in a memory slot 𝐦_u ∈𝐌 storing her specific preferences. Similarly, each item i corresponds to another memory slot 𝐞_i ∈𝐄 encoding the item's specific attributes.”◌ not checked against the paper’s text as it now stands

The paper uses embeddings in an external memory matrix to store long-term information about each user's role in the neighborhood.

“where 𝐜_v is another embedding vector for user v which is called external memory in the original memory network framework […]. Denoting the v^th column of the embedding matrix 𝐂 with the same dimensions as 𝐌. The external memory allows the storage of long-term information pertaining specifically to each user's role in the neighborhood.”◌ not checked against the paper’s text as it now stands

The paper applies elementwise product operations on user and item embeddings followed by projections to integrate global user-item interactions in the output module.

“For a given user u and item i the ranking score is given as: r̂_ui = 𝐯^𝖳ϕ(𝐔(𝐦_u ⊙𝐞_i) + 𝐖𝐨_ui + 𝐛) where ⊙ is the elementwise product”

This paper proposed a method to learn image representations that are invariant to image transformations by minimizing a contrastive loss between representations of original and transformed images.

“Pretext-Invariant Representation Learning constructs image representations that are similar to the representation of transformed versions of the same image and different from the representations of other images.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that invariant representations substantially outperform covariant representations across multiple vision tasks including object detection and image classification.

“We find that substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning.”◌ not checked against the paper’s text as it now stands

This paper showed that learned representations can be effectively used as feature embeddings for downstream tasks by training linear classifiers on fixed representations extracted from the pre-trained network.

“We follow the evaluation setup from […] and measure the performance of such classifiers on four image-classification datasets: […], […], […], and iNaturalist2018 […]. These datasets involve diverse tasks such as object classification, scene recognition and fine-grained recognition.”

The paper identifies feature embeddings from intermediate layers as a key form of knowledge that can be transferred from teacher to student models for improved learning.

“The activations, neurons or features of intermediate layers also can be used as the knowledge to guide the learning of the student model”◌ not checked against the paper’s text as it now stands

The paper discusses feature-based knowledge distillation that matches feature embeddings between teacher and student models using transformation functions when their shapes differ.

“The transformation functions, Φ_t(f_t(x)) and Φ_s(f_s(x)), are usually applied when the feature maps of teacher and student models are not in the same shape.”◌ not checked against the paper’s text as it now stands

The paper presents relation-based knowledge that leverages embeddings and their relationships, including instance-level feature embeddings that preserve similarities between samples.

“Based on idea of manifold learning, the student network is learned by feature embedding, which preserves the feature similarities of samples in the intermediate layers of the teacher networks”◌ not checked against the paper’s text as it now stands

The paper explains that deep learning extracts high-level features through hierarchical feature learning where upper layer features are generated from previous layers.

“In this era of machine learning, deep learning automates the extraction of high-level features via hierarchical feature learning mechanism wherein the upper layer of features are generated on the previous set of layer/layers.”◌ not checked against the paper’s text as it now stands

The paper discusses how ensemble deep learning models combine multiple deep learning architectures to generate diverse feature representations for improved performance.

“Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance.”◌ not checked against the paper’s text as it now stands

The paper reviews boosting methods that learn complex feature representations iteratively through multiple models, such as boosted deep belief networks for facial expression recognition.

“Boosted deep belief network (DBN) […] for facial expression recognition unified the boosting technique and multiple DBN's via objective function which results in a strong classifier. The model learns complex feature representation to build a strong classifier in an iterative manner.”◌ not checked against the paper’s text as it now stands