Latent Representation & Generative Spaces

The paper introduced a reparameterization technique that enables the variational lower bound to be optimized using standard stochastic gradient methods with low variance.

“First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods.”◌ not checked against the paper’s text as it now stands

The paper used neural networks as recognition models to learn an embedding that approximates the intractable posterior distribution of latent variables.

“When a neural network is used for the recognition model, we arrive at the variational auto-encoder.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that learned latent embeddings can be used for representation and visualization purposes through the recognition model.

“The learned approximate posterior inference model can also be used for a host of tasks such as recognition, denoising, representation and visualization purposes.”◌ not checked against the paper’s text as it now stands

The paper introduces a recognition model that serves as a stochastic encoder to embed observed data into a latent variable space for variational inference.

“we use introduce an approximate representation of the posterior over the latent variables using a recognition model that acts as a stochastic encoder of the data.”◌ not checked against the paper’s text as it now stands

The paper specifies the approximate posterior as a Gaussian distribution where the mean and covariance are functions learned by deep neural networks that map from observed data to latent variable parameters.

“where the mean _l(·) and covariance _l(·) are generic maps represented by deep neural networks.”◌ not checked against the paper’s text as it now stands

The paper uses a recognition model to perform amortised inference, enabling generalisation across posterior estimates for all latent variables through a learned mapping from data to embedding distributions.

“A recognition model allows us introduce a form of amortised inference […] for variational methods in which we share statistical strength by allowing for generalisation across the posterior estimates for all latent variables using a model.”◌ not checked against the paper’s text as it now stands

The paper developed latent feature discriminative models (M1) that use deep generative models to provide robust embeddings of data for classification.

“A commonly used approach is to construct a model that provides an embedding or feature representation of the data. Using these features, a separate classifier is thereafter trained. The embeddings allow for a clustering of related observations in a latent feature space that allows for accurate classification, even with a limited number of labels. Instead of a linear embedding, or features obtained from a regular auto-encoder, we construct a deep generative model of the data that is able to provide a more robust set of latent features.”◌ not checked against the paper’s text as it now stands

The paper used inference networks parameterized as deep neural networks to amortize the cost of inference and compute global variational parameters instead of per-data point parameters.

“Using an inference network, we avoid the need to compute per data point variational parameters, but can instead compute a set of global variational parameters. This allows us to amortise the cost of inference by generalising between the posterior estimates for all latent variables through the parameters of the inference network, and allows for fast inference at both training and testing time”◌ not checked against the paper’s text as it now stands

The paper demonstrated that stacking generative models (M1+M2) learns low-dimensional embeddings that separate class information from intra-class variability.

The paper demonstrated that DCGANs learn useful embeddings in the discriminator that can be extracted and used as features for image classification tasks.

“To evaluate the quality of the representations learned by DCGANs for supervised tasks, we train on Imagenet-1k and then use the discriminator's convolutional features from all layers, maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM classifier is trained on top of them.”◌ not checked against the paper’s text as it now stands

The paper showed that the generator learns a latent space embedding where semantic changes occur smoothly when walking between points, indicating the space encodes meaningful representations.

“Walking on the manifold that is learnt can usually tell us about signs of memorization (if there are sharp transitions) and about the way in which the space is hierarchically collapsed. If walking in this latent space results in semantic changes to the image generations (such as objects being added and removed), we can reason that the model has learned relevant and interesting representations.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that the Z latent space embeddings have linear structure where arithmetic operations on averaged Z vectors produce semantically meaningful results.

The paper learns an embedding that disentangles transformations like rotation and lighting into separate latent variables.

“This representation is disentangled with respect to transformations such as out-of-plane rotations and lighting variations.”◌ not checked against the paper’s text as it now stands

The paper creates embeddings where specific neurons in the graphics code layer represent particular transformations through a novel training procedure.

“We propose a training procedure to encourage each group of neurons in the graphics code layer to distinctly represent a specific transformation.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that the learned embedding can be decomposed into interpretable factors corresponding to scene properties like pose, light, and shape.

“Z can be a disentangled representation containing a factored set of latent variables z_i ∈ Z such as pose, light and shape.”◌ not checked against the paper’s text as it now stands

The paper shows that the embedding achieves equivariance between latent variables and their corresponding generating parameters.

The paper uses context encoders to learn visual feature embeddings by training a convolutional neural network to predict missing image regions from surrounding context.

“By analogy with auto-encoders, we propose Context Encoders – a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that context encoders learn a semantic embedding representation that captures both appearance and the semantics of visual structures.

“We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures.”◌ not checked against the paper’s text as it now stands

The paper validates the quality of learned embeddings by fine-tuning the encoder features for image understanding tasks including classification, detection, and segmentation.

“We further validate the quality of the learned feature representation by fine-tuning the encoder for a variety of image understanding tasks, including classification, object detection, and semantic segmentation.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that conditional PixelCNN can be conditioned on latent embeddings created by other networks to generate diverse images.

“The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks.”◌ not checked against the paper’s text as it now stands

The paper shows that embeddings from a convolutional network trained on face images capture high-level facial features that enable generation of varied portraits.

“We can see that the embeddings capture a lot of the facial features of the source image and the generative model is able to produce a large variety of new faces with these features in new poses, lighting conditions, etc.”◌ not checked against the paper’s text as it now stands

The paper uses embeddings in an autoencoder framework where the encoder learns to extract high-level abstract information rather than low-level pixel statistics.

“since so much of the low level pixel statistics can be handled by the PixelCNN, the encoder should be able to omit these from h⃗ and concentrate instead on more high-level abstract information.”◌ not checked against the paper’s text as it now stands

The paper used speaker embeddings as a form of global conditioning to enable a single WaveNet model to generate speech from different speakers.

“a speaker embedding in a TTS model”◌ not checked against the paper’s text as it now stands

The paper employed embedding-based conditioning through one-hot vector representations of speaker identity to control which speaker the model generated.

“The conditioning was applied by feeding the speaker ID to the model in the form of a one-hot vector.”◌ not checked against the paper’s text as it now stands

The paper explains how to encode classical data vectors into quantum states using quantum random access memory for use in quantum machine learning algorithms.

“we choose a data vector v⃗_j at random, and use a quantum random access memory (qRAM […]) to map that vector into a quantum state: v⃗_j → |v_j⟩.”◌ not checked against the paper’s text as it now stands

The paper describes how quantum principal component analysis decomposes data vectors into principal components by embedding them as quantum states and using quantum algorithms.

“allowing one to take the quantum version of any data vector |v⃗⟩ and to decompose it into the principal components |c_k⟩, revealing the eigenvalue of C at the same time:|v⟩→∑_k v_k |c_k⟩|ẽ_k⟩.”◌ not checked against the paper’s text as it now stands

The paper discusses how quantum support vector machines process data by representing it as quantum states for classification tasks.

“The data input can come from various sources, such as from qRAM accessing classical data or a quantum subroutine preparing quantum states. Once the data are made available to the quantum computing device, they are processed with quantum phase estimation and matrix inversion (the HHL algorithm).”◌ not checked against the paper’s text as it now stands

The paper introduces style token embeddings that compress prosody information from variable-length audio into fixed-length vectors for conditioning speech synthesis.

“The, proposed in, compresses the prosody of a variable-length audio signal into a fixed-length vector, which we call the.”◌ not checked against the paper’s text as it now stands

The paper learns style embeddings through a style token layer that uses attention to determine similarity between a reference embedding and a bank of learned token embeddings without requiring explicit style labels.

“The reference embedding is passed to a style token layer, where it is used as the query vector to an. Here, attention is not used to learn an alignment. Instead, it learns a similarity measure between the reference embedding and each token in a bank of.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that style embeddings can be viewed as external memory storing style information extracted from training data, with the reference signal guiding memory operations at both training and inference time.

“GST embeddings can also be viewed as an external memory that stores style information extracted from training data. The reference signal guides memory writes at training time, and memory reads at inference time.”◌ not checked against the paper’s text as it now stands

This paper proposed Contrastive Predictive Coding as a method to learn representations by compressing high-dimensional data into compact latent embeddings and predicting future information in that embedding space.

“first, we compress high-dimensional data into a much more compact latent embedding space in which conditional predictions are easier to model. Secondly, we use powerful autoregressive models in this latent space to make predictions many steps in the future.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that CPC learns useful embeddings across multiple modalities by achieving strong performance on speech, images, text, and reinforcement learning tasks.

“We apply the resulting model, Contrastive Predictive Coding (CPC) to widely different data modalities, images, speech, natural language and reinforcement learning, and show that the same mechanism learns interesting high-level information on each of these domains, outperforming other approaches.”◌ not checked against the paper’s text as it now stands

The paper showed that CPC embeddings capture high-level semantic information such as phone identity and speaker identity in speech, as evidenced by strong performance on phone and speaker classification tasks.

“Interestingly, CPCs capture both speaker identity and speech contents, as demonstrated by the good accuracies attained with a simple linear classifier, which also gets close to the oracle, fully supervised networks.”

The paper introduces variational autoencoders as a framework that learns representations of data through a recognition model that maps inputs to latent variables.

“The VAE can be viewed as two coupled, but independently parameterized models: the encoder or recognition model, and the decoder or generative model.”◌ not checked against the paper’s text as it now stands

The paper explains that VAEs use amortized inference to learn a single set of parameters that models the relation between input and latent variables, making representations more efficient than traditional variational inference.

The paper describes how VAEs create meaningful learned representations of data through the coupling of encoder and decoder models that support each other.

“The generative model is a scaffolding of sorts for the recognition model to learn meaningful representations of the data, including possibly class-labels.”◌ not checked against the paper’s text as it now stands

The paper positions VAEs as a framework for unsupervised representation learning that discovers disentangled, semantically meaningful factors of variation in data.

“This quest for disentangled, semantically meaningful, statistically independent and causal factors of variation in data is generally known as unsupervised representation learning, and the variational autoencoder (VAE) has been extensively employed for that purpose.”

This paper applied contrastive learning to learn embeddings that maximize mutual information between representations of different image channel views of the same scene.

“We learn a feature embedding such that views of the same scene map to nearby points (measured with Euclidean distance in representation space) while views of different scenes map to far apart points.”◌ not checked against the paper’s text as it now stands

This paper extended contrastive learning to learn embeddings from more than two views and showed that representation quality improves with more views.

“We extend the framework to learn from more than two views, and show that the quality of the learned representation improves as number of views increase. Ours is the first work to explicitly show the benefits of multiple views on representation quality.”◌ not checked against the paper’s text as it now stands

This paper demonstrated that embeddings learned via contrastive loss outperform those learned from cross-view prediction objectives.

“we compare the contrastive formulation of multiview learning to the recently popular approach of cross-view prediction, and find that in head-to-head comparisons, the contrastive approach learns stronger representations.”◌ not checked against the paper’s text as it now stands

The paper introduces data re-uploading as a technique to encode classical data multiple times throughout a quantum circuit to overcome single-qubit limitations.

“The central idea to build a universal quantum classifier with a single qubit is thus to re-upload classical data along with the computation.”◌ not checked against the paper’s text as it now stands

The paper proposes that data is embedded as rotations of qubits, specifically using three-dimensional data vectors that can be re-uploaded using unitaries.

“Classical data are re-introduced several times in a sequence interspaced with processing units. We shall consider the introduction of data as a rotation of the qubit. This means that data from three-dimensional space, x⃗, can be re-uploaded using unitaries that rotate the qubit U(x⃗).”◌ not checked against the paper’s text as it now stands

The paper demonstrates that data points are embedded linearly into rotational gates, with non-linearities arising from the structure of subsequent gates.

“Notice that data points are introduced linearly into the rotational gate. Non-linearities will come from the structure of these gates.”◌ not checked against the paper’s text as it now stands

VL-BERT extends the Transformer model to accept both visual and linguistic embedded features as input, where each element is either a word or a region-of-interest from an image.

“VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image.”◌ not checked against the paper’s text as it now stands

VL-BERT creates visual feature embeddings by combining visual appearance features from Fast R-CNN with visual geometry embeddings that encode the spatial location of regions-of-interest.

“The visual feature embedding is attached to each of the input elements, which is the output of a fully connected layer taking the concatenation of visual appearance feature and visual geometry embedding as input.”◌ not checked against the paper’s text as it now stands

VL-BERT incorporates a visual feature embedding as a new type of embedding added to token embedding, segment embedding, and sequence position embedding for each input element.

“For each input element, its embedding feature is the summation of four types of embedding, namely, token embedding, visual feature embedding, segment embedding, and sequence position embedding. Among them, the visual feature embedding is newly introduced for capturing visual clues, while the other three embeddings follow the design in the original BERT paper.”

BYOL learns image representations by training an online network to predict the target network's representation of the same image under different augmentations.

BYOL uses a projection step to embed the encoder output into a smaller dimensional space before computing the prediction loss.

“the representation y corresponds to the output of the final average pooling layer, which has a feature dimension of 2048 (for a width multiplier of 1×). As in […], the representation y is projected to a smaller space by a multi-layer perceptron (MLP) g_, and similarly for the target projection g_.”◌ not checked against the paper’s text as it now stands

BYOL's target network produces embeddings that serve as regression targets for training the online network's projections.

“The target network provides the regression targets to train the online network, and its parameters are an exponential moving average of the online parameters”◌ not checked against the paper’s text as it now stands

The paper proposes generating CLIP image embeddings from text captions using a prior model, where these embeddings serve as an intermediate representation for image generation.

“To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that CLIP embeddings can be used to enable semantic image manipulations by moving in the direction of encoded text vectors in the embedding space.

“one notable advantage of using the CLIP latent space is the ability to semantically modify images by moving in the direction of any encoded text vector”◌ not checked against the paper’s text as it now stands

The paper reduces the dimensionality of CLIP image embeddings using PCA to make the autoregressive prior more efficient.

“To train and sample from the AR prior more efficiently, we first reduce the dimensionality of the CLIP image embeddings z_i by applying Principal Component Analysis (PCA)”◌ not checked against the paper’s text as it now stands