Dimensionality Reduction & Visualization Embeddings

The paper introduced DeCAF, a feature representation extracted from deep convolutional network activations that can be transferred to novel visual recognition tasks.

“We investigate semi-supervised multi-task learning of deep convolutional representations, where representations are learned on a set of related problems but applied to new tasks which have too few training examples to learn a full deep representation. Our model can either be considered as a deep architecture for transfer learning based on a supervised pre-training phase, or simply as a new visual feature DeCAF defined by the convolutional network weights learned on a set of pre-defined object recognition tasks.”◌ not checked against the paper’s text as it now stands

The paper demonstrated that deep convolutional embeddings exhibit semantic clustering properties that enable the features to generalize across datasets and tasks.

“More interestingly, in Figure […] we can see the top performing features (DeCAF_6) on the SUN-397 dataset. Even there, the features show very good clustering of semantic classes (e.g., indoor vs. outdoor). This suggests DeCAF is a good feature for general object recognition tasks. Consider the case where the object class that we are trying to detect is not in the original object pool of ILSVRC-2012. The fact that these features cluster several intermediate nodes of WordNet implies that these features are an excellent starting point for generalizing to unseen classes.”◌ not checked against the paper’s text as it now stands

The paper showed that embeddings from deeper network layers capture richer semantic information than shallower layers when visualized in 2D embedding space.

The paper released DeCAF embeddings and pre-trained network parameters to enable other researchers to use deep convolutional features for visual recognition tasks without retraining.

“We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.”◌ not checked against the paper’s text as it now stands

The paper introduced a recognition model as a stochastic encoder that provides an approximate posterior distribution over latent variables represented as embeddings.

“we use introduce an approximate representation of the posterior over the latent variables using a recognition model that acts as a stochastic encoder of the data.”◌ not checked against the paper’s text as it now stands

The paper used deep neural networks to parameterize the mean and covariance of the recognition model, enabling flexible embeddings of the posterior distribution.

The paper demonstrated that the recognition model learns to embed data points into a latent space where samples from the model concentrate on the true posterior mass.

“Samples from the recognition model (blue) are concentrated on the posterior mass, indicating that the recognition model has learnt the correct posterior statistics, which should lead to efficient learning.”◌ not checked against the paper’s text as it now stands

The paper embeds fused features from multiple layers into reproducing kernel Hilbert spaces to match distributions for feature adaptation.

“We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation.”◌ not checked against the paper’s text as it now stands

The paper uses tensor product to perform lossless multi-layer feature fusion for embedding features before applying MMD.

“To adapt multiple feature layers effectively, we propose the tensor product between features of multiple layers to perform lossless multi-layer feature fusion, i.e. 𝐳_i^s ≜⊗ _ℓ∈ℒ𝐱_i^sℓ and 𝐳_j^t ≜⊗ _ℓ∈ℒ𝐱_j^tℓ.”◌ not checked against the paper’s text as it now stands

The paper reduces feature dimensions by adding a bottleneck layer on top of the last feature layer of CNNs for feature embedding.

The paper defines embedding approaches for non-Euclidean domains by describing methods that map data points into low-dimensional spaces while preserving affinity structure.

“Many methods for non-linear dimensionality reduction consist of two steps: first, they start with constructing a representation of local affinity of the data points (typically, a sparsely connected graph). Second, the data points are embedded into a low-dimensional space trying to preserve some criterion of the original affinity.”◌ not checked against the paper’s text as it now stands

The paper discusses embedding words and vertices using techniques adapted from successful word embedding models applied to graph data.

“Most recent approaches […] tried to apply the successful word embedding model […] to graphs. Instead of embedding the vertices, the graph structure can be processed by decomposing it into small sub-graphs called motifs […] or graphlets […].”◌ not checked against the paper’s text as it now stands

The paper reviews spectral embedding methods including Laplacian eigenmaps and diffusion maps as approaches to non-linear dimensionality reduction for manifold learning.

“Examples of manifold learning include different flavors of multidimensional scaling (MDS) […], locally linear embedding (LLE) […], stochastic neighbor embedding (t-SNE) […], spectral embeddings such as Laplacian eigenmaps […] and diffusion maps […], and deep models […].”

This paper introduced UMAP, a novel manifold learning technique that constructs topological representations by patching together local fuzzy simplicial sets from the high-dimensional data.

“UMAP uses local manifold approximations and patches together their local fuzzy simplicial set representations to construct a topological representation of the high dimensional data.”◌ not checked against the paper’s text as it now stands

This paper developed a mathematical framework using fuzzy simplicial sets and category theory to convert metric spaces into topological representations that can be merged into a consistent global structure for embedding.

“This can be done in a natural way by converting the metric spaces into fuzzy simplicial sets. We will use functors between the relevant categories to convert from metric spaces to fuzzy topological representations.”◌ not checked against the paper’s text as it now stands

This paper defined the fuzzy singular set functor and fuzzy realization functor as adjoint functors to translate between finite metric spaces and fuzzy simplicial sets for dimension reduction purposes.

The paper proposes embedding data into a quantum circuit through multiple re-uploads of classical data as rotations of qubits.

The paper embeds data linearly into rotational gates in the quantum classifier.

“Notice that data points are introduced linearly into the rotational gate. Non-linearities will come from the structure of these gates.”◌ not checked against the paper’s text as it now stands

The paper uses an embedding strategy where data is encoded with weights in a Hadamard product form for each layer of the quantum circuit.

“each layer gate can be taken as L(i)=U(θ⃗_i+w⃗_i∘x⃗), where w⃗_i∘x⃗=(w_i^1x^1,w_i^2x^2,w_i^3x^3) is the Hadamard product of two vectors.”◌ not checked against the paper’s text as it now stands

The paper embeds the final quantum state onto the Bloch sphere as a geometric feature space for classification.

“Data embedded in this feature space can be easily separated into classes employing the regions on the Bloch sphere.”◌ not checked against the paper’s text as it now stands

The paper defines interpretable machine learning as the use of machine-learning models for the extraction of relevant knowledge about domain relationships contained in data.

“We define interpretable machine learning as the use of machine-learning models for the extraction of relevant knowledge about domain relationships contained in data.”◌ not checked against the paper’s text as it now stands

The paper introduces the PDR framework with three desiderata for evaluating interpretations: predictive accuracy, descriptive accuracy, and relevancy.

The paper categorizes interpretation methods into model-based and post-hoc categories with sub-groups including sparsity, modularity and simulatability.

“we introduce a categorization of existing techniques into model-based and post-hoc categories, with sub-groups including sparsity, modularity and simulatability.”◌ not checked against the paper’s text as it now stands

The paper defines descriptive accuracy as the degree to which an interpretation method objectively captures the relationships learned by machine learning models.

“We define descriptive accuracy, in the context of interpretation, as the degree to which an interpretation method objectively captures the relationships learned by machine learning models.”

The paper demonstrates that embeddings learned through contrastive learning are shaped by the mutual information shared between different views of the data.

The paper shows that optimal embeddings achieve a balance where they contain task-relevant information while minimizing shared irrelevant information between views.

“We hypothesize that the best performing views will be close to the sweet spot: containing as much task-relevant information while discarding as much irrelevant information in the input as possible.”◌ not checked against the paper’s text as it now stands

The paper proves that minimal sufficient encoders produce optimal embeddings by extracting only the task-relevant information needed for contrastive learning.

“Among those encoders which are sufficient, the minimal ones only extract relevant information of the contrastive task and throw away other irrelevant information.”◌ not checked against the paper’s text as it now stands

The paper demonstrates that embeddings can be improved through data augmentation that reduces mutual information between views to reach the optimal information level.

“We additionally demonstrate that the InfoMin principle can be practically applied by simply seeking stronger data augmentation to further reduce mutual information toward the sweet spot.”

This paper introduced a method that maps image features to prototype vectors to create embeddings for cluster assignments instead of directly comparing feature embeddings.

“we avoid comparing every pair of images by mapping the image features to a set of trainable prototype vectors.”◌ not checked against the paper’s text as it now stands

This paper computed codes that serve as embeddings by matching features to prototype vectors using a mapping process.

“we compute their codes 𝐪_t and 𝐪_s by matching these features to a set of K prototypes {𝐜_1,…, 𝐜_K}.”◌ not checked against the paper’s text as it now stands

This paper projected features to the unit sphere to create normalized embeddings before mapping them to prototype vectors.

“The feature is then projected to the unit sphere,, 𝐳_nt = f_θ(𝐱_nt) / f_θ(𝐱_nt)_2.”◌ not checked against the paper’s text as it now stands

This paper learned embeddings online where the prototype vectors and image encoder parameters are jointly optimized through backpropagation.

“This loss function is jointly minimized with respect to the prototypes 𝐂 and the parameters θ of the image encoder f_θ used to produce the features (𝐳_nt)_n,t.”