Representation Learning

Representation Learning#

Representation Learning involves learning an effective way to represent data, typically by mapping it into a feature space where meaningful patterns and structures are preserved. This approach aims to automatically discover useful features from raw data, enhancing the performance of various machine learning tasks. Techniques like contrastive loss and triplet loss are often used in this context to ensure that similar data points are close together and dissimilar ones are far apart in the learned representation space.

So, Representation learning involves:

Encoding static observations (e.g., images) into a latent space.
Evolving weights that align with the ground truth in the observation space.

RL_Manfold

Representation learning can be divided into:

Supervised representation learning: learning representations on task A using annotated data and used to solve task B

Unsupervised representation learning: learning representations on a task in an unsupervised way (label-free data)

Whats Downstream tasks#

Downstream tasks refer to specific applications or problems that benefit from the features or representations learned by a model. In the context of machine learning, once a model learns to represent data (e.g., identifying patterns or features in images, text, etc.), these representations can be used to improve performance on related tasks, known as downstream tasks.

For example:

In Natural Language Processing (NLP), if a model learns to understand the structure and meaning of sentences, it can then be applied to downstream tasks like sentiment analysis, translation, or summarization.
In Computer Vision (CV), if a model learns to recognize basic features in images (like edges, shapes, etc.), these features can be used for downstream tasks like object detection, image classification, or facial recognition.

Latent $Z$ -space#

In machine learning, the goal is typically to predict a target value $y \in Y$ given input data $x \in X$ . To achieve this, we aim to learn a function $f : X \to Y$ that maps the input data $x$ to the predicted output $y$ .

In representation learning, the approach shifts from directly learning a function $f : X \to Y$ to first learning a representation $g : X \to Z$ . Then, a classifier or predictor $f : Z \to Y$ is learned on top of this representation.

Goal: The aim is to develop representations $g$ for each domain, enabling the learning of simple predictors for various tasks. This makes the process faster and more efficient.

Advantage: Learning the representation $g$ often requires little or different forms of supervision, allowing us to leverage more data, including unlabeled or weakly labeled data.

Learning $g$ #

We start by learning a mapping $g : X \to Z$ , where $X$ is the original input space, and $Z$ is the learned representation space.
The goal is to transform the input data $x \in X$ into a new representation $z = g (x) \in Z$ that captures the essential features needed for various tasks.

Learning Simple Predictors $f$ #

Once we have the representation $z$ , we learn a function $f : Z \to Y$ for a specific task. For instance, if the task is classification, $f$ could be a linear model or a simple neural network that maps $z$ to the output $y$ .

Mathematical Explanation#

Assume we have multiple tasks $T_{1}, T_{2}, \dots, T_{n}$ , each requiring us to predict different outputs $y_{1}, y_{2}, \dots, y_{n}$ .
Instead of learning separate functions $f_{1} : X \to Y_{1}, f_{2} : X \to Y_{2}, \dots, f_{n} : X \to Y_{n}$ for each task directly from the original input space $X$ , we first learn a common representation $g : X \to Z$ regardless of Task.
Then, for each task $T_{i}$ , we learn a simpler function $f_{i} : Z \to Y_{i}$ on top of the shared representation $z = g (x)$ .

or

For task T_{i} : y_{i} = f_{i} (g (x))

RL_MTL

Latent with Deep#

In deep learning, a model $f$ is typically composed of a sequence of simpler layers $f_{1}, f_{2}, \dots, f_{L}$ . This composition can be represented as:

f := f_{L} \circ f_{L - 1} \circ \dots \circ f_{2} \circ f_{1}

Where:

$f_{1}, f_{2}, \dots, f_{L}$ are individual layers or transformations.
$f$ is the overall model that maps the input $x$ to the output $y$ .

Learning the Representation $g$ #

When the model $f$ is trained to solve a specific task (e.g., object detection in images), it inherently learns not only the final task output but also an intermediate representation:

g := f_{L - 1} \circ \dots \circ f_{2} \circ f_{1}

Here, $g$ is a function that maps the input $x$ to an intermediate representation $z$ , often called a “feature representation.” This representation $z = g (x)$ is optimized to be particularly effective for the task at hand.

Latent_Deep

Cost function for Representation Learning#

In representation learning, the cost function, or loss function, is crucial for guiding the learning process. The choice of cost function depends on the specific goals of the learning task.

Triplet Loss#

Objective: To ensure that the distance between an anchor and a positive sample (from the same class) is smaller than the distance between the anchor and a negative sample (from a different class) by a margin.
Cost Function: $ $L_{triplet} = max (0, D_{anchor, positive}^{2} - D_{anchor, negative}^{2} + α)$ $w h e r e$ D_{\text{anchor, positive}} $a n d$ D_{\text{anchor, negative}} $a r e t h e d i s t a n c e s b e t w e e n t h e a n c h o r a n d p o s i t i v e, a n d t h e a n c h o r a n d n e g a t i v e s a m p l e s, r e s p e c t i v e l y, a n d$ \alpha $ is a margin.

NCA (Neighborhood Component Analysis) Loss#

Objective: To learn a distance metric that improves nearest neighbor classification accuracy.
Cost Function: $ $L_{NCA} = - \sum_{i} \log (\frac{\exp (- ‖ L x_{i} - L x_{j} ‖^{2} / τ)}{\sum_{k \neq i} \exp (- ‖ L x_{i} - L x_{k} ‖^{2} / τ)})$ $w h e r e$ L $i s t h e l i n e a r t r a n s f o r m a t i o n m a t r i x,$ \tau $ is a temperature parameter, and the sum is over all training instances.

Combining simple concepts to derive complex structures#

Deep neural networks are fundamentally representation learning models, particularly in supervised settings. These networks typically consist of two main components:

Encoder: transforms the input data into a low-level representation subspace.
Linear classifier: to separate classes

These learned representations are dense, compact, and transferable to similar data modalities. From edge and corner transfer to label of image.

Example:

Train a Convolutional Neural Network (CNN), then transfer the learned knowledge to another task, often one with insufficient labels to train a deep network from scratch. This approach, known as transfer learning, has been successfully applied across various domains and is widely used in commercial applications.

Example for Understanding the Term “From Scratch”

Building a Model from Scratch: Instead of relying on libraries like TensorFlow or PyTorch, you would write your own code to define the model architecture, initialize parameters, and implement training algorithms.

Insufficient label in pixel level#

For tasks such as object classification, detection, and segmentation, the number of labeled examples diminishes substantially as we move from whole-image labels to pixel-level annotations.

In healthcare applications like breast cancer metastasis detection, annotating large datasets is both expensive and time-consuming, often requiring hours of work from specialized expert pathologists with extensive training

Soloution of Problem#

All of these examples highlight the importance of learning generalizable representations from non-annotated data. Many research areas, including

semi-supervised
self-supervised learning try to learn representations that can be transferred to new tasks using just a few or not using annotated examples at all.

Deep Unsupervised Representation Learning

Key Focus:
Deep unsupervised representation learning is centered on extracting useful features from unlabeled data. The main goal is to improve downstream tasks while reducing dependence on human annotations.

Recent Developments:

Importance of Unsupervised Learning:
- Grown due to advances in NLP, specifically with models like BERT and GPT (Miniproject). These models showcase the power of label-free training.
- Prototypical Contrastive Learning of Unsupervised Representations
- Self-supervised learning involves devising a predictive task (pretext task)

pretext task A pretext task in self-supervised learning is a task designed to help the model learn useful features from unlabeled data.

it doesn’t require manual annotations. For example, a common pretext task in image processing might involve predicting the rotation angle of an image. Although the model doesn’t have explicit labels for what each rotation represents, learning to predict this rotation angle helps the model understand important features of the image, such as shapes and textures. These learned features can then be applied to more traditional tasks like image classification or object detection.

Roration_UnsupervisedLearning_ExtractionImportantFeature

Another examples:

Text processing (BERT)

Image Inpainting: unsupervised Learning Self-supervised pretext tasks consist of taking out some parts of the data and challenging the network to predict that missing part.

Inpainting1

Labeling data can indeed be expensive#

Complexity of Labeling Task

Simple Labels: Tasks like labeling images with straightforward categories (e.g., “cat” or “dog”) may be relatively inexpensive if you can use crowd-sourcing platforms.
Complex Labels: Tasks requiring detailed annotation or expert knowledge (e.g., medical imaging, legal documents) can be much more costly due to the need for specialized skills and time.

Volume of Data

Large Datasets: For large datasets, the cost can add up quickly. For example, labeling thousands or millions of images or text samples can require significant resources.

Quality of Labels

Consistency and Accuracy: Ensuring high-quality, accurate labels often requires multiple rounds of verification and validation, which adds to the cost.
Training and Quality Control: Training annotators and implementing quality control measures can further increase expenses.

Domain Expertise

Specialized Knowledge: Data that requires domain expertise (e.g., medical images, legal documents) demands annotators with specialized training, which can be more expensive than general labeling tasks.

Time Constraints

Speed of Labeling: Fast turnaround times for large-scale labeling projects can lead to higher costs, as additional resources may be needed to meet deadlines.

Tools and Infrastructure

Annotation Tools: Developing or purchasing sophisticated annotation tools and infrastructure can also add to the overall cost.