Unsupervised Learning: Exploring the Power of Data Without Labels

Chapter 2: Dimensionality Reduction

Introduction to Dimensionality Reduction

Dimensionality reduction is a crucial technique in machine learning and data analysis, particularly when working with large, high-dimensional datasets. It aims to reduce the number of input features or variables, while maintaining the essential information of the data. By lowering the dimensionality, we make the data easier to analyze and visualize, and it can also speed up the performance of machine learning algorithms.

In real-world applications, datasets often contain hundreds or even thousands of features, many of which may be irrelevant, redundant, or correlated. This phenomenon, known as the curse of dimensionality, can degrade the performance of machine learning models. Dimensionality reduction helps mitigate this issue by simplifying the dataset without losing too much valuable information.

In this chapter, we will explore popular dimensionality reduction techniques, including Principal Component Analysis (PCA), t-SNE (t-distributed Stochastic Neighbor Embedding), and Autoencoders. We will walk through the theoretical foundations of each technique, followed by Python code implementations and visualizations.

2.1 Principal Component Analysis (PCA)

What is PCA?

Principal Component Analysis (PCA) is one of the most widely used dimensionality reduction techniques. It works by transforming the original features of the data into a smaller set of new features, called principal components (PCs). These components capture the directions of maximum variance in the data, thus retaining most of the essential information while reducing the number of features.

The main idea behind PCA is to project the data onto a new set of axes, so that the first principal component captures the greatest variance, the second component captures the second greatest variance, and so on. The transformation is done in such a way that the new components are orthogonal (uncorrelated) to each other.

Steps in PCA:

1. Standardize the data: Since PCA is sensitive to the scale of the data, we first standardize the dataset to have zero mean and unit variance.
2. Compute the covariance matrix: The covariance matrix helps us understand the relationships between different variables.
3. Calculate eigenvalues and eigenvectors: Eigenvectors represent the direction of the principal components, and eigenvalues indicate the importance (variance) of each principal component.
4. Sort eigenvectors by eigenvalues: The eigenvector with the highest eigenvalue is the first principal component, and so on.
5. Project the data: Finally, we project the original data onto the new axes defined by the principal components.

Code Sample (PCA in Python)

import numpy as np

from sklearn.decomposition import PCA

import matplotlib.pyplot as plt

from sklearn.preprocessing import StandardScaler

# Generate synthetic data

X = np.random.rand(100, 5) * 10

# Standardize the data

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

# Apply PCA

pca = PCA(n_components=2)

X_pca = pca.fit_transform(X_scaled)

# Visualize the results

plt.scatter(X_pca[:, 0], X_pca[:, 1], color='blue')

plt.title("PCA Projection")

plt.xlabel("Principal Component 1")

plt.ylabel("Principal Component 2")

plt.show()

# Explained variance ratio

print("Explained variance ratio:", pca.explained_variance_ratio_)

Output: A scatter plot showing the projection of the data on the first two principal components.

Explained Variance Ratio:

• The explained_variance_ratio_ shows how much variance is explained by each principal component. For example, if the first component explains 90% of the variance and the second component explains 5%, then together, the first two components explain 95% of the total variance.

2.2 t-SNE (t-distributed Stochastic Neighbor Embedding)

What is t-SNE?

t-SNE is a non-linear dimensionality reduction technique that is particularly effective for visualizing high-dimensional data in a 2D or 3D space. Unlike PCA, which focuses on preserving variance, t-SNE focuses on preserving the local structure of the data. It minimizes the divergence between probability distributions representing pairwise similarities in the high-dimensional space and the low-dimensional space.

t-SNE is particularly useful when dealing with datasets where the relationships between points are complex and cannot be captured by linear methods like PCA.

Steps in t-SNE:

1. Compute pairwise similarities: For each pair of data points, compute the probability that they are neighbors in the high-dimensional space.
2. Initialize the low-dimensional representation: Randomly initialize the data points in the lower-dimensional space (e.g., 2D or 3D).
3. Minimize the Kullback-Leibler divergence: Using gradient descent, adjust the low-dimensional representations to match the pairwise similarities in the high-dimensional space.

Code Sample (t-SNE in Python)

from sklearn.manifold import TSNE

import matplotlib.pyplot as plt

# Generate synthetic data

X = np.random.rand(100, 5) * 10

# Apply t-SNE

tsne = TSNE(n_components=2, random_state=0)

X_tsne = tsne.fit_transform(X)

# Visualize the results

plt.scatter(X_tsne[:, 0], X_tsne[:, 1], color='green')

plt.title("t-SNE Projection")

plt.xlabel("t-SNE Component 1")

plt.ylabel("t-SNE Component 2")

plt.show()

Output: A scatter plot showing the data projected onto two dimensions using t-SNE.

Pros of t-SNE:

• Ideal for visualizing high-dimensional data.
• Captures local structures and patterns that linear methods like PCA may miss.

Cons of t-SNE:

• Computationally expensive for large datasets.
• Not suitable for preserving global structures or for use in downstream machine learning tasks.

2.3 Autoencoders for Dimensionality Reduction

What are Autoencoders?

Autoencoders are a type of neural network used for unsupervised learning, specifically for dimensionality reduction and feature extraction. They consist of two parts:

• Encoder: Compresses the input data into a lower-dimensional representation (latent space).
• Decoder: Reconstructs the original input from the lower-dimensional representation.

The network is trained to minimize the reconstruction error, meaning that the decoder’s output is as close as possible to the input. By learning to represent the data in a compact form, autoencoders can effectively reduce the dimensionality of the data.

Steps in Autoencoder-based Dimensionality Reduction:

1. Encoder: The input data is passed through an encoder (a neural network) that compresses it into a lower-dimensional space.
2. Latent Representation: The compressed representation (latent space) captures the most important features of the data.
3. Decoder: The decoder reconstructs the original input data from the latent representation.
4. Loss Function: The network is trained using a loss function (e.g., mean squared error) that minimizes the difference between the input and the reconstructed output.

Code Sample (Autoencoder in Python)

from sklearn.preprocessing import MinMaxScaler

from keras.models import Sequential

from keras.layers import Dense

import numpy as np

# Generate synthetic data

X = np.random.rand(100, 5) * 10

# Normalize the data

scaler = MinMaxScaler()

X_scaled = scaler.fit_transform(X)

# Define the Autoencoder model

model = Sequential()

model.add(Dense(3, activation='relu', input_dim=X_scaled.shape[1]))  # Encoder

model.add(Dense(X_scaled.shape[1], activation='sigmoid'))  # Decoder

model.compile(optimizer='adam', loss='mean_squared_error')

# Train the Autoencoder

model.fit(X_scaled, X_scaled, epochs=100, batch_size=10, verbose=0)

# Get the compressed representation (latent space)

encoder = Sequential(model.layers[:1])  # Only take the encoder part

latent_space = encoder.predict(X_scaled)

# Visualize the latent space

plt.scatter(latent_space[:, 0], latent_space[:, 1], color='red')

plt.title("Autoencoder Latent Space")

plt.show()

Output: A scatter plot showing the compressed representation of the data in the latent space.

2.4 Comparison of Dimensionality Reduction Techniques

Below is a comparison table summarizing the key differences between PCA, t-SNE, and Autoencoders:

Conclusion

Dimensionality reduction techniques such as PCA, t-SNE, and Autoencoders are essential tools in machine learning and data analysis. By reducing the number of features while maintaining the data’s inherent structure, these techniques make it easier to analyze complex datasets, improve the efficiency of machine learning algorithms, and visualize high-dimensional data.