Understanding PCA Discrepancies: Sklearn Vs. Scratch

10 min read 11-15- 2024

Understanding PCA Discrepancies: Sklearn Vs. Scratch

Understanding Principal Component Analysis (PCA) can be a challenging yet rewarding endeavor. It is an essential technique in data science and machine learning that helps in reducing the dimensionality of large datasets while preserving as much variance as possible. However, discrepancies can arise when implementing PCA using different tools and libraries, particularly when comparing Scikit-learn (Sklearn) with a custom scratch implementation. This article aims to clarify these discrepancies and provide a comprehensive understanding of PCA through both methods.

What is PCA?

Principal Component Analysis (PCA) is a statistical procedure that uses orthogonal transformation to convert correlated variables into a set of uncorrelated variables called principal components. These components capture the maximum variance from the original dataset and facilitate dimensionality reduction, making it easier to visualize and analyze data without losing essential information.

The Importance of PCA

Data Visualization: 🎨 PCA allows data scientists to visualize high-dimensional data in 2D or 3D spaces.
Noise Reduction: ✨ By focusing on principal components that capture the most variance, PCA can help filter out noise from the data.
Feature Reduction: 📉 PCA enables the reduction of the number of features, leading to faster model training and potentially improved performance.

Implementing PCA with Sklearn

Sklearn, or Scikit-learn, is one of the most widely used libraries for machine learning in Python. It offers a straightforward implementation of PCA, making it accessible for practitioners. Let’s break down how to implement PCA using Sklearn.

Step-by-Step PCA with Sklearn

Import Required Libraries:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris

Load the Data:

The Iris dataset is a classic dataset used for showcasing PCA.
```
iris = load_iris()
X = iris.data
y = iris.target
```
Standardize the Data:

PCA is sensitive to the scales of the features; therefore, standardization is crucial.
```
from sklearn.preprocessing import StandardScaler
X_std = StandardScaler().fit_transform(X)
```
Apply PCA:

We can now apply PCA and reduce the dataset to two principal components.
```
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_std)
```

Visualize the Results:

Finally, we can visualize the PCA-transformed data.

plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
plt.title('PCA of Iris Dataset')
plt.show()

Important Notes on Sklearn Implementation

Efficiency: Sklearn’s PCA implementation is optimized for performance, making it suitable for large datasets.
Parameter Options: Users can choose the number of components, variance threshold, and more, providing flexibility.

Implementing PCA from Scratch

While Sklearn offers an efficient implementation, understanding the underlying mechanics of PCA is crucial. Here’s how to implement PCA from scratch.

Step-by-Step PCA from Scratch

Import Required Libraries:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris

Load the Data:

Similar to our Sklearn example, we will load the Iris dataset.
```
iris = load_iris()
X = iris.data
y = iris.target
```

Standardize the Data:

Standardization is also necessary when implementing PCA from scratch.

def standardize(X):
    return (X - np.mean(X, axis=0)) / np.std(X, axis=0)

X_std = standardize(X)

Calculate the Covariance Matrix:

The covariance matrix helps us understand how variables interact with each other.
```
covariance_matrix = np.cov(X_std, rowvar=False)
```
Calculate the Eigenvalues and Eigenvectors:

Eigenvalues and eigenvectors help in determining the principal components.
```
eigenvalues, eigenvectors = np.linalg.eigh(covariance_matrix)
```

Sort Eigenvalues and Eigenvectors:

We need to sort them in descending order.

sorted_indices = np.argsort(eigenvalues)[::-1]
sorted_eigenvalues = eigenvalues[sorted_indices]
sorted_eigenvectors = eigenvectors[:, sorted_indices]

Select the Top Principal Components:

We will select the top two principal components.
```
n_components = 2
selected_eigenvectors = sorted_eigenvectors[:, :n_components]
```
Transform the Data:

Finally, we can transform the standardized data into the PCA space.
```
X_pca = X_std.dot(selected_eigenvectors)
```

Visualize the Results:

As with our Sklearn implementation, let’s visualize the results.

plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
plt.title('PCA of Iris Dataset from Scratch')
plt.show()

Important Notes on Scratch Implementation

Flexibility: Implementing PCA from scratch gives you the flexibility to modify and experiment with various techniques.
Understanding: You gain a deeper understanding of how PCA works, which is beneficial when troubleshooting or optimizing your model.

Comparing PCA with Sklearn vs. Scratch

Aspect	Sklearn Implementation	Scratch Implementation
Ease of Use	Easy with minimal code	Requires more code
Performance	Optimized for performance	May be slower for large datasets
Flexibility	Limited to Sklearn’s parameter options	High flexibility for modifications
Learning Opportunity	Less opportunity to learn	Great for understanding the algorithm
Dependencies	Requires Sklearn	Only requires NumPy

Understanding Discrepancies

Discrepancies between PCA implementations can arise due to multiple factors, such as:

Floating Point Precision: Different libraries may handle floating point operations differently, leading to slight variations in results.
Standardization Differences: If standardization is not applied consistently, the resulting principal components may vary significantly.
Numerical Stability: Libraries like Sklearn are designed to be numerically stable, while scratch implementations might face issues with large matrices.

Practical Considerations

When implementing PCA, whether using Sklearn or building from scratch, consider the following:

Data Preprocessing: Ensure that data is standardized before applying PCA.
Choosing Components: Selecting the right number of components is crucial and can be informed through explained variance ratios.
Use Cases: Understand the context in which PCA is appropriate—especially in applications like image processing or genetics, where high dimensionality is common.

Conclusion

Understanding PCA and its discrepancies between Sklearn and a scratch implementation equips data scientists with the tools to make informed decisions when analyzing and interpreting data. The choice between using a library or implementing from scratch depends on the specific use case, performance needs, and learning goals. By diving into the mechanics of PCA, practitioners can better harness its power for real-world applications, making sense of complex datasets with ease.

Remember, whether you choose Sklearn or opt to build your own implementation, mastering PCA is a vital step in your data science journey! 🚀