High-dimensional data often varies along only a few directions. PCA finds the directions of greatest variance, so the data can be projected onto fewer dimensions without losing much.
Suppose you have to flatten 100-dimensional data down to 2 dimensions. The original axes are an arbitrary choice. If the data really varies along only a handful of directions, most of those axes carry noise, and the signal lives in the directions where the data spreads out most. Principal component analysis (PCA) finds those directions. Unlike the models in Parts 1 to 3, it has no target to match; it works from the data alone.
The idea is geometric. A cloud of points has a shape, stretched long in one direction and flat in another. PCA finds the axes of that shape: the longest is the first principal component, the next longest (perpendicular to it) is the second, and so on. When the data is stretched along a few directions, keeping only the axes with the most variance reduces the dimension without losing much.
Variance is usually taught as a single number, but for data in more than one dimension it depends on the direction you measure it in. Project the data onto an axis and measure the spread; rotate the axis and the spread changes. Figure 1 shows 50 simulated, correlated points in an elliptical cloud.
Figure 1. Simulated data. Drag the projection axis (or focus it and use the arrow keys) to rotate it through the cloud. The gray drop lines meet the axis at right angles, and the histogram of projected positions, below the cloud on narrow screens, updates as you drag.
When the axis lines up with the long direction of the ellipse, the histogram spreads wide and the variance peaks; perpendicular to it, the variance bottoms out. The direction of maximum variance is the first principal component. Variance is what we want to keep because a direction where all the points overlap says nothing about how they differ.
PCA automates the dragging. It centers the data, then finds the directions where it stretches most from the eigendecomposition of the covariance matrix. The covariance matrix measures how much each pair of data variables moves together. Its eigenvectors point along the directions of maximum and minimum variance, and its eigenvalues give the variance along each.
The eigenvector equation says: when we apply the covariance matrix to a principal direction, the direction does not rotate. It only scales, by an amount equal to the variance along it.
Figure 2. PCA on the simulated cloud from Figure 1, animated: compute the mean, center the data, then grow PC1 along the direction of maximum variance and PC2 perpendicular to it. The bars (beside the plot, or below it on narrow screens) show each eigenvalue as a share of the total variance.
The result is a rotated coordinate system aligned with the data's own axes of variation, in which the new coordinates are uncorrelated. In data like this, the first few carry most of the variance and the rest can be dropped.
To project to fewer dimensions, keep the first few components and drop the rest. Stack the kept eigenvectors as the columns of a matrix W. The projection of a data point is a matrix multiply, and the reconstruction error is the squared distance between the original and its round trip back.
Each dropped component loses some variance. Figure 3 shows a 3D dataset shaped like a flattened ellipsoid. Remove PC3, the thinnest direction, and the data collapses to a plane, losing little because it was already nearly flat that way. Remove PC2 as well and it collapses to a line, and now the reconstruction error is substantial.
Figure 3. A simulated 3D point cloud in isometric projection, with its three principal axes. Toggle components off to project the data into fewer dimensions: faint gray dots mark where the points were, and red lines join each point to its projection.
The reconstruction error, the sum of squared distances between the original points and their projections, equals the number of points times the total variance of the dropped components. Dropping only PC3 here costs 7.4, which is 100 points times PC3's variance of 0.074.
With many dimensions, a scree plot helps decide how many components to keep. It shows the variance explained by each component in order, with a line for the running total. The typical shape is a steep drop followed by a leveling off. Components before the bend, the "elbow", tend to carry structure and those after it mostly noise, so the elbow is a common place to cut.
Figure 4. A scree plot of the 8 principal components of 120 simulated points in 8 dimensions: bars for each component's share of the variance, the line for the cumulative total. The slider sets how many components to keep. Below, each point is rebuilt from that many components and drawn on the first two of its eight coordinates: gray dots are the originals, blue dots the rebuilt points, and red lines the part of each point's error that falls in those two coordinates. The error readout sums the full 8D error, which is always 120 times the variance of the dropped components. The bars fall steeply from PC2 to PC3 and then level off, so the elbow is at 2 to 3 components.
In practice the elbow is not always sharp. For noisy data the curve slopes gradually; for data with clear low-dimensional structure it drops sharply. Some practitioners instead keep enough components to explain 90% or 95% of the variance; others use cross-validation to see whether more components improve predictions. In Figure 4, the 90% rule keeps 3 components and the 95% rule keeps 5.
PCA finds directions of high variance. It is not designed to find clusters or nonlinear patterns, though it separates clusters when they differ along high-variance directions. A ring of points has equal variance in every direction, so PCA has no preferred axis, and projecting onto any line folds opposite sides of the ring onto each other.
PCA is used to compress, visualize and denoise high-dimensional data. Gene expression datasets with about 20,000 genes are often reduced to a few dozen components before clustering cells into types; the dimensionality reduction essay in the bioinformatics series runs PCA next to t-SNE on synthetic data, including a case where PCA's variance ranking picks the wrong directions. The principal components of a set of face photos look like ghostly faces, the "eigenfaces" used in early face recognition. Financial returns across hundreds of stocks compress to a few market factors.