← Information Geometry

Play: Drag points P (the data) and R (a model) or adjust the fixed-variance submanifold 𝓜 (σ = σ₀). Notice that the m-projection point Q always creates a Fisher right angle, guaranteeing that KL(P‖R) exactly equals KL(P‖Q) + KL(Q‖R), checked dynamically in the status bar.

The Core Concept

The generalised Pythagorean theorem states that if an e-geodesic from to meets an m-geodesic from to at a Fisher right angle at , then Kullback-Leibler divergence decomposes additively like squared Euclidean distance:

The classical theorem trades Euclidean distance for KL divergence, Euclidean lines for e- and m-geodesics, and Euclidean right angles for Fisher right angles. The identity holds globally on any dually flat manifold (like exponential families).

This theorem is the geometric engine behind Maximum Likelihood Estimation (MLE). When we fit a model to data, we are performing an m-projection of the empirical data distribution onto the model manifold. By the Pythagorean theorem, the total error KL(Data‖Model) decomposes precisely into the irreducible error KL(Data‖MLE) plus the parameter error KL(MLE‖Model).

Concept Summary

• KL as Squared Distance: When geometries align properly, KL divergence acts like squared Euclidean distance, meaning it decomposes additively without cross terms.
• Orthogonality: For the theorem to hold, the paths must meet at a "Fisher right angle", which in exponential families simply means a zero dot product between the natural parameter displacement and the expectation parameter displacement .
• Maximum Likelihood as Projection: MLE is an m-projection. It finds the point on the model manifold closest (in KL) to the data, dropping a perpendicular that guarantees the Pythagorean split of errors.

Other algorithms that are projections

Many algorithms turn out to be projections too. EM (Part 12) alternates e- and m-projection between two flat submanifolds; the Pythagorean theorem forces monotone decrease, which is the convergence proof. Variational inference is e-projection onto a simpler family. Belief propagation is iterated projection on a factor graph. Information bottleneck is a constrained projection.

What comes next

Part 9 generalises further: every strictly convex potential on an affine space induces a Bregman divergence with its own dually flat geometry and Pythagorean theorem. KL comes from the log-partition function as ; squared Euclidean distance from half the squared norm; Itakura-Saito and logistic from other convex choices.