← Information Geometry

Play: Step through the Expectation-Maximization (EM) algorithm fitting a 2-component Gaussian mixture. The algorithm starts with a bad guess. Click "Step" to watch how alternating between assigning data to clusters (E-step) and updating the cluster shapes (M-step) monotonically increases the log-likelihood until it converges.

The Core Concept

The EM algorithm fits models with hidden variables (like Gaussian mixtures, HMMs, and topic models) by alternating between estimating the hidden variables (E-step) and updating the parameters (M-step). The textbook proof that EM always increases the log-likelihood relies on Jensen's inequality.

Information Geometry reveals what is really happening: EM is just tracing a zig-zag path of alternating projections between two flat submanifolds. By our Pythagorean theorem (Part 8), this geometric path guarantees monotone convergence in one line.

• Two Flat Submanifolds: The model family (all possible mixture models) is an e-flat submanifold. The data submanifold (all joint distributions that match the empirical data) is an m-flat submanifold.
• E-step is an m-projection: The E-step finds the point on the data submanifold closest in KL divergence to the current model. This "dropping an m-perpendicular" is exactly computing the posterior over latent variables.
• M-step is an e-projection: The M-step finds the point on the model submanifold closest in KL divergence to the current posterior. This is the expected maximum-likelihood update.

Concept Summary

• Geometry of EM: EM is not just an algebraic trick; it is geometric alternating projection. The E-step projects onto the data manifold, and the M-step projects back onto the model manifold.
• Pythagorean Proof: Because one manifold is e-flat and the other is m-flat, each projection reduces the KL divergence. By the generalized Pythagorean theorem, this means the total log-likelihood must strictly increase.
• A Universal Pattern: This geometric view of alternating projections isn't limited to EM. It unifies variational inference, mean-field methods, belief propagation, and Gibbs sampling under a single mathematical umbrella.

What EM is not

The picture covers exponential-family mixtures with complete-data likelihoods in exponential form: Gaussian, Bernoulli, Poisson, multinomial mixtures (LDA's core step), HMMs with exponential-family emissions. Stochastic EM, Variational EM, and Generalised EM relax different parts and get weaker guarantees.

The model family must be dually flat — automatic for exponential families. Part 9's Bregman framework generalises this to "flat with respect to the chosen Bregman generator," giving Bregman EM variants used in signal processing.

Variational inference, belief propagation, mean-field, Gibbs sampling, and convex clustering are also (single or iterated) projection schemes.

What comes next

Acts I and II built the machinery; Act III so far applied it to clustering and EM.

The remaining three parts pivot to the Moonshine ecosystem. Part 13 frames the connection. Parts 14 and 15 sketch two: the Fisher metric on CFT moduli spaces as the Weil-Petersson metric, and the -function of Monstrous Moonshine as an entropic potential on a Koszul-Souriau thermodynamic manifold. Claims will be labelled established, plausible, or speculative as we go.