Part 10 of 15. Vanilla SGD is the right way to descend a loss landscape only if the parameter space is Euclidean. On a statistical manifold it is subtly wrong, and the fix is to pre-multiply the gradient by the inverse Fisher matrix. The resulting algorithm is parameterisation-invariant, second-order-aware, and the reason every variational-inference and reinforcement-learning paper since 2000 has the phrase "natural gradient" in it.
Play: Compare Vanilla SGD (amber) and Natural Gradient Descent (blue) navigating the same Gaussian loss landscape. Click "Run 100" or step manually. Notice how SGD zigzags wildly due to the distorted geometry, while NGD takes a smooth, direct path to the target.
Vanilla Gradient Descent assumes you are moving through a flat, Euclidean space. It defines the steepest direction using Euclidean distance. But when your parameters define probability distributions, Euclidean distance is meaningless. A step of in means something entirely different when is 0.1 versus when is 10.
Natural Gradient Descent (NGD) fixes this by measuring the steepest direction using Kullback-Leibler (KL) divergence instead of Euclidean distance. Operationally, this means pre-multiplying the standard gradient by the inverse Fisher Information Matrix :
For a Gaussian, the inverse Fisher matrix scales the gradient by , perfectly cancelling out the exploding gradients that occur when variance is small.
Act II is complete: dually flat geometry on every exponential family, two privileged connections, a Pythagorean theorem, the Bregman framework, and a gradient algorithm that respects the manifold.
Act III turns to applications. Parts 11–12 show Bregman-Voronoi diagrams (clustering) and EM as alternating e- and m-projection. Parts 13–15 pivot to the frontier: the Fisher metric's isomorphism to the Weil-Petersson metric on moduli spaces of CFTs, and the Koszul-Souriau reading of the j-function of Monstrous Moonshine as an entropic potential.