Part 5 of 15. The infinitesimal and global pictures are the same picture, up to a Taylor expansion. The asymmetry of KL, the bruises of the triangle inequality, and every other rough edge disappear when you zoom in, leaving behind exactly the Fisher quadratic form.
Play: Adjust the `h_max` slider to see how forward KL, reverse KL, and the Fisher quadratic form behave as you move away from a base Gaussian. At small distances, all three perfectly overlap. At larger distances, the asymmetry of KL emerges.
The Fisher metric is infinitesimal and symmetric. Kullback-Leibler (KL) divergence is global and asymmetric. However, a single Taylor expansion proves they are the exact same object at small scales.
Local equivalence: For any smooth parametric family at base point , the KL divergence from to is, to second order in the displacement , exactly half the Fisher quadratic form:
Because the first-order term vanishes, the asymmetry of KL vanishes at second order. Expanding in either direction gives the same quadratic coefficient. Asymmetry is strictly a large-distance phenomenon (a third-order effect).