← Information Geometry

Play: Toggle between forward and reverse KL divergence from a fixed reference Gaussian. Hover over the heatmap to see how the landscape drastically changes based on the divergence direction. Notice how forward KL penalizes narrow distributions heavily compared to reverse KL.

Explore: Observe the effects of KL asymmetry when fitting a bimodal target with a single Gaussian. Forward KL results in a broad, mode-covering fit, while reverse KL seeks out a single sharp mode.

The Core Concept

Kullback-Leibler divergence (KL) is a global measure of how one probability distribution diverges from a second, expected distribution :

For two Gaussians, the closed form is:

KL divergence is strictly non-negative and zero if and only if . However, it is explicitly asymmetric () and does not satisfy the triangle inequality, meaning it is not a true geometric metric.

The direction of KL matters significantly in practice. Minimizing forward KL encourages "mode-covering" behavior (often leading to blurry generation, common in VAEs), while minimizing reverse KL encourages "mode-seeking" or mode-collapse behavior (common in GANs).

Concept Summary

• KL Divergence: Measures the excess description length required when using a suboptimal code, or the statistical surprise.
• Asymmetry is a Feature: The lack of symmetry encodes the real-world difference between approximating a sharp true distribution with a blurry model vs. a blurry true distribution with a sharp model.
• Divergence vs. Metric: Because it lacks symmetry and the triangle inequality, KL is a divergence, not a metric. However, its local (infinitesimal) behavior recovers the Fisher metric.