← Information Geometry

Play: Drag the parameter point in either natural coordinates θ (left) or expectation coordinates η (right). Observe how the Legendre transform perfectly syncs the two representations. Notice that dragging a straight line in one space results in a curved path in the other.

The Core Concept

An exponential family has the canonical form:

Every exponential family is dually flat, meaning it comes with two distinct, flat coordinate systems linked by a Legendre transform:

1. Natural Parameters : The coefficients of the sufficient statistics in the exponent. A straight line in this space is an e-geodesic.
2. Expectation Parameters : The expected values of the sufficient statistics (e.g., mean and variance). A straight line in this space is an m-geodesic.

The gradient of the log-partition function (a strictly convex function) maps natural parameters to expectation parameters. Its Legendre dual (negative entropy) maps them back.

Concept Summary

• Ubiquity: Almost all named distributions (Gaussian, Bernoulli, Poisson, Categorical) are exponential families.
• Convex Potentials: The geometry of an exponential family is completely determined by a single strictly convex function (the log-partition function ). The Fisher metric is exactly its Hessian matrix.
• Dually Flat Coordinates: Moving in natural parameters holds the form of the distribution constant (e-flat). Moving in expectation parameters directly interpolates moments (m-flat). You can always switch to the coordinate system that makes your problem linear.

Gallery of exponential families

The same structure applies to every exponential family. Table 1 lists a handful of the most common ones with their sufficient statistics, natural parameters, log-partition functions, and the resulting expectation parameters. Each row is a self-contained Legendre pair, and in each the Fisher matrix is the Hessian of in the natural coordinates.

Family	T(x)	θ	ψ(θ)	η = E[T]
Bernoulli(p)
Poisson(λ)
Exponential(λ)
Gaussian(μ, σ²)
Categorical(p)

The categorical row is the exponential family behind softmax. Softmax converts natural to expectation parameters via , and cross-entropy is the KL divergence that respects the dually flat geometry. Natural gradient descent on a softmax (Part 10) is gradient descent in the flat expectation coordinates.

What "dually flat" buys you. The natural parameters flatten the e-connection; the expectation parameters flatten the m-connection. The hyperbolic curvature of the Poincaré half-plane is a property of Levi-Civita, not of the e- or m-connections.

KL divergence in the dually flat coordinates

One payoff we can already extract. KL divergence on an exponential family has a closed-form expression in terms of the two potentials:

This is the Bregman divergence of (Part 9 generalises). The Legendre dual:

Same number, different coordinates. The duality is what makes the Part 8 Pythagorean theorem possible.

What comes next

Part 8 proves the generalised Pythagorean theorem: if lie on a dually flat manifold with the e-geodesic from to Fisher-orthogonal to the m-geodesic from to , then . KL decomposes additively under orthogonal projection — the engine of maximum likelihood, EM, and variational inference.