From Information Theory to Information Geometry

01

The bridge: distributions become points

Information theory begins with questions like: How surprising was this outcome? and How many bits does this message require? Information geometry asks a new question: What kind of space do all possible models form?

The move is simpler than it sounds. A Bernoulli distribution is completely specified by one number p, so every possible coin lives somewhere on the interval 0 < p < 1. A three-outcome categorical distribution needs three probabilities summing to one, so every such model lives inside a triangle. These parameter spaces are our first statistical manifolds.

Ordinary geometryA pointa location

↔

Information geometryA pointan entire probability distribution

The central idea

Two distributions count as “nearby” when observations have difficulty distinguishing them—not merely when their parameter values look numerically close. Fisher information converts that operational notion of distinguishability into geometry.

Here is the whole bridge before we walk it slowly. Each step answers a question created by the previous one; none of the geometric language is being added merely for decoration.

1

Start with prediction

Reality is P; your model is Q

Reality draws an outcome $x$ according to $P(x)$ . Your model does not know that draw in advance; it assigns its own probability $Q(x)$ . The less probability Q assigned to what actually happened, the more surprised Q is—and the longer the ideal codeword must be.

s_Q(x)=-\log_2 Q(x)

Why this forces Step 2: One outcome only gives one surprise. To judge the model itself, we must average Q's surprise over all outcomes that P actually produces.

2

Average the prediction error

Cross-entropy becomes KL divergence

Averaging Q's code lengths over data from P gives the cross-entropy $H(P;Q)$ . But some bits are unavoidable even with a perfect model: that irreducible cost is $H(P)$ . Subtract it, and what remains is exactly the penalty for using Q when reality is P.

H(P;Q)=\mathbb E_{x\sim P}[-\log_2 Q(x)]

D_{\mathrm{KL}}(P\|Q)=H(P;Q)-H(P)

Why this forces Step 3: KL compares two separated models, but geometry begins locally. We therefore ask what KL looks like when Q is only an infinitesimal change away from P.

3

Zoom in around one model

The local shape of KL is Fisher information

Begin with a parameterized family $\mathcal M=\{p_\vartheta:\vartheta\in\Theta\}$ —for example, every Bernoulli distribution obtained by changing its probability parameter. Fix one member as the current distribution $P=p_\theta$ . Changing its parameters by $\Delta\theta$ selects another member, $Q=p_{\theta+\Delta\theta}$ . We are asking how the probabilities assigned to all possible outcomes change as a function of that parameter displacement. Call the resulting KL divergence $F(\Delta\theta)=D_{\mathrm{KL}}(p_\theta\|p_{\theta+\Delta\theta})$ .

If the model has several parameters, $\nabla F(0)$ is the gradient: the column vector containing every first partial derivative of $F$ , evaluated at $\Delta\theta=0$ . The superscript ${\mathsf T}$ means transpose, which turns that column into a row. Multiplying it by the parameter-change column $\Delta\theta$ forms the dot product $\nabla F(0)^{\mathsf T}\Delta\theta$ : one number giving the first-order change in $F$ along that particular direction. The transpose has no additional calculus meaning; it is needed only because multidimensional gradients and parameter changes are conventionally written as column vectors. We could equivalently write the dot product as $\nabla F(0)\mathbin{\cdot}\Delta\theta$ . For a one-parameter model, it reduces to the familiar $F'(0)\Delta\theta$ .

When $\Delta\theta=0$ , the distributions match, so $F$ reaches its minimum value of zero. KL's linear Taylor term $\nabla F(0)^{\mathsf T}\Delta\theta$ therefore vanishes. The first term that can survive is the quadratic term $\tfrac12\Delta\theta^{\mathsf T}I(\theta)\Delta\theta$ , where $I(\theta)$ is the Fisher information matrix.

F(\Delta\theta)=\underbrace{F(0)}_{\text{constant term }=0}+\underbrace{\nabla F(0)^{\mathsf T}\Delta\theta}_{\text{linear term }=0}+\underbrace{\frac12\Delta\theta^{\mathsf T}I(\theta)\Delta\theta}_{\text{quadratic term}}+O(\|\Delta\theta\|^3)

D_{\mathrm{KL}}^{(\mathrm{nats})}(p_\theta\|p_{\theta+\Delta\theta})=\frac12\Delta\theta^{\mathsf T}I(\theta)\Delta\theta+O(\|\Delta\theta\|^3)

The two symbols in that equation play different roles. The KL divergence is one scalar assigned to the pair $(p_\theta,p_{\theta+\Delta\theta})$ . The Fisher information $I(\theta)$ belongs to the starting distribution $p_\theta$ and describes the local curvature of KL in every parameter direction. It is one number for a one-parameter family such as Bernoulli, and a matrix when $\theta$ contains several parameters.

One-dimensional cross-section through model space

The local picture: the bottom of a KL bowl

Scroll horizontally to compare the full curves →

Hold

p_\theta

fixed and treat

F(\Delta\theta)

as the height above each possible parameter displacement

\Delta\theta

. At the center,

\Delta\theta=0

, the models match and KL is zero. Because KL cannot be negative, this is the bottom of the bowl, with no first-order slope. The first visible local shape is therefore curvature:

I(\theta)

tells us whether KL rises gently or steeply in each direction. A steeper direction means that data can distinguish a small model change more easily. The solid curve is exact; the dashed quadratic curve is the close-up supplied by Fisher information.

Aside

What Fisher information was built to answer

Fisher information began as a tool of statistical estimation, before its later geometric interpretation. The geometry did not invent a new quantity; it recognized that the same object used to quantify parameter precision also supplies a natural local metric on a family of probability distributions.

Suppose an observable random variable $Y$ is generated from a known family $p(y\mid\theta)$ , but the parameter value $\theta$ is unknown. The estimation problem is to infer that hidden value from measurements $y_1,\ldots,y_n$ . Fisher information asks: how well does an observation of $Y$ locate the parameter value that generated it?It assigns a larger value when nearby parameter choices produce observably different distributions, and a smaller value when many parameter choices would produce nearly indistinguishable data.

Shannon informationUncertainty about an outcome

How surprising or uncertain is the random state $Y$ ?

Fisher informationSensitivity to a parameter

How strongly does the distribution of $Y$ change when $\theta$ changes?

Fisher information is therefore not a bit count and not the mutual information between two random variables. In its classical use, $\theta$ is an unknown fixed value rather than a random variable with a prior. Fisher information is local: $I_Y(\theta)$ describes the measurement model's sensitivity near a particular parameter value. The subscript $Y$ reminds us that a different measurement procedure could carry a different amount of information about the same parameter.

A clean example: known Gaussian noise, unknown mean

Begin with a noise variable $\varepsilon\sim\mathcal N(0,\sigma^2)$ . The symbol $\sim$ means “is distributed according to”: the noise is Gaussian with known mean zero and known variance $\sigma^2$ . Now define the measurement by $Y=\mu+\varepsilon$ , where $\mu$ is the unknown signal. Adding $\mu$ shifts the noise distribution from being centered at zero to being centered at $\mu$ , without changing its variance. Therefore the observation has one Gaussian measurement model, $Y\mid\mu\sim\mathcal N(\mu,\sigma^2)$ ; the first Gaussian described its noise, while this one describes the resulting observation.

FROM ZERO-MEAN NOISE TO THE OBSERVATION MODEL

\varepsilon\sim\mathcal N(0,\sigma^2),\qquad Y=\mu+\varepsilon\qquad\Longrightarrow\qquad Y\mid\mu\sim\mathcal N(\mu,\sigma^2)

Capital $Y$ denotes the measurement before it is observed; lowercase $y_i$ denotes an actual observed value. Conditional on $\mu$ , the unknown mean is treated as one fixed constant; only $\varepsilon$ varies from one possible observation to another. Expectation therefore shifts by $\mu$ , while variance is unchanged because adding a constant moves every possible value equally without changing any value's displacement from the center.

Y=\mu+\varepsilon,\qquad \mathbb E[\varepsilon]=0,\qquad \operatorname{Var}(\varepsilon)=\sigma^2

Begin with a fixed signal μ plus random zero-centered noise ε.

\mathbb E[Y\mid\mu]=\mathbb E[\mu+\varepsilon\mid\mu]=\mu+\mathbb E[\varepsilon]=\mu

Conditioning holds μ fixed, so it passes through the expectation; the average noise contribution is zero.

Y-\mathbb E[Y\mid\mu]=(\mu+\varepsilon)-\mu=\varepsilon

Subtracting the observation mean removes the constant shift and leaves exactly the noise.

\operatorname{Var}(Y\mid\mu)=\mathbb E[(Y-\mu)^2\mid\mu]=\mathbb E[\varepsilon^2]=\sigma^2

Variance averages squared displacement from the mean, so the observation inherits the noise variance.

The noise therefore does not move the distribution's mean; it moves individual observations away from that mean. Small $\sigma$ keeps likely observations near $\mu$ , so an observed value points back to a narrow range of plausible means. Large $\sigma$ permits likely observations much farther from $\mu$ . The same observed value can then be explained by many candidate means combined with different plausible noise values, making the true mean harder to locate.

Forward sampling and inverse estimation

How observation noise becomes uncertainty about the unknown mean

Scroll horizontally to compare the two noise levels →

The top row is the forward measurement model

p(y\mid\mu^\ast)

: both cases have the same fixed true mean

\mu^\ast=0

, but larger noise scatters the possible observations farther away. The bottom row reverses the question. Each observation creates a likelihood over candidate means; multiplying four independent likelihoods produces the bold combined curve. Small noise makes those curves overlap within a narrow region, while large noise leaves a broad range plausible. For one observation,

I_1(\mu)=1/\sigma^2

and

\sigma=1/\sqrt{I_1}

. For four observations, the combined width is

1/\sqrt{4I_1}=\sigma/2

. The curves are height-normalized so their widths can be compared.

From log-likelihood slope to Fisher information

After observing $Y=y$ , regard $p(y\mid\mu)$ as a function of the candidate parameter $\mu$ . This is the likelihood. Its logarithm is the log-likelihood $\ell(\mu;y)$ . Differentiating the log-likelihood with respect to the parameter gives the score: one observation's local recommendation about which way the parameter should move to make that observation more likely.

Y\mid\mu\sim\mathcal N(\mu,\sigma^2),\qquad \sigma^2\ \text{known},\ \mu\ \text{unknown}

Specify the measurement model and the parameter-estimation problem.

\ell(\mu;y)=\ln p(y\mid\mu)=-\frac12\ln(2\pi\sigma^2)-\frac{(y-\mu)^2}{2\sigma^2}

Write the Gaussian log-likelihood as a function of the candidate mean μ.

s_\mu(y)=\frac{\partial}{\partial\mu}\ell(\mu;y)=\frac{y-\mu}{\sigma^2}

Differentiate once. The score is the slope of the log-likelihood.

How does score variance tell us where the parameter is?

Return to the original goal: determine how well an observation of $Y$ locates the parameter that generated it. Each possible observation produces its own likelihood curve—and corresponding log-likelihood curve—over candidate parameter values. A narrow curve casts a precise vote for a small range of candidates; a broad curve leaves a wide range plausible. The score records the local direction and strength of that vote. A positive score recommends increasing the parameter, a negative score recommends decreasing it, and a score near zero provides little local pressure in either direction.

One candidate parameter, three possible observations

The score turns each log-likelihood curve into a directional vote

Scroll horizontally to follow all three votes →

Each colored curve is the log-likelihood over candidate means after a particular observation. The three observed values

y=-1.8,0,1.8

are inputs to the same score function

s_\mu(y)

. The dots hold its parameter input fixed at the same candidate mean

\mu_0

; the short tangent lines show

s_{\mu_0}(y)

for each observation. The tangent's sign says which way to move the candidate mean—not a prior mean—to make that observation more likely, while its steepness gives the recommendation's strength.

In the diagram, $y=-1.8$ , $y=0$ , and $y=1.8$ are three possible inputs to the same score function. We hold its parameter input fixed at the candidate $\mu_0=0$ and calculate $s_{\mu_0}(y)$ for each observation. If $y=-1.8$ , the negative score says that moving the candidate mean left would make that observation more likely. The candidate $\mu_0$ is simply the parameter value at which we choose to measure the slopes—often the current estimate in an iterative estimation procedure. Holding $\mu_0$ fixed while varying the possible observation explains why one parameter point has a whole distribution of possible scores.

From a candidate value to the generating parameter

The generating parameter, written $\theta^\ast$ , is the fixed but unknown true value used by the data-generating process: $Y\sim p(y\mid\theta^\ast)$ . In the Gaussian example it is the true mean $\mu^\ast$ in $Y\sim\mathcal N(\mu^\ast,\sigma^2)$ . If we evaluate scores at a candidate far from $\theta^\ast$ , observations may systematically recommend moving toward the truth. At $\theta^\ast$ itself, however, the scores should have no shared direction: observations on one side recommend moving up, observations on the other recommend moving down, and their recommendations balance.

Fisher information is written as a function $I(\theta)$ because this question can be asked at every point in the model family. At each candidate point $\theta$ , imagine that it were the generating parameter, draw many possible observations $Y\sim p(y\mid\theta)$ , and examine the distribution of scores $s_\theta(Y)$ that those possible observations produce at that same point. Each realized observation produces one score; the plural refers to the collection across possible observations. The actual experiment has one unknown $\theta^\ast$ ; the function $I(\theta)$ tells us how informative the experiment would be at every possible parameter value.

Why can we rely on the score being centered at zero?

The symmetry of the Gaussian model makes this centering plausible, but Fisher information is meant to work beyond Gaussians. The following argument establishes it for a broad class of models. The score is a random variable because it depends on the still-random observation $Y$ . Its average is zero whenever three ordinary assumptions hold: $p(y\mid\theta)$ changes smoothly with $\theta$ ; the set of possible observations does not change with $\theta$ ; and the parameter derivative may be moved through the integral. These assumptions are often bundled together under the name regularity conditions, and they hold for the Gaussian model above. They let us use the fact that the total probability $\int p(y\mid\theta)\,dy$ remains equal to one for every parameter value.

s_\theta(y)=\frac{\partial}{\partial\theta}\ln p(y\mid\theta)

Begin with the score: the parameter derivative of the log-density.

\mathbb E_\theta[s_\theta(Y)]=\int p(y\mid\theta)\,\frac{\partial}{\partial\theta}\ln p(y\mid\theta)\,dy

Write the expectation explicitly by weighting each possible score by its probability.

\frac{\partial}{\partial\theta}\ln p(y\mid\theta)=\frac{1}{p(y\mid\theta)}\frac{\partial p(y\mid\theta)}{\partial\theta}

Use the chain rule: the derivative of ln u is (1/u) times the derivative of u.

p(y\mid\theta)\,\frac{1}{p(y\mid\theta)}\frac{\partial p(y\mid\theta)}{\partial\theta}=\frac{\partial p(y\mid\theta)}{\partial\theta}

The probability multiplying the score cancels its reciprocal.

\mathbb E_\theta[s_\theta(Y)]=\int\frac{\partial p(y\mid\theta)}{\partial\theta}\,dy=\frac{\partial}{\partial\theta}\int p(y\mid\theta)\,dy

A regularity condition permits moving the parameter derivative outside the integral.

\mathbb E_\theta[s_\theta(Y)]=\frac{\partial}{\partial\theta}1=0

Every probability density integrates to one, and changing θ cannot change that total.

This result establishes the needed center point. Changing the parameter may increase the log-likelihood slope for some possible observations, but normalization forces compensating decreases elsewhere. When weighted by the model itself, those positive and negative scores balance to zero. The score spread is therefore centered on “no preferred direction,” allowing its spread to measure how strongly observations separate nearby parameter values rather than how strongly they collectively recommend moving somewhere else.

From zero-centered scores to Fisher information

Zero average alone cannot distinguish an informative experiment from an uninformative one: both strong votes that cancel and uniformly weak votes can average to zero. Steep, narrow likelihoods produce strong positive and negative scores; flat, broad likelihoods produce scores clustered near zero. Variance distinguishes these cases. For one parameter, $\operatorname{Var}(S)=\mathbb E[S^2]-\mathbb E[S]^2$ . Substituting the zero score mean gives $\operatorname{Var}_\theta[s_\theta(Y)]=\mathbb E_\theta[s_\theta(Y)^2]$ —the scalar Fisher information $I_Y(\theta)$ . Squaring removes the signs, and averaging measures the typical strength of an observation's recommendation. This is how Fisher information answers the original question: a large value means an observation tends to distinguish nearby parameter values strongly; a small value means it leaves them difficult to tell apart.

s_\mu(Y)=\frac{Y-\mu}{\sigma^2},\qquad Y\sim\mathcal N(\mu,\sigma^2)

For each possible mean μ, imagine that μ generates the observation and evaluate the score at that same μ.

\mathbb E_\mu[s_\mu(Y)]=\frac{\mathbb E_\mu[Y-\mu]}{\sigma^2}=0

The Gaussian observation is centered at μ, so upward and downward recommendations balance.

I_Y(\mu)=\operatorname{Var}_\mu[s_\mu(Y)]=\mathbb E_\mu[s_\mu(Y)^2]

Because the score mean is zero, its variance equals its expected square.

I_Y(\mu)=\frac{\mathbb E_\mu[(Y-\mu)^2]}{\sigma^4}=\frac{\sigma^2}{\sigma^4}=\frac{1}{\sigma^2}

The squared displacement averages to the known variance σ², leaving Fisher information 1/σ².

The same two noise levels, viewed through their scores

Why the width of the score distribution measures parameter information

Scroll horizontally to compare both cases →

Both distributions are centered at zero because observations generated at the true mean do not systematically recommend moving left or right. What differs is their spread. With small noise, each likelihood is narrow and steep, so observations on opposite sides of

\mu

produce strongly negative or positive scores. With large noise, the likelihoods are flatter and their scores remain near zero. Fisher information is the variance of these scores: wide here means the parameter is located sharply.

Score view

I(\theta)=\mathbb E_\theta[s_\theta s_\theta^{\mathsf T}]

The covariance of the log-likelihood slopes.

Curvature view

I(\theta)=-\mathbb E_\theta[\nabla_\theta^2\ell_\theta]

The average sharpness of the log-likelihood peak.

Estimation view

\operatorname{Cov}(\hat\theta)\succeq I_n(\theta)^{-1}

A lower bound on unbiased-estimator uncertainty.

The Fisher matrix is not itself the gradient of a distribution. The score is one observation's log-likelihood gradient; the Fisher matrix is the covariance of those score vectors across possible observations. The curvature expression uses the Hessian, the matrix of second derivatives of the log-likelihood. In one dimension, the Hessian does not disappear: it becomes the single ordinary second derivative $\partial_\theta^2\ell$ , or equivalently a $1\times1$ matrix. Fisher information is the negative expected value of that Hessian, not generally the Hessian produced by one particular observation. The equality between the score and curvature views follows from probability normalization under these same assumptions; Step 4 of the later KL Taylor derivation proves the identity explicitly.

Closing the estimation loop: from scores to a parameter estimate

Suppose the current candidate mean is $\mu_0$ , while the observations were generated around the unknown true mean $\mu^\ast$ . One observation's score may point left or right merely because of noise. With several independent observations, however, their scores add. For the Gaussian mean, the total score is

U_n(\mu_0)=\sum_{i=1}^n s_{\mu_0}(y_i)=\sum_{i=1}^n\frac{y_i-\mu_0}{\sigma^2}=\frac{n(\bar y-\mu_0)}{\sigma^2}

Add the directional recommendations from all observations; their common direction is determined by the sample mean ȳ.

\mathbb E_{\mu^\ast}[U_n(\mu_0)]=\frac{n(\mu^\ast-\mu_0)}{\sigma^2}

If the candidate is below the true mean, the expected total score is positive; if it is above, the expected score is negative.

I_n(\mu_0)=nI_1(\mu_0)=\frac{n}{\sigma^2}

Independent observations also add Fisher information: n observations supply n times the local sensitivity.

\mu_{\mathrm{new}}=\mu_0+I_n(\mu_0)^{-1}U_n(\mu_0)=\mu_0+\frac{\sigma^2}{n}\frac{n(\bar y-\mu_0)}{\sigma^2}=\bar y

Inverse Fisher information converts the score into a change measured in parameter units. Here one Fisher-scoring step reaches the sample mean.

The score therefore supplies the direction of correction, while inverse Fisher information calibrates its size according to how sensitive the model is to parameter changes. This update is called Fisher scoring. The Gaussian mean is exceptionally simple: its score is linear and its Fisher information is constant, so one full update reaches the maximum-likelihood estimate $\hat\mu=\bar y$ . In more complicated families, both quantities can change with the candidate parameter, so the score and Fisher matrix are recomputed through several updates.

Reaching $\bar y$ does not mean recovering $\mu^\ast$ exactly. The sample mean is still displaced by the particular noise in a finite sample. Fisher information does not give an average number of samples required to recover a continuous parameter exactly; instead, it quantifies how quickly attainable precision improves. The Cramér–Rao bound converts the accumulated information into a best-case variance for an unbiased estimator:

N INDEPENDENT GAUSSIAN OBSERVATIONS

I_n(\mu)=\frac{n}{\sigma^2},\qquad \operatorname{Var}(\hat\mu)\geq\frac{1}{I_n(\mu)}=\frac{\sigma^2}{n},\qquad \operatorname{SD}(\hat\mu)\geq\frac{\sigma}{\sqrt n}

In this model the sample mean attains the bound: doubling the number of observations doubles Fisher information, while reducing the best-achievable standard error by a factor of $\sqrt2$ . Lower measurement noise and more independent observations are therefore two direct routes to more precise parameter estimation.

From one parameter to a parameter space

With several parameters, the score becomes a gradient vector and Fisher information becomes its covariance matrix. Large curvature in a direction means observations distinguish movements in that parameter combination well; small curvature means a poorly identified direction. A smoothly parameterized, identifiable family of distributions is already a statistical manifold: each parameter choice names a point, and nearby parameters name nearby models. A categorical family is the special case that can be drawn as a probability simplex; a Gaussian family has a different parameter space. Attaching the Fisher matrix to every point equips the statistical manifold with a metric—a local rule for assigning statistical size to movements in each parameter direction.

Why Fisher information appears in the KL bowl

Both constructions ask how sensitively a probability model responds to a small parameter change. Estimation theory measures that sensitivity through score variance or expected log-likelihood curvature. Local KL measures how quickly a nearby model becomes distinguishable from the current one. When KL is Taylor-expanded, normalization removes the linear score term and the first surviving coefficient is the negative expected log-density Hessian—the same $I(\theta)$ defined above. The derivation that follows proves this identification explicitly.

Estimation perspective and equivalent formulas: measurement intuition, score and curvature visualization, and normalization-based derivation.

The entropy formula $D_{\mathrm{KL}}(P\|Q)=H(P;Q)-H(P)$ evaluates KL exactly for any permitted pair of models. The quadratic expression evaluates a Taylor approximation to that same KL function when $Q=p_{\theta+\Delta\theta}$ is close to the fixed $P=p_\theta$ . Thus “different scales” means exact at finite separation versus a close-up around $\Delta\theta=0$ . As $\Delta\theta$ shrinks, the cubic and higher corrections shrink faster than the quadratic term; farther away, they can no longer be ignored. The calculation below begins with the exact cross-entropy formula and shows precisely where the local quadratic approximation comes from.

One convention before the algebra

Let $\ell_\theta(x)=\ln p_\theta(x)$ . Natural logarithms measure information in nats and give the standard Fisher matrix without an extra constant. KL in bits is the same quantity divided by $\ln 2$ .

1

Insert the nearby model into the cross-entropy definition

Set $P=p_\theta$ and $Q=p_{\theta+\Delta\theta}$ . Both expectations are over data drawn from the unchanged distribution $p_\theta$ .

D_{\mathrm{KL}}(P\|Q)=H(P;Q)-H(P)

Start with KL as the extra coding cost: cross-entropy minus entropy.

H(P;Q)=\mathbb E_{X\sim P}[-\ln Q(X)]

Write out cross-entropy: average Q’s negative log-probability over outcomes generated by P.

H(P)=\mathbb E_{X\sim P}[-\ln P(X)]

Write out entropy: average P’s own negative log-probability over the same outcomes.

H(p_\theta;p_{\theta+\Delta\theta})=\mathbb E_\theta[-\ln p_{\theta+\Delta\theta}(X)]

Replace P by pθ and Q by the nearby model. The averaging distribution remains pθ.

H(p_\theta)=\mathbb E_\theta[-\ln p_\theta(X)]

Make the same replacement in the entropy term.

D_{\mathrm{KL}}=\mathbb E_\theta[-\ln p_{\theta+\Delta\theta}(X)]-\mathbb E_\theta[-\ln p_\theta(X)]

Substitute those two expanded definitions back into cross-entropy minus entropy.

D_{\mathrm{KL}}=\mathbb E_\theta[-\ell_{\theta+\Delta\theta}(X)]-\mathbb E_\theta[-\ell_\theta(X)]

Use ℓϑ(x) = ln pϑ(x) as shorthand for each pointwise log-density.

D_{\mathrm{KL}}=\mathbb E_\theta[\ell_\theta(X)-\ell_{\theta+\Delta\theta}(X)]

Use linearity to combine the averages and simplify the minus signs.

The nearby log-density $\ell_{\theta+\Delta\theta}(X)$ is a value assigned to the random outcome $X$ ; it is not itself an entropy or cross-entropy. Only $\mathbb E_\theta[-\ell_{\theta+\Delta\theta}(X)]$ —the negative log-density averaged over data from $p_\theta$ —is the cross-entropy. Likewise, $\mathbb E_\theta[-\ell_\theta(X)]=H(p_\theta)$ is the entropy of the current model.

2

Taylor-expand the nearby model's log-density

The general problem a Taylor expansion addresses is this: when a smooth function is difficult to evaluate directly, but its value and derivatives are known at one input, how can we estimate the value of different inputs that are nearby? It replaces the difficult function locally with a polynomial built from its value, slope, curvature, and higher derivatives at that known input. Here the known input is $\theta$ and the nearby input is $\theta+\Delta\theta$ . In the exact KL expression above, all dependence on the nearby model sits inside $\ell_{\theta+\Delta\theta}(X)$ , so this is the term we expand.

Define the score $s_\theta=\nabla_\theta\ell_\theta$ , the vector of first derivatives, and the log-density Hessian $J_\theta=\nabla_\theta^2\ell_\theta$ . With one parameter, the Hessian is simply the second derivative. With several parameters, it is the matrix of every second partial derivative, recording curvature along—and interactions between—the parameter directions. Then

\ell_{\theta+\Delta\theta}(x)=\ell_\theta(x)+s_\theta(x)^{\mathsf T}\Delta\theta+\frac12\Delta\theta^{\mathsf T}J_\theta(x)\Delta\theta+O(\|\Delta\theta\|^3)

Apply the multivariable Taylor expansion to the log-density around θ.

D_{\mathrm{KL}}=\mathbb E_\theta[\ell_\theta(X)-\ell_{\theta+\Delta\theta}(X)]

Return to the expression for KL obtained in Step 1.

D_{\mathrm{KL}}=\mathbb E_\theta\!\left[\ell_\theta(X)-\left(\ell_\theta(X)+s_\theta(X)^{\mathsf T}\Delta\theta+\frac12\Delta\theta^{\mathsf T}J_\theta(X)\Delta\theta+O(\|\Delta\theta\|^3)\right)\right]

Replace the nearby log-density inside the exact KL formula by its Taylor expansion.

D_{\mathrm{KL}}=\mathbb E_\theta\!\left[-s_\theta(X)^{\mathsf T}\Delta\theta-\frac12\Delta\theta^{\mathsf T}J_\theta(X)\Delta\theta+O(\|\Delta\theta\|^3)\right]

The two copies of ℓθ(X) cancel inside the expectation.

D_{\mathrm{KL}}=-\mathbb E_\theta[s_\theta]^{\mathsf T}\Delta\theta-\frac12\Delta\theta^{\mathsf T}\mathbb E_\theta[J_\theta]\Delta\theta+O(\|\Delta\theta\|^3)

Take the expectation term by term; Δθ is fixed, so it passes through the expectation.

The complicated change in log-density has now become a local slope term and a local curvature term, plus an error that begins at third order. To learn the actual local shape of KL, we next determine what happens to each coefficient after averaging over $p_\theta$ .

3

The linear term vanishes because every distribution normalizes to one

Carry the result from Step 2 forward. Its first coefficient is the expected score $\mathbb E_\theta[s_\theta]$ :

D_{\mathrm{KL}}=-\mathbb E_\theta[s_\theta]^{\mathsf T}\Delta\theta-\frac12\Delta\theta^{\mathsf T}\mathbb E_\theta[J_\theta]\Delta\theta+O(\|\Delta\theta\|^3)

This is the KL approximation obtained at the end of Step 2; we now evaluate its linear coefficient.

\mathbb E_\theta[s_\theta]=\int p_\theta(x)\,\nabla_\theta\ln p_\theta(x)\,dx

Expand the expectation of the score as an integral over pθ.

\mathbb E_\theta[s_\theta]=\int \nabla_\theta p_\theta(x)\,dx

Use p∇ ln p = ∇p, which follows from the chain rule.

\mathbb E_\theta[s_\theta]=\nabla_\theta\int p_\theta(x)\,dx

Move the parameter derivative outside the integral under the regularity assumptions below.

\mathbb E_\theta[s_\theta]=\nabla_\theta 1

Every probability distribution is normalized: its total probability is one.

\mathbb E_\theta[s_\theta]=0

Differentiate the constant one. The score therefore has zero mean.

D_{\mathrm{KL}}=-\frac12\Delta\theta^{\mathsf T}\mathbb E_\theta[J_\theta]\Delta\theta+O(\|\Delta\theta\|^3)

Substitute Eθ[sθ] = 0 back into KL. The entire linear term disappears; the Hessian term remains.

This normalization identity is the algebraic reason “KL is minimal where the models match” removes the first-order term. We have not left the original KL formula; we have simplified its Taylor approximation one coefficient at a time.

4

The remaining Hessian is the Fisher information matrix

Step 3 left KL in terms of the expected log-density Hessian. Differentiating the same normalization identity a second time lets us rewrite that remaining curvature as Fisher information, then substitute it directly back into KL:

D_{\mathrm{KL}}=-\frac12\Delta\theta^{\mathsf T}\mathbb E_\theta[J_\theta]\Delta\theta+O(\|\Delta\theta\|^3)

Carry forward the quadratic KL expression obtained at the end of Step 3.

0=\nabla_\theta^2\!\int p_\theta(x)\,dx=\int\nabla_\theta^2p_\theta(x)\,dx

Differentiate ∫pθ(x) dx = 1 twice; the second derivative of one is zero.

\nabla_\theta^2p_\theta(x)=p_\theta(x)\bigl(J_\theta(x)+s_\theta(x)s_\theta(x)^{\mathsf T}\bigr)

Use ∇pθ = pθsθ for the first derivative, then apply the product rule to differentiate again.

0=\mathbb E_\theta[J_\theta+s_\theta s_\theta^{\mathsf T}]

Insert that identity into the integral and recognize an expectation under pθ.

-\mathbb E_\theta[J_\theta]=\mathbb E_\theta[s_\theta s_\theta^{\mathsf T}]\equiv I(\theta)

Rearrange. The expected score outer product is the Fisher information matrix.

\boxed{D_{\mathrm{KL}}^{(\mathrm{nats})}(p_\theta\|p_{\theta+\Delta\theta})=\frac12\Delta\theta^{\mathsf T}I(\theta)\Delta\theta+O(\|\Delta\theta\|^3)}

Replace −Eθ[Jθ] in the quadratic coefficient by I(θ). This is the local Fisher approximation to KL.

D_{\mathrm{KL}}^{(\mathrm{bits})}=\frac{1}{\ln 2}\,D_{\mathrm{KL}}^{(\mathrm{nats})}

Convert nats to bits by dividing by ln 2.

This calculation assumes the model is smooth, its support does not change with $\theta$ , and differentiation may pass through the integral. For discrete outcomes, replace each integral by a sum.

KL has not become a global distance: it is still asymmetric for separated distributions. What loses orientation is only the second-order term, because the quadratic form $\Delta\theta^{\mathsf T}I(\theta)\Delta\theta$ is symmetric under $\Delta\theta\mapsto-\Delta\theta$ . The higher-order terms retain KL's directional information. This symmetric microscopic part is enough to supply a local metric; the directional higher-order part returns later when we reach dual connections.

What “zooming in” and “ruler” mean

Zooming in means holding one distribution $p_\theta$ fixed, restricting attention to nearby members of its parameterized family $p_{\theta+\Delta\theta}$ , and taking $\Delta\theta$ small enough that cubic and higher terms are negligible. We are studying how the entire probability output $p_\theta(x)$ changes with its parameters, not merely how the parameter numbers change on paper.

The Fisher matrix is a local ruler because $ds^2=\Delta\theta^{\mathsf T}I(\theta)\Delta\theta$ converts a small coordinate displacement into the leading-order KL distinguishability of the resulting distributions. Equal-sized parameter changes can therefore have different statistical sizes at different models or in different directions. This ruler is local: measuring a long route requires attaching it at every distribution along the path and adding the resulting infinitesimal lengths.

What the zoom accomplished: KL supplied a local ruler without becoming a distance itself. Repeating this construction at every

p_\theta

gives every distribution its own ruler. How that ruler changes from point to point is the geometry of the statistical manifold in Step 4.

4

Repeat at every possible model

The family becomes a statistical manifold

Collect every distribution in the model family into one space $\mathcal M$ . Each distribution is a point, its parameters $\theta$ are coordinates, and $d\theta$ is a tangent direction—an infinitesimal way to change the distribution. Attaching the Fisher matrix to every point gives the Fisher metric.

\mathcal M=\{p_\theta(x):\theta\in\Theta\},\qquad ds^2=d\theta^{\mathsf T}I(\theta)d\theta

Why this still is not the end: A metric tells us lengths and angles. It does not, by itself, tell us how to carry a direction from one distribution to another and decide whether a path kept going “straight.” That requires a connection.

5

Add a rule for preserving direction

Two statistical notions of straightness appear

Information geometry supplies two natural connections. The mixture connection calls a path straight when probabilities change at a constant rate. The exponential connection calls it straight when log-probability ratios change at a constant rate. They disagree about the route, but the Fisher metric pairs them as dual descriptions.

mixture-straight

p_t=(1-t)p_0+tp_1

exponential-straight

p_t(x)\propto p_0(x)^{1-t}p_1(x)^t

This is the destination: “Dual geometry” means one Fisher metric together with two mutually paired rules for not turning. KL asymmetry helps generate this pair; the interactive probability triangle later makes their different routes visible.

02

One distribution modeling another

Let P be reality: the distribution actually generating observations. Let Q be your model: the probabilities you use to predict and encode them. Your example is perfect—reality can be a fair coin while your model insists on 75% heads.

Interactive toy model

Encode coin P using a code designed for coin Q

Reality P: heads probability 0.50Model Q: heads probability 0.75Message length: 100 flips

P — the coin generating the dataHeads 50% · Tails 50%

H

T

Q — your model and codebookHeads 75% · Tails 25%

H

T

Best possible1.000 bits/flipH(P): code matched to reality

Using model Q1.208 bits/flipH(P;Q): expected code length

Model penalty0.208 bits/flipDₖₗ(P‖Q): excess bits

Across 100 flips20.75 extra bitsthe mismatch accumulates

If the next result isHeads0.415 ideal bits

-\log_2 Q(H)

If the next result isTails2.000 ideal bits

-\log_2 Q(T)

Your model expects 75 heads per 100 flips, while reality produces about 50. Q rewards its expected outcomes with short codewords, but pays dearly whenever its “surprising” outcome occurs.

What does it mean for Q to model P?

Q assigns a probability—and therefore a degree of expectedness—to every outcome P might generate. We judge Q by averaging its surprises under P:

\mathbb{E}_{x\sim P}[-\log Q(x)]

. A good Q places probability where P actually produces data.

03

KL divergence: the price of the wrong codebook

If an outcome has model probability $Q(x)$ , its ideal information content is $-\log_2 Q(x)$ bits. Average that quantity over outcomes generated by P and you get cross-entropy. Subtract the best possible average code length $H(P)$ , and the remainder is KL divergence.

CROSS-ENTROPY

H(P;Q)=\mathbb{E}_{x\sim P}\!\left[-\log_2 Q(x)\right]

KL DIVERGENCE

D_{\mathrm{KL}}(P\|Q)=H(P;Q)-H(P)=\mathbb{E}_{x\sim P}\!\left[\log_2\frac{P(x)}{Q(x)}\right]

Your 50/50 reality versus 75/25 model

A Q-code gives heads an ideal length of $-\log_2(0.75)\approx0.415$ bits.
It gives tails a much longer length: $-\log_2(0.25)=2$ bits.
Reality produces each half the time, so $H(P;Q)\approx\tfrac12(0.415)+\tfrac12(2)=1.2075$ bits/flip.
A fair-coin code needs $H(P)=1$ bit/flip.
Therefore $D_{\mathrm{KL}}(P\|Q)\approx0.2075$ excess bits/flip.

Why the order matters

D_{\mathrm{KL}}(P\|Q)

asks how Q performs on data from P. Reverse the arguments and you ask a different question. If Q declares an event impossible while P can produce it, Q eventually encounters an outcome requiring an infinite ideal code length.

But fractional bits are not literal one-flip codewords—right?

Right. A single binary prefix code cannot use 0.415 of a physical bit. The statement becomes operational for long messages: arithmetic coding and related schemes approach the expected $-\log_2 Q(x)$ per symbol. This is the source-coding interpretation developed from Shannon’s theory and treated carefully by Cover and Thomas.

Coding interpretation: Shannon (1948); Cover & Thomas (2006).

04

Fisher information: zoom into KL

KL divergence compares separated distributions. Now place two models extremely close together: $p_\theta$ and $p_{\theta+d\theta}$ . As derived in the bridge above, normalization makes the first-order score average vanish; the leading term is quadratic. Its coefficient is Fisher information.

LOCAL KL EXPANSION (NATS)

D_{\mathrm{KL}}(p_\theta\|p_{\theta+d\theta})\approx\frac12 d\theta^{\mathsf T}I(\theta)d\theta

FISHER INFORMATION FOR A BERNOULLI COIN

I(p)=\mathbb{E}\!\left[\left(\frac{\partial}{\partial p}\log P(X\mid p)\right)^2\right]=\frac{1}{p(1-p)}

The formula for $I$ depends on the family being studied. A Bernoulli family has one parameter $p$ , so its Fisher information is one number. A Gaussian family parameterized by $(\mu,\sigma)$ has a $2\times2$ Fisher matrix; a Beta family parameterized by $(\alpha,\beta)$ has another. In every case, the entries describe how sensitive that particular family is to small movements in its own parameter directions.

Interactive local geometry

How distinguishable are two nearby coins?

Starting coin p = 0.50Small change dp = 0.020Independent observations N = 100

P

p = 0.500

local separation⟷

Nearby Q

p = 0.520

Fisher informationI(p) = 4.000sensitivity at this coin

Exact KL0.000801 natsDₖₗ(P‖Q)

Local approximation0.000800 nats½ I(p) dp²

Best-case estimator SD≥ 0.0500Cramér–Rao, N observations

Exact KL0.0008006

Fisher approximation0.0008000

Fisher–Rao distance0.04001

Try moving $p$ toward 0 or 1. The same numerical change $dp$ becomes more informative because a rare outcome changes by a large proportion. That is why $I(p)=1/[p(1-p)]$ grows near the edges.

The same numerical step need not be the same statistical step

For a Bernoulli coin,

I(0.5)=4

, while

I(0.01)\approx101

. Therefore, for the same sufficiently small change

dp

, the quadratic KL cost near

p=0.01

is about twenty-five times the cost near

p=0.5

. Near the center, a tiny probability change is difficult to detect from data. Near an edge, that same absolute change is a large relative change in a rare event and is much easier to distinguish. The Fisher ruler records this difference even though an ordinary number-line ruler sees the same

dp

.

Fisher information versus the Fisher information metric

At one modelFisher information

$I(p)$ is one local sensitivity value—or a matrix when there are multiple parameters.

Across all modelsFisher metric

Attach that matrix to every point. The smoothly varying field defines $ds^2=I_{ij}\,d\theta^i d\theta^j$ .

Same object, different viewpoint

“Fisher information matrix” emphasizes estimation at a particular distribution. “Fisher information metric” emphasizes how those matrices fit together over the whole family. The metric is what turns local distinguishability into lengths and angles.

Derive I(p) for the Bernoulli distribution

For $X\in\{0,1\}$ , the Bernoulli log-likelihood is:

\log P(X\mid p)=X\log p+(1-X)\log(1-p)

\frac{\partial}{\partial p}\log P(X\mid p)=\frac{X}{p}-\frac{1-X}{1-p}

The score is $1/p$ when $X=1$ and $-1/(1-p)$ when $X=0$ . Square it and average using probabilities $p$ and $1-p$ :

I(p)=p\left(\frac1{p^2}\right)+(1-p)\left(\frac1{(1-p)^2}\right)=\frac1{p(1-p)}

Statistical information and its geometry: Fisher (1922); Rao (1945).

05

A statistical manifold—without the intimidation

A manifold is a space that looks like ordinary Euclidean space when you zoom in. The surface of Earth is curved globally, but your neighborhood admits an ordinary flat map. Likewise, a family $p(x\mid\theta)$ can be described locally by parameter coordinates $\theta$ .

A family here means a smoothly parameterized collection with a common mathematical form: all Bernoulli coins as $p$ varies, all Gaussians as $(\mu,\sigma)$ vary, or all Beta distributions as $(\alpha,\beta)$ vary. In this tutorial, paths and distances stay inside one chosen family, where the required derivatives are defined. Two differently named model classes can still be compared geometrically if they are embedded in a larger common statistical manifold, but unrelated parameterizations do not automatically provide a path between them.

PointOne complete distribution

p(x\mid\theta)

CoordinatesThe parameters

\theta

used to name it

Tangent vectorAn infinitesimal way to change the distribution

MetricA rule for measuring tiny displacements

ConnectionA rule for saying whether a direction changed while moving

GeodesicA path with zero turning according to a chosen connection

From points to paths

Following the point analogy, a continuously changing distribution traces a path through the manifold. If its parameters follow $\theta(t)$ , then its derivative is a tangent direction: it records how the entire distribution is changing at that instant, just as the velocity of an ordinary curve records how a point is moving.

A DISTRIBUTION PATH AND ITS TANGENT

\dot p(x)=\frac{d}{dt}p_{\theta(t)}(x)=\nabla_\theta p_\theta(x)^{\mathsf T}\dot\theta

The Fisher metric assigns that tangent a statistical speed. Adding those infinitesimal lengths along the path gives its total Fisher–Rao length:

FISHER SPEED AND PATH LENGTH

\begin{aligned} \|\dot p\|_F^2&=\dot\theta^{\mathsf T}I(\theta)\dot\theta\\[4pt] L[\theta]&=\int\sqrt{\dot\theta^{\mathsf T}I(\theta)\dot\theta}\,dt \end{aligned}

Metric distance is obtained by comparing the lengths of paths joining two distributions within the chosen manifold. This is where the familiar shortest-path intuition enters—but it must be separated from the more general meaning of a geodesic.

Important correction about geodesics

“The shortest path” is a good first intuition for a Riemannian geodesic using the metric’s Levi–Civita connection. More generally, a geodesic means “move without turning according to this connection.” An affine geodesic need not minimize metric distance. This distinction is exactly what we need for dual geometry.

Probability coordinate $p\in(0,1)$ The fair coin is

p=0.5

.

same coins, renamed

Log-odds coordinate $\theta=\log\frac{p}{1-p}$ The fair coin is

\theta=0

.

Changing coordinates changes the numbers in the Fisher matrix, but not the intrinsic length ds. That coordinate independence is what earns Fisher information the word metric rather than merely “a useful Hessian.” Rao made this geometric viewpoint explicit.

06

Dual geometry: two meanings of “do not turn”

Imagine driving while trying to hold a direction steady. On a curved or abstract space, “hold steady” requires a comparison rule: a connection. Information geometry has two especially natural rules. One treats probabilities as the quantities that should change uniformly; the other treats log-probability ratios as the quantities that should change uniformly.

Interactive categorical manifold

Two routes between the same distributions

Travel from P₀ to P₁: t = 0.50

Mixture path: average probabilities Exponential path: average log-probabilities

m(t)A: 0.400B: 0.225C: 0.375

e(t)A: 0.356B: 0.301C: 0.343

Mixture route

m_i(t)=(1-t)P_{0i}+tP_{1i}

Constant velocity in probability coordinates.

Exponential route

e_i(t)\propto P_{0i}^{1-t}P_{1i}^{t}

Constant velocity in log-ratio coordinates.

m-connectionMixture-straight

Probabilities change linearly. In probability coordinates, this route looks visually straight.

e-connectionExponential-straight

Log ratios change linearly. In probability coordinates, this route usually looks curved.

Levi–Civita connectionMetric-straight

The Fisher–Rao geodesic is locally distance-minimizing. It is generally a third route.

So why call the first two dual?

The Fisher metric pairs the mixture and exponential connections so that what one connection does to a direction is balanced by what the other does to the direction it is paired with. In exponential families, natural parameters

\theta

are straight for the e-connection, expectation parameters

\eta

are straight for the m-connection, and

\theta

and

\eta

are related by Legendre duality.

Where KL asymmetry enters

The asymmetry is not the entire definition of dual geometry; it is the visible clue. Expand a divergence around P = Q. Its second-order term cannot see orientation and gives one symmetric metric. Its third-order terms can see orientation and generate a pair of dual connections. Reverse KL, and the two roles swap.

THE LOCAL PICTURE

D\xrightarrow{\text{second derivatives}}g\qquad D\xrightarrow{\text{third derivatives}}(\nabla,\nabla^{*})

The simplest honest summary

A metric tells you how much an infinitesimal move costs. A connection tells you whether you turned during the move. Information geometry has one canonical metric and two statistically meaningful, mutually paired notions of not turning.

Dual connections and divergence geometry: Amari & Nagaoka (2000); Nielsen (2020).

07

How to study this without drowning in prerequisites

Learn it as a spiral: revisit the same few distributions with one new tool each time. Do not wait to “finish differential geometry.” The Bernoulli, categorical, and Gaussian families are enough to make nearly every new idea concrete.

1Rebuild the information layer

Entropy, cross-entropy, KL, JS, and coding. Derive each for a Bernoulli coin.

2Learn only the geometry you need

Coordinates, tangent vectors, metrics, connections, and geodesics—always tied to a distribution family.

3Compute Fisher information

Do Bernoulli first, then categorical and univariate Gaussian. Verify that coordinate changes preserve $ds^2$ .

4Meet exponential families

Natural $\theta$ and expectation $\eta$ coordinates make duality concrete instead of mystical.

5Apply it

Implement a natural-gradient update and compare it with an ordinary parameter gradient.

Continue through the tutorials already here

Sources and further reading

C. E. Shannon (1948), “A Mathematical Theory of Communication.” Bell System Technical Journal.
T. Cover and J. Thomas (2006), Elements of Information Theory, 2nd ed.
R. A. Fisher (1922), “On the Mathematical Foundations of Theoretical Statistics.” Philosophical Transactions of the Royal Society A.
C. R. Rao (1945), “Information and the Accuracy Attainable in the Estimation of Statistical Parameters.” Reprinted in Breakthroughs in Statistics.
S. Amari and H. Nagaoka (2000), Methods of Information Geometry.
F. Nielsen (2020), “An Elementary Introduction to Information Geometry”. Open-access survey.
D. J. C. MacKay (2003), Information Theory, Inference, and Learning Algorithms.

08

Bonus: natural-gradient variational inference

Fisher geometry becomes operational when a probability distribution is itself being optimized. Variational inference does exactly that: it adjusts an approximate posterior $q_\phi(s)$ until it resembles the Bayesian posterior over the hidden state $s$ . The subscript $\phi$ names the adjustable coordinates used to represent the beliefs. If $\phi$ is the vector of probabilities, changing $\phi$ changes those probabilities directly. If $\phi$ is a vector of logits, the logits change first and $q_\phi$ changes through a softmax.

$s$ The hidden state whose value is uncertain

$\theta$ Parameters of the generative model

p_\theta(o,s)

; they may be fixed or learned

$\phi$ Coordinates of the approximate posterior

q_\phi(s)

that variational inference adjusts

These roles should not be collapsed. Inferring $s$ with $q_\phi(s)$ does not require learning the generative parameters $\theta$ . Those parameters may be supplied mechanically, estimated from previous data, or placed under their own prior and inferred in a larger model. Merely writing $p(s\mid\theta)$ says that the state distribution depends on $\theta$ ; it does not make $\theta$ an inferred quantity. Parameter inference additionally requires a prior $p(\theta)$ and data, producing a posterior such as $p(\theta\mid o)$ or a variational approximation $q(\theta)$ .

From a free-energy gradient to a belief update

A gradient is a vector: an arrow through the coordinate space. At the current coordinates $\phi$ , the free-energy gradient $\nabla_\phi F$ points in the direction where a small movement increases free energy fastest. Multiplying a vector by $-1$ reverses its arrow, so $-\nabla_\phi F$ points in the direction where free energy decreases fastest locally. The components of this reversed vector may be positive, negative, or zero, depending on the current location. Ordinary gradient descent scales the reversed vector by a positive learning rate $\rho$ :

ORDINARY COORDINATE-GRADIENT STEP

\Delta\phi=-\rho\nabla_\phi F

The gradient magnitude and $\rho$ together determine how far the coordinates move. The resulting belief update is $q_\phi(s)\rightarrow q_{\phi+\Delta\phi}(s)$ . But coordinate distance is not yet belief distance: the same numerical $\Delta\phi$ can barely alter one distribution and radically alter another.

Write a binary distribution as $q_p=(p,1-p)$ , where the single number $p$ is its coordinate within the binary family. Consider $q_A=(0.50,0.50)$ and $q_B=(0.25,0.75)$ . Their coordinates are $p_A=0.50$ and $p_B=0.25$ , so the signed coordinate displacement from $q_A$ to $q_B$ is $\Delta p=p_B-p_A=-0.25$ and its coordinate distance is $|\Delta p|=0.25$ . Component by component, the probability vector changes by $q_B-q_A=(-0.25,+0.25)$ .

This coordinate distance does not yet say how distinguishable the two distributions are. To see the problem, compare two movements in the same probability coordinate with exactly the same numerical displacement, $\Delta p=-0.001$ :

q_{0.500}=(0.500,0.500)\longrightarrow q_{0.499}=(0.499,0.501)

Near the center of the family, the first probability decreases by 0.001.

D_{\mathrm{KL}}(q_{0.500}\|q_{0.499})\approx2.00\times10^{-6}\ \text{nats}

These two nearly balanced distributions are extremely difficult to distinguish.

q_{0.020}=(0.020,0.980)\longrightarrow q_{0.019}=(0.019,0.981)

Near the edge of the family, the first probability again decreases by exactly 0.001.

D_{\mathrm{KL}}(q_{0.020}\|q_{0.019})\approx2.64\times10^{-5}\ \text{nats}

The same parameter step causes more than thirteen times as much distributional change near the rare-event edge.

Raw coordinate displacement has another limitation: its numerical size depends on how the family is parameterized. The same binary family can instead be parameterized by the log-odds $\phi=\ln\!\bigl(p/(1-p)\bigr)$ . This formula does not produce new distributions; it assigns different coordinate values to the distributions already named $q_A$ and $q_B$ . Substituting their probabilities shows where the value $\ln(1/3)$ comes from:

\phi_A=\ln\frac{p_A}{1-p_A}=\ln\frac{0.50}{0.50}=\ln1=0

The 50/50 distribution has odds of one to one, so its log-odds coordinate is zero.

\phi_B=\ln\frac{p_B}{1-p_B}=\ln\frac{0.25}{0.75}=\ln\frac13\approx-1.10

The 25/75 distribution has odds of one to three, which gives the negative log-odds coordinate.

\Delta\phi=\phi_B-\phi_A=\ln\frac13-0\approx-1.10

The same two distributions are now separated by about 1.10 units of log-odds rather than 0.25 units of probability.

The identical change from $q_A$ to $q_B$ therefore has displacement $\Delta p=-0.25$ in probability coordinates but $\Delta\phi\approx-1.10$ in log-odds coordinates. The distributions did not change; only the coordinates used to describe them changed. Together, the two examples expose both problems with raw coordinate distance: its numerical scale depends on the parameterization, and its statistical effect depends on the distribution's current location.

KL divergence avoids both problems by comparing the probability distributions themselves rather than their coordinate labels. For a sufficiently small movement from $\phi$ to $\phi+\Delta\phi$ , the local KL expansion approximates that distributional change. The Fisher matrix supplies the location-dependent conversion from coordinate displacement to local statistical change: it records how sensitively the current distribution $q_\phi$ responds to movement in each parameter direction.

LOCAL BELIEF CHANGE

D_{\mathrm{KL}}\!\left(q_\phi\middle\|q_{\phi+\Delta\phi}\right)\approx\frac12\Delta\phi^{\mathsf T}I(\phi)\Delta\phi

For the binary family in probability coordinates, $I(p)=1/[p(1-p)]$ . Substituting the two starting locations and the shared step $\Delta p=-0.001$ shows how Fisher information produces the different local KL changes:

I(0.500)=\frac1{0.500(0.500)}=4

At the balanced distribution, Fisher information assigns relatively low sensitivity to movement in p.

\frac12 I(0.500)(\Delta p)^2=\frac12(4)(0.001)^2=2.00\times10^{-6}

The local approximation reproduces the tiny KL change near the center.

I(0.020)=\frac1{0.020(0.980)}\approx51.02

At the rare-event distribution, the same probability coordinate is much more sensitive.

\frac12 I(0.020)(\Delta p)^2\approx\frac12(51.02)(0.001)^2\approx2.55\times10^{-5}

The parameter step is unchanged; the larger Fisher information makes its statistical effect much larger.

Large Fisher information in a directionThe beliefs are sensitive

A small coordinate movement already causes a large statistical change, so the inverse Fisher suppresses movement in that direction.

Small Fisher information in a directionThe beliefs are insensitive

A larger coordinate movement is needed to cause the same statistical change, so the inverse Fisher permits more movement.

The ordinary gradient already identifies the direction that decreases free energy most among coordinate movements of the same Euclidean length. Here, “the same Euclidean length” means that every candidate direction is tested using an equally long arrow in the parameter coordinates. It does not refer to the learning rate. This equal-length comparison selects the downhill direction; the learning rate is applied afterward to help determine the length of the update actually taken.

Aside

Why the negative gradient is the steepest coordinate direction

At the current coordinates $\phi$ , write the gradient as $g=\nabla_\phi F$ and consider a small candidate movement $d$ . The first-order change in free energy is $F(\phi+d)-F(\phi)\approx g^{\mathsf T}d$ . To compare directions fairly, first give every candidate movement the same ordinary coordinate length $\|d\|_2=c$ , where $\|d\|_2=\sqrt{d_1^2+\cdots+d_n^2}$ .

g^{\mathsf T}d=\|g\|_2\,\|d\|_2\cos\alpha=\|g\|_2c\cos\alpha

The dot product depends on the angle α between the gradient and the candidate movement.

\cos\alpha=-1\quad\Longrightarrow\quad d=-c\frac{g}{\|g\|_2}

The free-energy change is most negative when the candidate points exactly opposite the gradient.

This comparison determines a direction; it does not choose the final step length. Ordinary gradient descent then uses $\Delta\phi=-\rho g$ , whose Euclidean length is $\|\Delta\phi\|_2=\rho\|g\|_2$ . The learning rate $\rho$ scales the step, while the gradient's magnitude also contributes to how far the coordinates actually move.

The natural gradient changes what is held fixed in this comparison. Instead of giving every candidate direction the same Euclidean coordinate length, give every candidate the same maximum amount of local belief change, measured by KL divergence. The question becomes: among the steps allowed to change the belief distribution by this amount, which one produces the greatest local decrease in free energy? This is written compactly as:

MOST FREE-ENERGY DECREASE PER UNIT OF BELIEF CHANGE

\min_{\Delta\phi}\;\bigl(\nabla_\phi F\bigr)^{\mathsf T}\Delta\phi \quad\text{subject to}\quad \frac12\Delta\phi^{\mathsf T}I(\phi)\Delta\phi\leq\varepsilon \qquad\Longrightarrow\qquad \Delta\phi\propto-I(\phi)^{-1}\nabla_\phi F

Read the expression from left to right. The symbol $\min_{\Delta\phi}$ means “search over the possible candidate steps $\Delta\phi$ and choose the one that makes the following quantity as small as possible.” Each part then has a direct interpretation:

F(\phi+\Delta\phi)-F(\phi)\approx(\nabla_\phi F)^{\mathsf T}\Delta\phi

This dot product predicts the first-order change in free energy. Making it as negative as possible gives the greatest local decrease.

\frac12\Delta\phi^{\mathsf T}I(\phi)\Delta\phi\approx D_{\mathrm{KL}}(q_\phi\|q_{\phi+\Delta\phi})

This quadratic form predicts how much the entire belief distribution changes under the candidate step.

\frac12\Delta\phi^{\mathsf T}I(\phi)\Delta\phi\leq\varepsilon

The constraint rejects steps whose predicted belief change exceeds the small budget ε. Unless the gradient is already zero, the best step uses the available budget.

\Delta\phi\propto-I(\phi)^{-1}\nabla_\phi F

Solving the comparison yields this direction: the inverse Fisher transforms the ordinary gradient before its sign is reversed to point downhill.

The goal is therefore not to find the step that changes the beliefs as little as possible. A zero step would win that trivial contest while learning nothing. The goal is to obtain the greatest predicted reduction in free energy for a fixed small budget of belief change.

The natural gradient is the transformed gradient $\widetilde{\nabla}_\phi F=I(\phi)^{-1}\nabla_\phi F$ . The inverse Fisher first transforms the ordinary gradient; the learning rate $\rho$ then scales that transformed result. The update can therefore be written in two equivalent steps:

NATURAL-GRADIENT STEP

\widetilde{\nabla}_\phi F=I(\phi)^{-1}\nabla_\phi F,\qquad \Delta\phi=-\rho\widetilde{\nabla}_\phi F=-\rho I(\phi)^{-1}\nabla_\phi F

With one parameter, $I(\phi)^{-1}$ is a single positive number. It can only rescale the ordinary gradient: large Fisher information means that a small coordinate movement would cause a large belief change, so its small inverse suppresses the step; small Fisher information has a larger inverse and permits a larger coordinate movement. In one dimension it cannot steer left or right independently of the gradient.

With several parameters, $I(\phi)^{-1}$ is a matrix. It can reweight and mix the gradient components, so the natural gradient can point in a different coordinate direction from the ordinary gradient. Imagine ordinary gradient descent as a skater dropping straight down a ramp according to the ramp's coordinate grid. Fisher geometry recognizes that equal movements in different directions on that grid may produce very unequal changes in the represented beliefs. The natural gradient redirects the local descent toward the greatest drop in free energy allowed by the belief-change budget. Because the gradient and Fisher matrix are recalculated along the way, successive steps may trace a curve across the ramp rather than the ordinary straight-down path. The path can approach a minimum from a different side; the geometry changes the route, not the objective being minimized.

The three-state categorical example

Consider one observation $o$ and one hidden state with three possible values. Write the approximate posterior as $q=(q_1,q_2,q_3)$ . The likelihood times the prior is simply the joint probability $p(o,s)=p(o\mid s)p(s)$ , so its variational free energy is:

THE SAME FREE ENERGY USED BY THE TOY MODEL

F[q]=\sum_s q_s\!\left(\ln q_s-\ln p(o\mid s)-\ln p(s)\right)=\sum_s q_s\!\left(\ln q_s-\ln p(o,s)\right)

Differentiating directly with respect to each probability gives $g_s=\partial F/\partial q_s=\ln q_s+1-\ln p(o,s)$ . An ordinary additive step proposes $v=q-\rho g$ . Because this calculation treats the probabilities as unrestricted coordinates, its proposal need not remain a valid probability distribution: an entry may become negative, or the entries may no longer sum to one.

Aside

Three ways to keep a probability update on the simplex

The probability simplex is the set of valid probability vectors: $\Delta=\{q:q_i\geq0\text{ for every }i,\ \sum_iq_i=1\}$ . Begin with $q=(0.05,0.35,0.60)$ , gradient $g=(0.7,-0.4,-0.3)$ , and learning rate $\rho=0.1$ . The ordinary additive proposal is:

THE UNCONSTRAINED PROPOSAL

v=q-\rho g=(0.05,0.35,0.60)-0.1(0.7,-0.4,-0.3)=(-0.02,0.39,0.63)

The first proposed probability is $-0.02$ , so $v$ lies outside the simplex. The following methods answer the same practical question in different ways: how should this proposed movement be converted into a valid probability distribution?

1. Clip to zero, then renormalize

First replace each negative entry by zero. The surviving entries then sum to $1.02$ , so divide every entry by $1.02$ to restore a total of one:

v=(-0.02,0.39,0.63)\longrightarrow v_{\mathrm{clipped}}=(0,0.39,0.63)

The impossible negative probability is replaced by zero.

q_{\mathrm{new}}=\frac{v_{\mathrm{clipped}}}{\sum_i(v_{\mathrm{clipped}})_i}=\frac{(0,0.39,0.63)}{1.02}\approx(0,0.382,0.618)

Dividing by the new total rescales the surviving entries so they sum to one.

This is the repair used by the toy implementation. It is simple, but the clipping changes the proposed direction, and an exact zero is awkward when later calculations contain $\ln q_i$ . Adding a tiny positive constant avoids evaluating $\ln0$ , but does not change the fact that this is an after-the-step repair.

2. Project to the nearest valid probability vector

Euclidean projection searches the entire simplex and chooses the valid vector closest to the invalid proposal $v$ . “Closest” means the smallest sum of squared coordinate differences:

EUCLIDEAN PROJECTION ONTO THE SIMPLEX

q^*=\underset{q\in\Delta}{\operatorname{argmin}}\;\frac12\lVert q-v\rVert^2=\underset{q\in\Delta}{\operatorname{argmin}}\;\frac12\sum_i(q_i-v_i)^2

Here $q$ ranges over candidate valid distributions. The expression after $\operatorname{argmin}$ assigns each candidate a squared distance from $v$ . Unlike $\min$ , which returns the smallest distance, $\operatorname{argmin}$ returns the candidate $q$ at which that smallest distance occurs. The star labels the selected solution: $q^*$ is the whole projected vector, while $q_i^*$ is its $i$ th entry. Read the equation as: “Among every valid $q$ in $\Delta$ , return the $q$ that makes $\tfrac12\lVert q-v\rVert^2$ smallest.”

The solution subtracts one shared threshold $\tau$ from the entries that can remain positive and sets the rest to zero. The threshold is chosen so the result sums to one:

THE PROJECTION RULE

q_i^*=\max(v_i-\tau,0),\qquad\sum_iq_i^*=1

For $v=(-0.02,0.39,0.63)$ , the first entry remains zero. The other two exceed a total of one by $0.02$ , so the closest solution subtracts $\tau=0.01$ from each of them:

q^*=\bigl(0,0.39-0.01,0.63-0.01\bigr)=(0,0.38,0.62)

Projection therefore changes the coordinates additively by the shared threshold. Clip-and-renormalize instead rescales the surviving coordinates multiplicatively, which is why it produced the slightly different vector $(0,0.382,0.618)$ .

3. Use an exponentiated-gradient update

Exponentiated gradient avoids leaving the simplex rather than repairing an invalid additive proposal. It multiplies each old probability by a positive factor $e^{-\rho g_i}$ and then normalizes:

EXPONENTIATED-GRADIENT UPDATE

q_i^{\mathrm{new}}=\frac{q_i e^{-\rho g_i}}{\sum_jq_j e^{-\rho g_j}}=\operatorname{softmax}(\ln q-\rho g)_i

w_1=0.05e^{-0.07}\approx0.0466

The positive first gradient decreases the first category’s weight.

w_2=0.35e^{0.04}\approx0.3643

The negative second gradient increases the second category’s weight.

w_3=0.60e^{0.03}\approx0.6183

The negative third gradient increases the third category’s weight.

q_{\mathrm{new}}=\frac{(w_1,w_2,w_3)}{w_1+w_2+w_3}\approx(0.045,0.354,0.601)

Every weight is positive; normalizing them returns a valid probability vector.

In log-probability coordinates, multiplication becomes addition: $\ln q_i^{\mathrm{new}}=\ln q_i-\rho g_i-\ln Z$ . This is why the same update can be written as a softmax. It is also called a multiplicative-weights or entropic mirror-descent update: it limits relative change using KL geometry rather than finding the nearest repair by ordinary Euclidean distance.

Simplex projection: J. Duchi et al. (2008), “Efficient Projections onto the ℓ₁-Ball for Learning in High Dimensions”; exponentiated gradient: J. Kivinen and M. Warmuth (1997), “Exponentiated Gradient versus Gradient Descent”.

The toy model's clip-and-renormalize method is therefore one possible way to keep direct probability updates valid. In the tiny one-factor model it serves as a didactic optimization heuristic, but the repair is not the same as solving the constrained gradient problem and need not converge to the exact posterior. That limitation motivates the projection, exponentiated-gradient, and natural-gradient alternatives.

Natural-gradient variational inference takes another route: represent the same categorical beliefs by unconstrained logits $\eta$ and use the distribution's Fisher geometry to scale their update. Softmax converts those logits into probabilities:

CATEGORICAL BELIEFS IN LOGIT COORDINATES

q_s=\operatorname{softmax}(\eta)_s=\frac{e^{\eta_s}}{\sum_j e^{\eta_j}}

The Fisher matrix for these categorical beliefs follows from the score definition already developed earlier. If the sampled category is $S$ , let $e_S$ denote its one-hot vector: for example, $S=2$ gives $e_S=(0,1,0)$ . Differentiating the categorical log-probability with respect to every logit gives the score vector $e_S-q$ . Averaging its outer product produces:

\ln q_\eta(S)=\eta_S-\ln\sum_j e^{\eta_j}

Softmax assigns the observed category its logit and subtracts the shared log-normalizer.

\frac{\partial}{\partial\eta_j}\ln q_\eta(S)=\mathbf 1\{S=j\}-q_j

The indicator is one for the observed category and zero for every other category; differentiating the normalizer contributes -q_j.

\nabla_\eta\ln q_\eta(S)=e_S-q

Collecting those derivatives gives the categorical score vector.

I(\eta)=\mathbb E[(e_S-q)(e_S-q)^{\mathsf T}]

Fisher information is the expected outer product of the score vector.

I(\eta)=\mathbb E[e_Se_S^{\mathsf T}]-\mathbb E[e_S]q^{\mathsf T}-q\mathbb E[e_S]^{\mathsf T}+qq^{\mathsf T}

Expand the outer product before taking its expectation.

\mathbb E[e_S]=q,\qquad \mathbb E[e_Se_S^{\mathsf T}]=\operatorname{diag}(q)

The mean one-hot vector is the probability vector, while its outer product places one only on the sampled category’s diagonal.

I(\eta)=\operatorname{diag}(q)-qq^{\mathsf T}-qq^{\mathsf T}+qq^{\mathsf T}=\operatorname{diag}(q)-qq^{\mathsf T}

Substitute those two expectations and combine the three outer-product terms.

This matrix has diagonal entries $q_i(1-q_i)$ and off-diagonal entries $-q_iq_j$ . It depends on the current distribution $q$ : as the logits move and the probabilities change, the local Fisher geometry changes with them.

The exact Bayesian posterior is the softmax of the prior log-probability plus the log-likelihood. Define its target logits by $\eta_s^*=\ln p(s)+\ln p(o\mid s)=\ln p(o,s)$ . The formula for the ordinary logit gradient can now be derived rather than introduced as a new identity. Exponentiating and normalizing those target logits gives Bayes' rule directly:

TARGET LOGITS PRODUCE THE EXACT POSTERIOR

\operatorname{softmax}(\eta^*)_s=\frac{e^{\ln p(o,s)}}{\sum_j e^{\ln p(o,j)}}=\frac{p(o,s)}{p(o)}=p(s\mid o)

The derivative of free energy with respect to the probabilities is $\nabla_qF=\ln q+\mathbf1-\eta^*$ . Next, because $q$ depends on $\eta$ , the chain rule passes this derivative through the softmax Jacobian. That Jacobian is the same matrix $\operatorname{diag}(q)-qq^{\mathsf T}=I(\eta)$ derived above:

\frac{\partial q_i}{\partial\eta_j}=q_i\left(\mathbf1\{i=j\}-q_j\right)

Differentiating softmax shows how moving logit j changes probability i.

\frac{\partial q}{\partial\eta}=\operatorname{diag}(q)-qq^{\mathsf T}=I(\eta)

Collecting those derivatives produces the softmax Jacobian, which is exactly the categorical Fisher matrix derived above.

\nabla_\eta F=\left(\frac{\partial q}{\partial\eta}\right)^{\mathsf T}\nabla_qF

The chain rule converts a derivative with respect to probabilities into one with respect to the logits that produce them.

\nabla_\eta F=I(\eta)\left(\ln q+\mathbf1-\eta^*\right)

Substitute the Fisher matrix for the symmetric softmax Jacobian and the previously derived probability gradient.

\ln q=\eta-c\mathbf1,\qquad c=\ln\sum_j e^{\eta_j}

Every log-probability is its logit minus the same log-normalizer c.

I(\eta)\mathbf1=0\qquad\Longrightarrow\qquad\nabla_\eta F=I(\eta)(\eta-\eta^*)

The Fisher matrix removes terms shared equally by every logit, leaving only the meaningful relative differences.

The last cancellation follows directly from normalization: $\operatorname{diag}(q)\mathbf1=q$ and $qq^{\mathsf T}\mathbf1=q\sum_iq_i=q$ , so their difference is zero. The ordinary logit gradient is therefore:

ORDINARY LOGIT GRADIENT

\nabla_\eta F=I(\eta)(\eta-\eta^*)

Multiplying by the inverse Fisher removes this coordinate-induced scaling. Strictly, the categorical Fisher matrix has one redundant direction because adding the same constant to every logit leaves the softmax unchanged; fixing one reference logit or using a pseudoinverse gives the same probability update. Up to that irrelevant shared constant:

NATURAL LOGIT UPDATE

\widetilde{\nabla}_\eta F=I(\eta)^{+}\nabla_\eta F=\eta-\eta^*, \qquad \eta_{\mathrm{new}}=(1-\rho)\eta+\rho\eta^*

Ordinary descent in

q

Move, clip, renormalize

The numerical step depends on probability coordinates and may leave the simplex before being repaired.

Natural descent in

\eta

Interpolate toward Bayesian evidence

The softmax always produces a valid distribution, while Fisher scaling measures the step by its effect on beliefs.

Why this example is exceptionally clean

When

\rho=1

, the new logits equal

\eta^*

, so their softmax is exactly

q^*(s)\propto p(o\mid s)p(s)

. A full natural-gradient step therefore reproduces the exact Bayesian update for this single categorical factor. This one-step result comes from the simple categorical exponential-family structure; natural-gradient variational inference does not generally solve every model in one step.

What changes under a mean-field approximation?

For several hidden-state factors, mean field restricts the approximate posterior to a product family:

MEAN-FIELD FAMILY

q_\phi(s_1,\ldots,s_m)=\prod_{i=1}^{m}q_{\phi_i}(s_i)

Because the score of factor $i$ depends only on that factor, score covariances between different factors vanish under the product distribution. The Fisher matrix is therefore block diagonal, with one block for each factor:

PRODUCT-FAMILY FISHER GEOMETRY

I_{\mathrm{MF}}(\phi)=\operatorname{blockdiag}\!\left(I_1(\phi_1),\ldots,I_m(\phi_m)\right)

In a conditionally conjugate exponential-family model, holding every other factor fixed produces an analytic target for factor $i$ . In categorical logit coordinates it has the form:

ONE MEAN-FIELD COORDINATE UPDATE

\eta_i^*(s_i)=\mathbb E_{q_{-i}}\!\left[\ln p(o,s_1,\ldots,s_m)\right]+\text{constant}, \qquad \eta_i\leftarrow(1-\rho)\eta_i+\rho\eta_i^*

With $\rho=1$ , this is the familiar closed-form coordinate-ascent mean-field update. It can also be read as a full natural-gradient step for that factor. Thus conjugate mean-field coordinate ascent already uses the factor's Fisher geometry implicitly; damping with $0<\rho<1$ takes only part of that natural step.

What the product geometry can—and cannot—do

The block-diagonal Fisher matrix makes each retained factor update statistically well scaled. It cannot restore correlations excluded by

q(s_1,\ldots,s_m)=\prod_iq_i(s_i)

. Capturing those dependencies requires a full joint posterior or a structured mean-field family. In nonconjugate, stochastic, or amortized inference, the analytic coordinate target may also disappear; then the natural gradient must be computed or approximated explicitly.

Natural gradients and variational inference: S. Amari (1998), “Natural Gradient Works Efficiently in Learning”; M. Sato (2001), “Online Model Selection Based on the MDL Principle”; M. Hoffman et al. (2013), “Stochastic Variational Inference”.