Scale-Free Active Inference
Loading tutorial…
From Pixels to Planning: One Algorithm at Every Scale
Based on Friston et al., “From Pixels to Planning: Scale-Free Active Inference,” Frontiers in Network Physiology, 2025 · doi:10.3389/fnetp.2025.1521963
Prerequisite: From Binomial to Dirichlet
Active inference is a framework where perception, learning, and action are all consequences of one imperative: minimize variational free energy. An agent doesn’t have separate modules for seeing, learning, and deciding — it has one algorithm that does all three by maintaining a generative model of its environment and minimizing the discrepancy between what it expects and what it observes.
The “scale-free” contribution of this paper is showing that this same algorithm can be applied hierarchically, at every level of abstraction, using renormalization group theory from physics. The result: a single mathematical object — the renormalizing generative model — that processes raw pixels at the bottom and discovers plans at the top.
The functional form of belief updating is conserved under renormalization. This means the same equations that update beliefs about pixels also update beliefs about objects, events, and plans. Only the Dirichlet parameters — the learned counts — change scale.
We build this understanding piece by piece: first the single-level POMDP, then the free energy objective, then learning, and finally the hierarchical extension that makes it “scale-free.”
The foundation is a Partially Observed Markov Decision Process. The agent cannot see the true state of the world directly. It receives noisy observations and must infer what’s actually happening.
The generative model is defined by five categorical distributions, each parameterized as a Dirichlet distribution. This Dirichlet parameterization is crucial — it enables learning via simple count accumulation.
Maps hidden states to observations. “If the world is in state s, what would I see?”
How states evolve given actions (paths). Each action/path u indexes a different transition matrix.
Prior belief about what state the world starts in.
Log-preferences over observations. Encodes what the agent wants to see. Observations with higher C values are preferred. This replaces the reward function in classical RL: the agent minimizes divergence from preferred outcomes rather than maximizing scalar reward.
Initial distribution over paths/policies. This is where the agent’s “habits” or prior preferences for certain action sequences live.
Categorical distributions are the natural choice for discrete states and observations. The Dirichlet distribution is the conjugate prior for the categorical: when you observe data from a categorical, the posterior is also Dirichlet — you simply increment the concentration parameters. This makes learning a matter of counting.
Every categorical distribution above is not stored as a fixed probability table. Instead, it is stored as Dirichlet concentration parameters (counts) from which the expected probabilities are derived:
Where a is the vector of concentration parameters and the bar notation a denotes the expected value of the categorical distribution under the Dirichlet. The precision of the Dirichlet (how confident we are) is the sum of all concentrations:
Higher α0 means more concentrated probability mass — we’re more certain about which outcomes are likely. The log-expected-probability uses the digamma function:
where ψ is the digamma function (the derivative of the log gamma function). This appears in the variational message passing equations because we work in log-probability space for numerical stability.
The generative model stores concentration parameters (counts), not probability tables. The probabilities are derived from the counts. This is essential: it means learning is simply incrementing counts, and the model naturally becomes more confident (higher precision) as it accumulates more experience.
The complete generative model factorizes as:
Reading right to left: we have priors over parameters (P(A), P(B), P(D)), a path prior (P(u | C, E)) that depends on both the habit prior E and the preferences C (via expected free energy), state dynamics (P(s | u, B, D)), and the likelihood (P(o | s, A)). The agent must invert this model to go from observations o back to beliefs about states s, paths u, and parameters A, B, D.
Why are paths (u) separate from states (s)? Because states describe where the world is, while paths describe what sequence of actions brought it there. The same state can be reached via different paths, and different paths from the same state lead to different futures.
The agent cannot compute exact Bayesian inference (the posterior P(s | o) is intractable for all but the simplest models). Instead, it maintains an approximate posterior Q(s, u, a) and minimizes the divergence between Q and the true posterior. This approach is called variational inference, and the quantity that measures this divergence is the variational free energy.
This is the expected log-ratio of the approximate posterior to the joint model, under the approximate posterior itself. It can be decomposed in two illuminating ways.
Since the KL divergence is always ≥ 0, free energy is always ≥ −ln P(o | m). This means:
How far the posterior has moved from the prior. Penalizes overfitting — the agent shouldn’t update its beliefs too dramatically from a single observation.
How well the model explains the observation. Higher is better — the model should predict what it actually sees.
Minimizing free energy means finding the sweet spot between accuracy (explain the data well) and complexity (don’t overfit). This is a principled form of Occam’s razor: simpler models that explain the data equally well are preferred because they have lower complexity cost.
The name comes from statistical physics. In thermodynamics, free energy F = U − TS balances internal energy (accuracy) against entropy times temperature (complexity). The variational free energy has exactly the same structure: it balances fitting the data against staying close to the prior. This is not a metaphor — the mathematical structure is identical.
Given an observation ot, the agent updates its beliefs about hidden states. In active inference, this is done by minimizing variational free energy with respect to the approximate posterior Q(s). For the categorical-Dirichlet model, exact updates are available.
When the state space is small enough, we compute the posterior exactly:
This is just Bayes’ rule applied directly. The likelihood A[o, s] is the probability of observing o given state s, derived from the Dirichlet concentration parameters of the A tensor.
In the paper’s full model, belief updating happens via variational message passing on a factor graph. Each factor (A, B, C, D, E) sends messages to the variable nodes (s, u), and the variable nodes combine these messages using a softmax:
Where σ is the softmax function and each μ is a log-message from the corresponding factor:
Path beliefs update similarly:
In log-probability space, Bayes’ rule becomes addition: ln P(s|o) ∝ ln P(o|s) + ln P(s). The messages are in log space, so they add up. The softmax then converts back to a proper probability distribution. This is numerically stable and mathematically exact for the exponential family (which includes categorical-Dirichlet models).
The update equations above are applied iteratively until convergence. Each variable node updates its belief using the latest messages from all connected factors, then the factors recompute their messages. This is guaranteed to converge for the categorical-Dirichlet model because the factor graph has a tree structure (no loops at a single level).
Think of each variable node as a person and each factor as a conversation. After each round of conversations, each person updates their belief by combining what they heard from all their conversations. After enough rounds, everyone converges on a consistent belief. The softmax ensures nobody becomes infinitely certain — all beliefs remain proper probability distributions.
Variational free energy tells the agent how well its model explains current observations. But for action selection, the agent needs to evaluate future consequences. This is where expected free energy (EFE) comes in.
This looks complex, but it decomposes into interpretable pieces.
Seeks observations that reduce uncertainty about parameters. This is curiosity — the agent prefers actions that teach it something.
Penalizes states where observations are noisy or uninformative. This term is positive — it increases G, making actions that lead to ambiguous states less preferred. The agent favors states where observations cleanly indicate the hidden state.
KL divergence between predicted outcomes and preferred outcomes. This is goal-directedness — the agent avoids outcomes it doesn’t want.
The agent selects paths (policies) by converting EFE into a categorical distribution using a softmax with precision parameter α:
The negative sign means paths with lower G get higher probability. The precision α controls how selective the agent is:
Bayesian model averaging. The agent considers multiple paths, weighted by their expected value. More exploratory.
Bayesian model selection. The agent commits to the single best path. More exploitative. (Paper uses α = 512.)
EFE naturally balances exploration and exploitation. Information gain drives the agent to explore uncertain regions of its model. Risk drives it toward preferred outcomes. The balance is automatic — no separate exploration bonus is needed. Early in learning (high uncertainty, high potential information gain), curiosity dominates because there is much to learn. Later (low uncertainty, clear preferences), goal-seeking dominates because information gain diminishes.
Learning in active inference is updating the Dirichlet concentration parameters. Because the Dirichlet is conjugate to the categorical, this reduces to incrementing counts. (For a first-principles derivation of Dirichlet distributions from Bernoulli trials, see the From Binomial to Dirichlet tutorial.)
When the agent observes outcome o in state s, the likelihood concentrations update as:
Where η is the learning rate (typically 1.0). The expected categorical probability is then recomputed by normalizing the column:
The same applies to transition concentrations. When the agent observes a transition from state s to state s′ under action u:
The paper introduces a subtle but important mechanism: the agent doesn’t blindly accumulate counts for every observation. Instead, the count update is weighted by the posterior probability of the path that led to this observation:
Where Δat is the one-hot count increment (ao,s += η for the observed outcome o in state s), and P(u1) is the posterior probability of the path that led to this observation. The update is weighted by path confidence: if the agent is certain about which path it’s on (P(u1) ≈ 1), the full count is added. If uncertain (P(u1) ≈ 0.5), only half the count is added. This prevents learning from uncertain attributions.
The agent only learns (increases mutual information in its model) when the expected free energy decreases. This means: update only when doing so improves the model. This prevents pathological learning from noisy or uninformative observations.
The Dirichlet concentrations are not just bookkeeping — they are a direct encoding of the mutual information between observations and states:
Where the joint P(o, s) is derived from the concentration parameters. As concentrations grow, the mutual information increases — the model becomes more informative about the world.
The Dirichlet precision α0 = Σ ai grows monotonically as the agent accumulates observations. Higher precision means:
Imagine a restaurant with a running average rating. After 5 reviews, one bad review swings the average significantly. After 5000 reviews, one bad review barely moves it. The Dirichlet works the same way — α0 is like the total number of reviews. Early on, every observation matters. Later, the model is robust.
Beyond updating existing parameters, the agent can also prune unnecessary structure using Bayesian model reduction. This compares the evidence for the full model against a reduced model (with some parameters removed):
If ΔF < 0, the reduced model has better evidence — the removed parameter was not just unnecessary but actively harmful (overfitting). This is computed analytically using the ratio of Beta functions.
Parameter learning updates the values of existing concentrations. Structure learning adds or removes entire parameters (states, transitions). Both minimize the same objective (free energy), but at different levels: parameter learning changes the “weights,” structure learning changes the “architecture.”
This is where the paper makes its central contribution. The agent doesn’t just learn at one level — it constructs a hierarchy by grouping fine-grained observations into coarser abstractions. This process is borrowed from the renormalization group in statistical physics.
In statistical physics, renormalization starts with a “blocking” step: group adjacent elements into blocks. The paper applies the same idea to observations:
Group adjacent pixels into blocks (e.g., 2×2 pixel blocks become one “super-pixel”).
Group consecutive timesteps into events (e.g., 2 timesteps become one “event” at the higher level).
Apply SVD to each block. Retain the top k singular vectors. Quantize to discrete bins.
The critical mathematical property: the functional form of belief updating is conserved across the transformation. At every level n:
The equation looks identical at every level. What changes is:
“Scale-free” means the inference algorithm does not depend on which level of the hierarchy it operates at. You can run the exact same code at the pixel level and at the planning level. The only difference is the data (the Dirichlet parameters). This is directly analogous to how the renormalization group in physics shows that the same critical exponents describe phase transitions at every scale.
If a state at level n+1 generates k timesteps at level n (the blocking factor), the temporal scale separation grows exponentially. For k=2 (the paper’s default):
Level N (the top) updates once per macro-event. Level 0 (the bottom) updates every timestep. The ratio grows exponentially with the number of levels. More generally, with blocking factor k at each level, the formula becomes kN−n. The blocking factor can also vary per level, giving non-uniform temporal hierarchies. The key insight is that any blocking factor > 1 creates exponential temporal scale separation.
Level N sends a prior message downward. Level N−1 receives it and updates twice (2 belief iterations). Level N−2 receives from N−1 and updates twice. This cascades down to level 0, where the fastest updates happen. The result: lower levels update faster, receiving context from higher levels as empirical priors.
The full hierarchical model — the Renormalizing Generative Model — stacks POMDP levels. Each level has its own complete set of tensors (A, B, D, E) and its own belief updating. Levels are connected by a simple interface: a state at level n generates the initial conditions for level n−1.
A state sn at level n generates two things at level n−1:
A crucial property for computational efficiency: states at level n never share children at level n−1.
This means the Markov blanket of each state consists only of its parents, children, and co-parents of its children. Since there are no co-parents (children are not shared), states at the same level are conditionally independent given the level below.
Conditional independence means the D and E matrices at each level are small and sparse. Message passing is exact and efficient — we never need to consider interactions between all states at a level simultaneously. Each state’s belief can be updated independently, using only its local Markov blanket.
Every level n has its own fully specified POMDP:
| Component | Level n | Meaning |
|---|---|---|
| State space | Sn = {s1n, ..., sKn} | Discrete hidden states at this scale |
| Observation | Generated from level n−1 via Dn | The coarse-grained summary of the level below |
| Actions/Paths | Un = {u1n, ...} | Possible action sequences at this scale |
| Transitions | Bnu | Sparse matrix per path |
| Likelihood | An | Maps states to observations at this level |
| Inference | Minimize Fn(Qn) | Same algorithm at every level |
Higher-level states generate sequences at lower levels. Those sequences are themselves generated by even lower levels. This creates compositional structure: complex behaviors at the top decompose into simpler sub-behaviors, which decompose into primitive actions, which decompose into motor commands.
A paragraph (top level) is composed of sentences (level below). Each sentence is composed of phrases. Each phrase is composed of words. Each word is composed of letters. The same “grammar” (subject-verb-object) applies at the sentence level regardless of which specific words fill the slots. Similarly, the RGM’s inference algorithm applies at every level regardless of what specific states and transitions exist there.
The hierarchy is not strictly bottom-up. Higher levels send descending messages — top-down predictions about what the lower level should expect next. In the free energy framework, these descending messages act as empirical priors (Feldman & Friston, 2010): expectations set by the level above that constrain inference at the level below.
Ascending messages carry prediction errors (coarse-grained observations from boundary detection). Descending messages carry predictions (what the parent expects the child to observe). The child level blends its own local prediction with the descending prediction, weighted by the relative confidence (precision) of each source. This blending can be formalized as a generalized Product of Experts (gPoE — Hinton, 2002):
where the weights w reflect each source’s precision. The result is a sharpened prediction that incorporates both local evidence and hierarchical context. This is the computational analog of predictive coding in the brain (Friston, 2009), where higher cortical areas send top-down predictions to suppress expected sensory input — only prediction errors propagate upward.
Without descending messages, each level is an independent detector that ignores what the hierarchy above has learned. With descending messages, a confident parent can sharpen its child’s predictions (reducing false positive boundaries) or broaden them (increasing sensitivity when the parent is uncertain). This is the formal mechanism by which “context” influences “perception” in the active inference framework.
The total number of parameters across all levels scales approximately linearly with the data:
This is much smaller than a flat model that represents all states at the base level. The hierarchy achieves compression by discovering shared structure.
How does the agent know when one “event” ends and another begins? This is the segmentation problem, and active inference solves it elegantly: boundaries are where free energy spikes.
During normal processing, the agent’s model predicts observations well. Free energy stays low (the model explains what it sees). When something genuinely new happens — a new type of transition, a new context, a shift in task — the prediction error surges.
Where F is the running average of free energy and θ is a threshold. When F spikes above this threshold, the agent detects a boundary.
The use of free energy as a boundary signal is principled — free energy literally measures how well the model predicts, so spikes indicate regime changes. However, the specific threshold mechanism (F > F + θ) involves an engineering choice: the value of θ and the smoothing constant α must be tuned for the domain. The what to measure (free energy) is principled; the when to trigger (threshold) is a design parameter.
The running average uses exponential smoothing:
Where α is a small smoothing constant (e.g., 0.1). This means the running average is dominated by recent free energy values, not the full history. A spike must exceed the recent baseline, not the all-time average.
When Level 1 detects a boundary, it coarse-grains its buffer into a single observation for Level 2. Level 2 processes this coarser observation with its own POMDP. If Level 2’s free energy spikes, that signals a higher-order boundary — a shift in the task pattern, not just the current action.
Level 1 F-spike: “The agent switched from reading to editing.”
Level 2 F-spike: “The agent switched from the debugging task to the testing task.”
Level 3 F-spike: “The agent started a new project.”
When an F-spike occurs at a level, the model can optionally add a new state to its state space using Bayesian model comparison:
This is how the hierarchy grows: new states are added when the existing model cannot explain the data, and they are only kept if they improve evidence.
The paper does not define what constitutes a “boundary” a priori. Boundaries are detected when free energy spikes — when the model is genuinely surprised. Different data will produce different boundaries. The hierarchy self-organizes around the natural structure of the data.
The hierarchy sends predictions downward (Section 8), and prediction errors upward (Section 9). But how much should a level trust its parent’s predictions versus its own local evidence? This is the problem of precision — and it turns out to be the formal mechanism underlying attention in the brain.
In statistics, the precision of a distribution is the inverse of its variance. A high-precision (low-variance) distribution concentrates its probability mass narrowly — it is “confident.” A low-precision (high-variance) distribution spreads probability broadly — it is “uncertain.”
In the active inference hierarchy, each level has its own precision (how much evidence it has accumulated). The parent’s precision relative to the child’s determines how much influence the descending message should have.
Feldman & Friston (2010) showed that precision modulation can be formalized as a gain on prediction errors. In the context of boundary detection, this translates directly to modulating the F-spike threshold:
The logic is intuitive:
This is the formal basis for the claim that attention is precision optimization (Parr & Friston, 2017). When you “pay attention” to something, you are increasing the precision (gain) on prediction errors from that sensory channel. The hierarchy doesn’t have a separate attention module — precision is the attention mechanism. Higher-level beliefs about relevance directly modulate lower-level sensitivity.
Unconstrained precision ratios are dangerous. A parent with thousands of observations would dominate a newly-initialized child, potentially making it fire on every observation. Conversely, a parent with very few observations could make the child completely unresponsive. Engineering bounds prevent both extremes:
With GAINMIN = 0.5 and GAINMAX = 2.0, the threshold can at most double (very uncertain parent) or halve (very confident parent). These are not coupling parameters — they are safety rails that prevent pathological behavior while the gain naturally emerges from the precision ratio.
When a child level receives descending messages from multiple parents, the precisions are summed before computing the gain:
This means higher-precision sources naturally dominate the aggregate gain. A parent with 1,000 observations contributes more to the child’s gain than a parent with 10 observations — exactly as expected.
At the start of a session, higher levels have no evidence and send no descending messages. Without descending messages, the precision ratio is undefined — the child uses its base threshold, unmodulated. As the hierarchy accumulates evidence, descending messages begin flowing and precision modulation gradually activates. This provides a natural cold-start guard: the system degrades gracefully to unmodulated detection when higher levels haven’t learned yet.
Precision modulation connects three threads from the paper:
The key difference between precision modulation and hand-tuned gain constants is that precision emerges from the data. The Dirichlet total concentration grows as evidence accumulates (Section 6). The precision ratio between parent and child reflects the actual information asymmetry between levels — no human judgment about “how much should level 2 influence level 1” is needed. The only engineering choices are the safety bounds (GAINMIN, GAINMAX), which prevent degenerate behavior at the extremes.
Imagine a meeting where the CEO (parent level) gives strategic direction and engineers (child level) report implementation details. When the CEO is well-informed and confident, engineers pay close attention to strategic guidance (high gain) — even small deviations from the plan are worth flagging. When the CEO is new and uncertain, engineers rely more on their own expertise and only escalate major issues (low gain). The “gain” isn’t set by policy — it emerges naturally from how much evidence each party has.
At the highest levels of the hierarchy, the states represent goals or outcomes, and the paths represent plans. The agent doesn’t switch to a different algorithm for planning — it uses the same free energy minimization, applied to a coarser temporal scale.
Given a desired outcome state s*, the agent can work backward through the transition matrix to find which states lead to s* within a fixed horizon:
“Which states can reach the goal in at most k steps?” The agent then selects the path that maximizes the probability of ending in a goal-reachable state.
At the lowest level, “planning” is just one-step action selection (which button to press). At the highest level, it’s genuine multi-step planning (which sequence of sub-tasks to perform). The mathematical machinery is identical:
EFE selects which motor command to issue next, based on expected sensory consequences.
EFE selects which sub-task to pursue next, based on expected task-level consequences.
The difference is only in what the “states” and “observations” represent, not in how inference is performed.
The paper demonstrates this on Pong and Breakout (Atari games):
Classical reinforcement learning requires a reward function. Active inference replaces this with preferred observations (the C tensor). The agent doesn’t maximize reward — it minimizes the divergence between predicted and preferred outcomes. This is more general: preferences can encode arbitrary desiderata, not just scalar rewards.
| Scale | States represent | Actions represent | Free energy measures |
|---|---|---|---|
| Pixels | Pixel patterns | Local transformations | Perceptual prediction error |
| Objects | Object identities | Object motions | Object tracking error |
| Events | Event types | Event transitions | Event prediction error |
| Plans | Goals | Sub-task sequences | Plan feasibility |
The inference algorithm at every row is identical: observe → update beliefs (minimize F) → predict (compute G) → select path (minimize G) → act → learn (update concentrations).
If the same algorithm works at every scale, then perception, action, learning, and planning are not different cognitive faculties. They are the same process, operating on different data. A perceiving brain and a planning brain are doing the same computation — one on pixel-level states, the other on goal-level states. The renormalization group connects them mathematically.
This tutorial covers the core mathematical framework for discrete state spaces. The full paper includes additional material on Lorenz attractor analysis, birdsong generation, and music composition that demonstrate the same principles in continuous domains.