PCA meets RG
Describe the same measurements with fewer principal components, and watch what happens to their statistics.
What the procedure does
Imagine recording ten sensors at the same time, over and over. Each moment gives ten numbers: one measured snapshot of the system. Together, those snapshots form a recording. The sensors could measure temperatures, motion, or neural activity.
We want to see which statistical features survive when we describe that recording with progressively less detail. Here is the procedure:
Find principal directions. PCA finds directions describing how sensor values vary together. One might describe all ten rising together; another might describe some rising while others fall. These directions are unit eigenvectors of the covariance matrix, which summarizes how the variables vary together. Think of each eigenvector as a fixed pattern of variation. Its eigenvalue measures the variance along that direction.
Express each snapshot in this eigenvector basis. Subtract the mean snapshot, then calculate its coordinate along each eigenvector. These coordinates are principal-component scores. Multiplying each eigenvector by its score and adding the contributions reconstructs the centered snapshot. Restoring the means recovers the recorded measurements.
Discard the lowest-variance component and reconstruct again. Omit the contribution associated with the smallest eigenvalue at every recorded moment. This reduction of detail is called coarse-graining. A retained component consists of its eigenvector and its scores across the recording.
Compare the activity distributions at each stage. Follow the same reconstructed sensor across time. Subtract its mean and divide by its standard deviation at that stage, giving fluctuations with mean zero and variance one. A histogram estimates their probability distribution. This rescaling lets us compare shape on a common fluctuation scale.
Follow how those distributions change. As we retain fewer components, the rescaled activity distributions may approach a Gaussian bell shape, approach a different shape, or keep changing over the stages available.
A statistical description that reproduces itself under the chosen simplification and rescaling is called a fixed point. Approximately overlapping histograms suggest that the displayed shapes are becoming stable. A stable non-Gaussian shape is interesting, but by itself does not establish fractal structure or criticality. A Gaussian shape alone does not classify the system as uninteresting either.
The connection to RG, a method from physics, is this repeated removal of detail followed by rescaling and statistical comparison. PCA supplies the eigenvectors and scores whose contributions we remove. The central question is: as we retain fewer principal components, what happens to the statistical behavior of the reconstructed measurements?
Find covariance eigenvectors
Let’s use just two neurons and four equal time intervals. Each tick below is one spike. Counting the ticks in each interval gives the table directly underneath.
| Neuron | Interval 1 | Interval 2 | Interval 3 | Interval 4 |
|---|---|---|---|---|
| Neuron A | 5 | 4 | 2 | 1 |
| Neuron B | 4 | 5 | 1 | 2 |
Both neurons average 3 spikes per interval. They mostly rise and fall together, with a smaller difference between them. PCA calculates their covariance matrix and finds its eigenvectors: directions that the matrix scales without changing their direction. For this recording, the two unit eigenvectors describe shared movement and opposite movement:
Eigenvector 1: move together
Eigenvector 2: move oppositely
Each covariance eigenvector has one fixed coefficient per neuron. The first assigns equal positive coefficients to A and B. A positive score along that eigenvector raises both above their averages; a negative score lowers both. The second has opposite signs, so it raises one while lowering the other.
Each snapshot contains two numbers, so two eigenvector directions at right angles—perpendicular directions—are enough to describe it. Each eigenvector includes both neurons. The eigenvalues, 4.5 and 0.5, are the variances of the corresponding scores over all four intervals; dividing by their sum, 5, gives the 90% and 10% shares above.
Each score is the centered snapshot’s coordinate along an eigenvector, a principal direction. Taken together, the scores express a change of basis from neuron coordinates to principal-component coordinates. Those scores change with time; the pictured coefficients stay fixed. Together, the eigenvectors and their scores describe the recording.
Write each coefficient list vertically as a column vector, with neuron A on top and neuron B below. Call the two eigenvectors u₁ and u₂:
Eigenvector 1: move together
Eigenvector 2: move oppositely
1/√2 ≈ 0.707 multiplies both entries inside the parentheses, making the vector’s length one. ≈ means “approximately equal”: the decimals are rounded. The parentheses hold the two neuron coefficients. The score along the eigenvector at a particular time is separate from this fixed vector.
Where do (1, 1), (1, −1), and 1/√2 come from?
We want the direction in which this recording varies most. First subtract each neuron’s average, 3, so we describe changes around its usual activity:
To quantify shared movement, first measure whether the neurons tend to be above or below their averages together. Multiply their deviations at each matching time: a positive product means they are on the same side of their averages; a negative product means opposite sides. Adding these products is the dot product of the two centered recording vectors. Here it is 2 + 2 + 2 + 2 = 8. Dividing by the four intervals gives the average product, 2, called their covariance. Applying the same operation to a neuron’s own recording averages its squared deviations, giving its variance:
Put these results in a table called the covariance matrix, C. Its rows and columns are ordered A, B. The diagonal holds the two variances; the other entries hold their covariance.
We now want to choose the first eigenvector: which fixed pair of coefficients gives scores that vary most across this recording? Equal positive coefficients combine the neurons’ shared rises and falls; opposite signs measure their difference. To compare these possibilities, leave the coefficients undecided: call neuron A’s coefficient a and neuron B’s coefficient b. Choosing a and b chooses a candidate direction.
For each snapshot, we need the score of that candidate direction. Write A and B for the two centered values at that single time. Multiply them by the candidate direction’s coefficients and add: aA + bB. For the first interval, whose centered values are (2, 1), this gives 2a + b; for the second, (1, 2), it gives a + 2b. This is another dot product—this time between the two coefficients and one snapshot’s two centered values.
We want the comparison to reflect which direction captures more variation. Doubling both coefficients keeps the same direction but doubles every score, so we first put all candidate directions on the same scale: require their length to be one, a² + b² = 1.
To compare how much those scores vary across time, calculate their variance. Their average is zero because A and B were centered and the coefficients stay fixed. Their variance is therefore the average of their squared scores. Squaring makes both positive and negative departures contribute. To calculate that average using the variances and covariance above, expand the square into terms involving A², B², and AB:
We can now reuse the three averages we already calculated. Across the four intervals, A² averages to 2.5, B² to 2.5, and AB to 2. The fixed coefficients stay outside those averages. Substituting these values, then using a² + b² = 1, gives the variance for any candidate direction:
The 2.5 is the same for every candidate, so the direction with the largest score variance is the one with the largest product ab. To find how large ab can be under a² + b² = 1, use the fact that a squared difference cannot be negative:
The largest product occurs when a = b. That gives the direction (1, 1): equal coefficients, moving together. PCA’s next direction must be perpendicular to the first so it describes the remaining variation. For (1, 1), perpendicular coefficients must satisfy a + b = 0. Before adjusting the length, we can choose a = 1 and b = −1 to represent that direction as (1, −1).
Both lists have length √2. Dividing every entry by that length preserves the direction and makes its length one:
That is where 1/√2 comes from. The resulting coefficients have ab = 1/2 for eigenvector 1 and ab = −1/2 for eigenvector 2. The variances of their scores are therefore 2.5 + 2 = 4.5 and 2.5 − 2 = 0.5: 90% and 10% of the total, 5. These variances are the eigenvalues: C u₁ = 4.5 u₁ and C u₂ = 0.5 u₂.
These directions arise from this recording’s equal individual variances and positive covariance. Other recordings generally give different coefficients. Reversing every sign in an eigenvector describes the same direction; its time-dependent scores reverse sign with it.
We now have the two fixed eigenvectors: u₁ describes shared rises and falls, and u₂ describes differences between the neurons. To describe a particular snapshot with them, we still need its score along each eigenvector. How do we get those two scores from the recorded counts? We’ll calculate them for the first snapshot, then use them to rebuild it.
Calculate scores and reconstruct each snapshot
Start with the first snapshot: its counts are (5, 4), with neuron A first. Subtracting each neuron’s average, 3, leaves (2, 1). We want to express those deviations as u₁ multiplied by its score plus u₂ multiplied by its score.
To pick out one eigenvector’s score, multiply each neuron’s deviation by that eigenvector’s coefficient, then add the two products. This is the dot product of the centered snapshot and the eigenvector. It works because the eigenvectors are perpendicular and have length one: an eigenvector’s dot product with itself is one, and with the other eigenvector is zero. Applied to their combined contributions, it therefore returns just the chosen eigenvector’s score.
Score along u₁ in interval 1
The snapshot supplies the deviations; u₁ supplies the fixed coefficients:
| Neuron | Count − average | Eigenvector coefficient |
|---|---|---|
| A | 5 − 3 = 2 | 1/√2 |
| B | 4 − 3 = 1 | 1/√2 |
Multiply each row’s deviation by its coefficient, then add the two rows’ products:
Score along u₂ in interval 1
The snapshot supplies the deviations; u₂ supplies the fixed coefficients:
| Neuron | Count − average | Eigenvector coefficient |
|---|---|---|
| A | 5 − 3 = 2 | 1/√2 |
| B | 4 − 3 = 1 | −1/√2 |
Multiply each row’s deviation by its coefficient, then add the two rows’ products:
Do these scores rebuild the snapshot? Multiply each fixed eigenvector by its score, add the contributions, and restore the averages. Below, we do this for interval 1 and for interval 2, whose counts are (4, 5). The same dot-product calculation on interval 2’s deviations, (1, 2), gives scores 3/√2 and −1/√2.
Interval 1: (5, 4)
Eigenvector 1: score 3/√2
Eigenvector 2: score 1/√2
Add both contributions, then restore the averages:
Interval 2: (4, 5)
Eigenvector 1: score 3/√2
Eigenvector 2: score -1/√2
Add both contributions, then restore the averages:
The score along u₁ is the same in these two snapshots. The score along u₂ changes sign, swapping which neuron is higher. The eigenvectors stay fixed; their scores distinguish the snapshots. Adding both contributions and the averages recovers the counts we actually recorded.
We can now use those same contributions to simplify the recording. What happens to the reconstructed values—and to their fluctuations—when we omit the contribution associated with the smaller eigenvalue at every interval?
Compare the fluctuations at each stage
Keeping both components rebuilt every recorded snapshot. Now we want a simpler description of the same recording. Component 2 has the smaller covariance eigenvalue, 0.5 rather than 4.5, so leave out its contribution at every interval and rebuild using component 1 alone.
For interval 1, the shared-movement contribution was (1.5, 1.5). Adding the averages (3, 3) now gives (4.5, 4.5). The smaller correction (0.5, −0.5) is what previously recovered (5, 4). Apply the same omission to all four intervals:
| Interval | Both components: A, B | Component 1 only: A, B |
|---|---|---|
| 1 | 5, 4 | 4.5, 4.5 |
| 2 | 4, 5 | 4.5, 4.5 |
| 3 | 2, 1 | 1.5, 1.5 |
| 4 | 1, 2 | 1.5, 1.5 |
The shared rise and fall remains; the difference between neurons disappears. Values such as 4.5 are reconstructed approximations to counts. We still have the original four observation times.
What changed statistically? Follow neuron A across those same times. Its average is still 3 at both stages. Subtract that average to see its fluctuations—the departures above and below its usual activity:
| Components retained | Interval 1 | Interval 2 | Interval 3 | Interval 4 |
|---|---|---|---|---|
| 2 | 2 | 1 | -1 | -2 |
| 1 | 1.5 | 1.5 | -1.5 | -1.5 |
To compare how often different fluctuation sizes occur, group the values into equal-width intervals, called bins. A bar’s height is the fraction of recorded times falling in its bin. This gives a histogram, a binned estimate of the probability distribution of A’s reconstructed fluctuations. For example, with both components, A is 2 above its mean once in four intervals; with one component, it is 1.5 above its mean twice.
Both components
Component 1 only
Removing a contribution can change both the size and the shape of the fluctuations. To compare shape on its own, we need a common measure of size. Square each departure, average the squares, and take the square root. This is the standard deviation, written σ. Here it measures reconstructed neuron A’s fluctuations across time; the covariance eigenvalues instead measure the variances of principal-component scores. For A at our two stages:
Divide every departure by its own stage’s standard deviation. For interval 1 this gives 2/√2.5 ≈ 1.265 with both components and 1.5/1.5 = 1 with one. A value of +1 now means “one standard deviation above the mean” at either stage. This rescaling gives both distributions mean zero and variance one:
| Components retained | Interval 1 | Interval 2 | Interval 3 | Interval 4 |
|---|---|---|---|---|
| 2 | 1.265 | 0.632 | -0.632 | -1.265 |
| 1 | 1 | 1 | -1 | -1 |
Both components · rescaled
Component 1 only · rescaled
All plots use bins of width 0.5, centered at multiples of 0.5, and identical vertical scales. Tables give the underlying values; bar positions represent bins. Four observations make these deliberately coarse illustrations.
Even after putting them on the same width scale, the two descriptions differ: the original has smaller and larger departures; the simplified version has one departure size. We have completed one comparison: remove a component → reconstruct → measure fluctuations → rescale → compare shape.
One comparison shows what a single removal does. The next question is whether the shape keeps changing as we remove more components, or begins to settle.
Follow how those shapes change
To see whether a shape settles, we need several successive simplifications. Our two-neuron recording permits only the comparison we just made: two components to one. Removing the last component would leave every reconstructed value at its mean, with zero fluctuation size, so there would be nothing left to rescale.
A recording with many nonzero-variance components lets us repeat the same operation. For example, starting with eight components, we could compare reconstructions retaining eight, four, two, and one. At each stage we use the eigenvectors with the largest eigenvalues and their original scores, then repeat the fluctuation-and-rescaling calculation for the same neuron over the same recorded times.
Reconstruct both versions. Give each its own unit fluctuation width and compare their histograms.
Repeat the comparison. Does another removal change the shape as much as before?
Compare again. Look across the sequence for continuing change or an approximately stable shape.
This is a procedure for a larger recording, not additional measurements from our two-neuron example. Halving is one choice of stages; removing one component at a time is another.
To compare the distributions across these stages, place their rescaled histograms on the same axes. If successive versions increasingly overlap, the measured shape is becoming approximately unchanged by further simplification and rescaling. A statistical description that reproduces itself under that operation is called a fixed point. The sequence of changing descriptions is often called a flow.
The shape it approaches matters. A Gaussian limit would have the familiar smooth bell shape after rescaling. A non-Gaussian limit would retain a different shape—for example, different relative frequencies of large departures. The histograms might also continue changing throughout the stages available to us.
Rescaling has already made the means and variances match. What remains to compare is how observations are distributed within that common width: how many are near the center, how many are farther away, and how those frequencies change across stages. Equal widths alone do not make equal shapes.
A histogram of one neuron summarizes only that neuron’s fluctuations over time. This single-neuron distribution is a marginal distribution; we can repeat the comparison for other neurons. The joint distribution describes the simultaneous fluctuations of all neurons and includes how they occur together. Stable individual histograms are evidence about those displayed distributions, rather than proof that the full joint distribution has reached a fixed point.
This is the connection to renormalization group (RG): remove detail, restore a comparable fluctuation scale, and examine how the statistical description changes. Here PCA supplies the covariance eigenvectors and scores whose contributions we remove. Whether a larger recording approaches a Gaussian shape, another shape, or no stable shape is a question for its measurements; our small example establishes the mechanics of asking it.