DATA_REQUIREMENTS.md §1.6 specifies that E5 should publish the response kernel R̂(q) with "its concentration, day-to-day drift, and cross-model herding coefficient ĥ" — but not how those three are computed. Since the estimator choice determines what the experiment can claim, I worked through the candidates against synthetic cases with known answers before writing any harness code. Posting the result for correction.
The measurement question
Send the same prompt to k models; each returns an allocation over n tickers. What number expresses how much they agree — and does it support the claim in Component 4c that μ_retail = (1−h)μ_idio + h·c_t?
Why the obvious candidates fail
I tested four synthetic configurations. Each row is 3 models × 8 repeats over 4 tickers.
Mean pairwise cosine similarity cannot distinguish agreement from crowding:
| case |
cosine |
entropy(avg) |
| all identical, all-in one name |
1.000 |
0.000 |
| all identical, equal-weight |
1.000 |
1.000 |
Both score perfect agreement. But three models unanimously recommending one stock is a crowding event; three models unanimously recommending an index fund is not. Cosine is blind to what is agreed on.
Normalized entropy of the average fails in the opposite direction — total disagreement (0.613) scores higher than partial agreement (0.355), because averaging three disjoint concentrated bets produces a spread-out mean that is indistinguishable from three genuinely diversified recommendations.
Between-model variance share — the first form I tried, between / (between + within) — is inverted:
| case |
ĥ |
| identical models |
0.233 |
| completely different models |
0.999 |
Between-model variance is high exactly when models disagree. It measures systematic difference, not commonality. Worth recording as a trap: the formula looks principled and gives backwards answers.
Projection onto consensus with a uniform reference fixes the sign but breaks when the consensus is diversified — three models unanimously recommending equal-weight scored ĥ = 0.142, because the denominator collapses when c̄ ≈ u.
Proposal: two statistics, not one
Each failure above comes from compressing agreement and concentration into one number. They are independent axes, and the systemic-risk signal is their interaction.
Agreement — how tightly responses cluster around the consensus c̄ = mean(R):
agreement = 1 − E‖r − c̄‖² / D
Concentration — normalized HHI of the consensus, 0 = equal-weight, 1 = all-in:
concentration = (Σ c̄ᵢ² − 1/n) / (1 − 1/n)
Behavior on the same cases:
| case |
agreement |
concentration |
product |
| identical, all-in one name |
1.000 |
0.905 |
0.905 |
| identical, index fund |
0.999 |
0.000 |
0.000 |
| split 2-1 |
0.804 |
0.360 |
0.289 |
| all different |
0.709 |
0.094 |
0.067 |
Rows 1 and 2 now both register high agreement, and concentration separates them. Crowding risk is high only when both are high — which is the quantity Component 4c needs, since h is the weight on a common component and its market impact depends on how concentrated that component is.
Drift falls out naturally as the day-over-day distance between consensus vectors, ‖c̄_t − c̄_{t−1}‖, measured on a fixed prompt set.
Open problem: normalizing agreement
The scaling above is not yet defensible. I normalized by the simplex diameter (D = 2), which is such a loose bound that everything scores near 1 — the full range across all four cases is only 0.709–1.000, and "all different" should be near 0, not 0.709.
The fix is likely an empirical null rather than a theoretical bound: bootstrap or permute responses across models to get the dispersion expected under independence, then normalize against that. This is the same correction Lakonishok–Shleifer–Vishny apply in the institutional-herding literature (subtracting expected dispersion under independent trading). I'd want to implement and validate that before publishing any ĥ figure.
Questions
-
Does the two-statistic decomposition match your intent for E5, or did you have a single ĥ in mind? If the latter, the product is the natural candidate, but I'd argue the components should be reported separately regardless.
-
Should ĥ be estimated per-query, per-archetype, or pooled? Per-query gives R̂(q) as a proper kernel but needs many repeats per prompt; pooling is cheaper but loses the conditional structure that makes it a kernel.
-
Budget: I'd rather build this as a harness you can run on your own keys than spend on a one-off dataset. Plan is stub-mode + recorded-fixture replay so it's fully testable offline (consistent with the repo's zero-key policy), with a provider interface that's a thin swap to live calls. Does that fit how you want E5 executed?
Happy to open a PR with the metrics module and its test suite once the definitions are settled. #7 (deterministic stub) is a prerequisite — measuring response dispersion needs a baseline whose own variance is zero, otherwise sampling temperature and stub noise are indistinguishable.
DATA_REQUIREMENTS.md§1.6 specifies that E5 should publish the response kernel R̂(q) with "its concentration, day-to-day drift, and cross-model herding coefficient ĥ" — but not how those three are computed. Since the estimator choice determines what the experiment can claim, I worked through the candidates against synthetic cases with known answers before writing any harness code. Posting the result for correction.The measurement question
Send the same prompt to k models; each returns an allocation over n tickers. What number expresses how much they agree — and does it support the claim in Component 4c that μ_retail = (1−h)μ_idio + h·c_t?
Why the obvious candidates fail
I tested four synthetic configurations. Each row is 3 models × 8 repeats over 4 tickers.
Mean pairwise cosine similarity cannot distinguish agreement from crowding:
Both score perfect agreement. But three models unanimously recommending one stock is a crowding event; three models unanimously recommending an index fund is not. Cosine is blind to what is agreed on.
Normalized entropy of the average fails in the opposite direction — total disagreement (0.613) scores higher than partial agreement (0.355), because averaging three disjoint concentrated bets produces a spread-out mean that is indistinguishable from three genuinely diversified recommendations.
Between-model variance share — the first form I tried,
between / (between + within)— is inverted:Between-model variance is high exactly when models disagree. It measures systematic difference, not commonality. Worth recording as a trap: the formula looks principled and gives backwards answers.
Projection onto consensus with a uniform reference fixes the sign but breaks when the consensus is diversified — three models unanimously recommending equal-weight scored ĥ = 0.142, because the denominator collapses when c̄ ≈ u.
Proposal: two statistics, not one
Each failure above comes from compressing agreement and concentration into one number. They are independent axes, and the systemic-risk signal is their interaction.
Agreement — how tightly responses cluster around the consensus c̄ = mean(R):
Concentration — normalized HHI of the consensus, 0 = equal-weight, 1 = all-in:
Behavior on the same cases:
Rows 1 and 2 now both register high agreement, and concentration separates them. Crowding risk is high only when both are high — which is the quantity Component 4c needs, since h is the weight on a common component and its market impact depends on how concentrated that component is.
Drift falls out naturally as the day-over-day distance between consensus vectors, ‖c̄_t − c̄_{t−1}‖, measured on a fixed prompt set.
Open problem: normalizing agreement
The scaling above is not yet defensible. I normalized by the simplex diameter (D = 2), which is such a loose bound that everything scores near 1 — the full range across all four cases is only 0.709–1.000, and "all different" should be near 0, not 0.709.
The fix is likely an empirical null rather than a theoretical bound: bootstrap or permute responses across models to get the dispersion expected under independence, then normalize against that. This is the same correction Lakonishok–Shleifer–Vishny apply in the institutional-herding literature (subtracting expected dispersion under independent trading). I'd want to implement and validate that before publishing any ĥ figure.
Questions
Does the two-statistic decomposition match your intent for E5, or did you have a single ĥ in mind? If the latter, the product is the natural candidate, but I'd argue the components should be reported separately regardless.
Should ĥ be estimated per-query, per-archetype, or pooled? Per-query gives R̂(q) as a proper kernel but needs many repeats per prompt; pooling is cheaper but loses the conditional structure that makes it a kernel.
Budget: I'd rather build this as a harness you can run on your own keys than spend on a one-off dataset. Plan is stub-mode + recorded-fixture replay so it's fully testable offline (consistent with the repo's zero-key policy), with a provider interface that's a thin swap to live calls. Does that fit how you want E5 executed?
Happy to open a PR with the metrics module and its test suite once the definitions are settled. #7 (deterministic stub) is a prerequisite — measuring response dispersion needs a baseline whose own variance is zero, otherwise sampling temperature and stub noise are indistinguishable.