Revisit the audit weights now that rule reach is measured

[mmcky]

Note

Migrated from QuantEcon/audit.2026-05.style-guide#6, opened by @quantecon-services on 2026-08-25. The audit repository holds the 2026-05 pass as a frozen, published record and is being archived — which locks its issues — so the open questions it raised move here, to the standing ledger that now owns them. The original thread stays readable at its old address, and the published 2026-05 report stays live.

Re-derived against the completed pass

Reach figures from the completed pass's lectures/data/rule_reach.csv. Two rows moved; the rest are unchanged.

Weight	Rule	Lectures	Occurrences	Moved since filing
medium	qe-fig-005 — figures without a name:	273	1,115	—
low	qe-writing-008 — repeated spaces	237	7,122	occurrences, from 7,044
low	qe-fig-001 — unnecessary figsize=	224	892	—
low	qe-fig-008 — missing lw=2	196	1,194	from 216 / 1,382
very high	qe-math-006 — bare align	5	14	—
very high	qe-admon-003 — tick-count mismatch	1	2	—
high	qe-fig-010 — plotly without {only} latex	4	4	—
high	qe-math-007 — manual \tag	2	2	—
high	qe-admon-001, qe-admon-004	0	0	—

The disagreement the issue is about is unchanged. The four widest-reaching rules in the corpus are still weighted medium or low, and the two very-high-weight build-risk rules still reach six lectures between them. qe-writing-008 remains the largest count in the corpus at 7,122 occurrences of something a reader of the published page cannot see.

The caution in the original is now more binding, not less: lectures/data/history.csv carries two periods measured with the same code, so changing the weights breaks comparability with both rows and needs both re-derived, not one.

---

Original text, as filed on 2026-08-25

Context

lectures/spec.md §2 assigns every rule an audit weight — very high / high / medium / low — described as "how heavily a rule influences the score for its category, driven by visibility to readers and frequency of occurrence".

Those weights were assigned before anything was measured. Now that reach is measured over 348 lectures, the frequency half of that justification can be checked. Some of it does not hold.

Where weight and reach disagree

Weight	Rule	Lectures	Occurrences
medium	qe-fig-005 — figures without a name:	273	1,115
low	qe-writing-008 — repeated spaces	237	7,044
low	qe-fig-001 — unnecessary figsize=	224	892
low	qe-fig-008 — missing lw=2	216	1,382
…
very high	qe-math-006 — align inside $$	5	14
very high	qe-admon-003 — tick-count mismatch	1	2
high	qe-fig-010 — plotly without {only} latex	4	4
high	qe-math-007 — manual \tag	2	2
high	qe-admon-001, qe-admon-004	0	0

The four widest-reaching rules in the entire corpus are weighted medium or low. The two very-high-weight build-risk rules reach 6 lectures between them.

This is not straightforwardly wrong

For the build-risk rules it is correct and deliberate: qe-admon-003 reaching one lecture is the point — one occurrence breaks a build, and severity should not scale with rarity. Spec §5 already handles that separately, and it works.

The questionable cases are the wide, low-weight ones. qe-fig-005 reaches 78 % of the corpus and is the single largest convention gap there is; weighting it medium says a missing figure name matters less per instance than a \mathcal{N}, which is defensible, but the aggregate effect is that the corpus's most pervasive gap barely moves any score. Figures averages 6.5 rather than lower largely because its widest rules are weighted down.

The concrete question

The weights currently do two jobs at once — "how much does a reader notice this" and "how often does it happen" — and the second is now redundant, because the scoring function already accounts for frequency through its repetition factor (tools/qestyle_draft.py, repeat_factor). Carrying frequency in the weight as well double-counts it in one direction and, where the weight was set low on a rule that turned out to be pervasive, cancels it in the other.

Proposal: redefine audit weight as reader-visible importance only, and let repeat_factor carry frequency. Then re-examine the four wide low/medium rules on importance alone:

• qe-fig-005 — blocks {numref} cross-referencing, so it has a functional cost, not just a cosmetic one. Plausibly high.
• qe-fig-001, qe-fig-008 — genuinely cosmetic. Low is right.
• qe-writing-008 — invisible in rendered output. Low is right, and arguably it should not affect the score at all.

That last one is worth stating plainly: 7,044 occurrences of a rule that a reader of the published page cannot see is the largest count in the corpus, and it is noise in every aggregate it enters.

Caution

Changing weights changes every score, which breaks comparability with the history.csv rows this pass established. If the weights move, re-derive the previous period under the new weights so the trend stays like-for-like — the pipeline supports that (point qestyle_draft at the 2026-05 worktree), but it has to be done deliberately.

Doing this before the next pass is cheaper than after, since there is currently only one period to re-derive.

Related

• lectures/spec.md §2 (the weight tables), §8.2 (how they enter the score)
• tools/qestyle_draft.py — WEIGHT, WEIGHT_POINTS, repeat_factor, DECAY
• lectures/data/rule_reach.csv — the measured reach
• #3 and #5 — the other two scoring-calibration questions this pass surfaced. All three are cheapest to settle together, in one re-derivation.