Optimize the diff algorithm, part 3

[QZGao]

This PR is a follow-up of the efforts made in #4 and #2. During testing and benchmarking, I found a bunch of edge cases that are worthy of fixing, thus made the following changes:

1. Parse template endings without mistaking them for table endings

   WikiWho previously treated every |} sequence as the end of a table. This incorrectly split template endings such as |}} into two separate } tokens.

   Table endings are now recognized only at the beginning of a line. Inline template endings remain intact as }}.

2. Recover two moved words when an unchanged link provides sufficient context

   Existing moved-text recovery normally requires at least three contiguous content words. This misses sequences such as:

   lived in [[British Columbia]]

   The new rule can retain the two words immediately before the link when:

   • the link is complete, with both [[ and ]];
   • its target contains at least two content tokens;
   • the complete word-and-link sequence occurs exactly once in each revision.

   This rule recovers only the two preceding words. It does not assign the link’s provenance or change how ordinary two-word phrases are matched.

3. Retain template-field content when template-name spacing changes

   A change such as:

   {{singlechart|switzerland|62|...}}

   to:

   {{single chart|switzerland|62|...}}

   changes the structural keys of the template separators. The matcher can now retain field content such as switzerland when:

   • the template names differ only in spacing;
   • the content remains between the corresponding separators;
   • its surrounding token sequence is unique in both revisions;
   • neither token already belongs to another match.

   The templates themselves are not treated as identical; only the independently supported field content is retained.

4. Run the new template-field recovery only when relevant

   The additional search is skipped unless the compared revisions actually contain a template-name spacing change. Candidate windows remain bounded, preserving the normal Stage 2 fast path.

5. Recover moved text across punctuation

   Previously, each recovered token needed to belong to a unique window of consecutive informative tokens. That was too restrictive for genuine moved passages containing punctuation, such as comma-separated names:

   colosio, vizcaíno, marina y cerqueda, ...

   Although the entire passage was distinctive, punctuation divided it into short content runs, preventing words such as marina from retaining their provenance.

   The commit replaces that rule with _unique_moved_run_coverage():

   • It examines windows of 3–10 tokens within an already validated moved run.
   • A window must contain at least three informative tokens.
   • Punctuation may occur between those informative tokens.
   • The complete window must occur exactly once in both the previous and current text.
   • Only token positions covered by such a unique window are recovered.

   This allows punctuation-bearing text to retain provenance while still requiring exact, unique contextual evidence.

6. Strengthen the initial moved-run anchor

   The previous constant:

   WORD_MATCH_MOVE_TOKEN_WINDOW = 4

   was replaced with:

   WORD_MATCH_MOVE_MIN_ANCHOR_INFO_TOKENS = 4

   The initial candidate run must now contain an informative content core of at least four tokens before ordinary moved-run recovery can proceed. The existing balanced-internal-link exception remains available when the entire structured run is unique.

   Rationale: markup should not turn a weak phrase into a strong anchor. For example:

   across the country . <ref name=:0>

   The three common words are not enough evidence merely because reference markup follows them. This stronger run-level gate compensates for the more flexible punctuation-aware coverage rule.

   The design therefore uses two evidence levels:

   • Four informative tokens to validate the moved run itself.
   • Three informative tokens inside a unique local window to decide which positions within that validated run can be recovered.

   Neither rule depends on English stopword lists or any particular language.

7. Compute moved-run coverage once

   Within _recover_moved_word_runs(), the matcher now computes the set of uniquely supported positions once per accepted run and reuses that set while assigning matches.

   Previously, uniqueness was checked separately around every informative token.

   Rationale:

   • Makes the evidence rule explicit at the run level.
   • Avoids repeatedly counting the same subsequences.
   • Ensures punctuation and content tokens are retained only when covered by the same validated context.

   The specialized link-context path remains separate for low-content balanced links.

8. Partially restore a deleted historical sentence

   The old exact-sentence restoration logic could reuse a historical sentence only if none of its token objects had already been matched elsewhere. If one or two generic tokens—such as a comma or of—had been consumed by another match, the entire returning sentence was rejected, even when dozens of its other tokens clearly represented the same deleted sentence.

   The new _can_partially_restore_historical_sentence() permits conservative partial restoration when:

   • At least one historical token identity is already occupied.
   • No more than two identities are occupied.
   • At least 24 historical identities remain available.
   • Available identities outnumber occupied identities by at least 4:1.
   • Every available identity has previously been removed.
   • All available identities share the same most recent removal revision.
   • That removal did not occur in the immediately preceding revision.

   These conditions identify a delayed reinsertion supported by a large, coherent block of historical evidence—not an ordinary edit to the immediately previous sentence.

9. Preserve available identities without duplicating occupied ones

   When the partial-restoration predicate passes, WikiWho constructs the returning sentence position by position:

   • Unoccupied historical tokens retain their existing identities and provenance.
   • Positions whose historical identities are already used elsewhere receive new Word objects.
   • Those new objects receive the current revision as their origin and last-use revision.
   • Token IDs and original_adds are updated normally.
   • The historical sentence is marked as consumed so it cannot be reused again.

   Rationale: preserve the strongly supported majority of the returning sentence without assigning one historical token identity to two current positions. Ambiguous occupied positions are treated as new text rather than stealing provenance from their existing matches.

---

Several test cases and 3 more golden fixtures are added to reflect these changes; 2 existing golden fixtures are regenerated, one existing test case is removed as it is already covered by the new golden fixtures.

[ragesoss]

(Review by Claude Code)

I replayed this branch against master over a corpus of full article
histories to see what it actually does to attribution. Summary: the
moved-run changes are a clear net improvement and I'd like to see them
land, but the PR's own tests don't demonstrate that. One thing I'd
want resolved before merge (a code path that never fires on real data),
and one cheap bit of hardening (a structural invariant the restore path
breaks, currently latent).

Method

17 parity fixtures — full histories, 11 wikis (ar, de, en, es, he, hi,
ja, pt, ru, simple, zh), 86,676 revisions replayed in total.
Each history replayed end-to-end under master and under this branch,
then the final revision's token streams aligned by value with
SequenceMatcher (token counts differ slightly, so positional
comparison is misleading) and origin revisions compared on the aligned
tokens.

Where the two disagree, I classified each token by introduced vs
inherited: does the credited revision's text contain the token's
context window while its parent does not? I deliberately avoided a
"distance to origin" metric — the final text assembles late, so that
measure biases origin late and begs the question.

Six of the largest fixtures (Paris en+fr, Adolf Hitler, Barack Obama,
COVID-19 pandemic, Gaza war) were excluded by a 2 GB history cap, so
the most heavily-edited articles are not represented in these numbers.

Attribution impact

aligned final tokens	610,004
tokens whose origin changed	18,613 (3.05%)

Per-article it ranges from 0% up to 10.2% (Brasil) and 6.0% (القاهرة).

This is the part I'd flag hardest: the two regression goldens in
this PR show essentially no attribution change once you align them
properly — the golden churn is tokenization and paragraph re-ordering,
not o_rev_id. So test_token_authorship_matches_master passing does
not tell you this PR leaves attribution alone. It moves 3% of tokens on
real articles. That's not an argument against the change, but the test
suite currently gives false comfort about its blast radius.

Is the change better or worse?

Better, clearly. On the decisive cases — context-unique token windows
where exactly one of the two versions credited the introducing revision:

Of 5,624 decisive cases:

	credited the introducing revision
master	1,554 (27.6%)
this PR	4,070 (72.4%)

Fourteen articles produced decisive cases. This branch wins on 11 of
them, by wide margins on the high-divergence ones (Brasil 83%,
القاهرة 82%, ירושלים 72%, España 72%, भारत 72%, Albert Einstein 71%);
ties on one (Jesse Owens, 41–41); and loses on two very small samples
(2026 Memphis Classic n=48, simple/Wikipedia n=5) — noise. The remaining
three articles had no divergence at all.

Runtime is neutral: 0.95x–1.33x, median ~1.03x.

split_into_paragraphs — endorsed

I checked this on real wikitext rather than synthetic cases: 6,358
revisions sampled across 12 fixtures. This sample does include the
large articles excluded above (Paris, Adolf Hitler, Barack Obama,
COVID-19 pandemic, Gaza war) — paragraph splitting is per-revision, so
it was cheap to cover them here.

• 315 revisions split differently
• all 315 are merges — the new version stops over-splitting at inline
  |}; there is not a single case where it splits more than master
• 98 revisions have genuine line-initial table closes, all still split
  correctly

So it's a strict improvement with no observed regression. One note: a
line-initial |}} (multi-line template whose last param line is empty)
still gets split mid-token, exactly as on master — the regex requires
line-start but doesn't know whether |} closes a table or a template.
Correctly distinguishing those needs {| depth tracking, and the
tempting (?!\}) lookahead is wrong because it breaks genuine |}}}
(table close immediately followed by template close). This is a
pre-existing gap, not a regression — and it appears 0 times in 6,358
real revisions, so I'd just leave a comment noting the known case rather
than complicate the regex.

Concern 1: _recover_unique_template_field_words never fires

Across the whole corpus — 86,676 revisions, 17 articles, 11 wikis:

template_field_recovery_calls     48,312
template_field_recovery_assigns        0

It is invoked 48k times and makes zero assignments. The only thing
that exercises it is its own unit test.

I validated the probe against that unit test to be sure this isn't a
measurement artifact — the same instrumentation reports assigns=2 on
test_template_field_survives_spacing_change_and_move. So the
measurement is sound; the path is simply dead on real input.

That's ~150 lines plus _raw_context_ngram_sizes,
_pipe_key_changed_only_by_template_spacing,
_has_template_name_spacing_change and three module constants, running
an n-gram index on every word diff for no observed benefit. I'd either
drop it, or add a fixture from a real article where it does something —
if it's targeting a case you hit in practice, that case would make a
much better test than the synthetic row.

Concern 2 (minor): restored sentences never enter sentences_ht

The new partial-restore branch builds a fresh Sentence (sentence_reused)
and puts it in paragraph_curr, but only unmatched_sentences_curr gets
registered in self.sentences_ht. So sentence_reused is reachable from
the revision but invisible to every later hash lookup, while the stale
sentence_prev — whose word list no longer matches what's live — stays in
the table in its place.

Note this differs from the ordinary full-match path, which reuses the
sentence_prev object itself. There, one object with one matched flag
sits in both the table and the paragraph. Here there are two Sentence
objects sharing Word objects but carrying independent matched flags, so
the flag no longer protects the shared words.

Measured on Jesse Owens (6,461 revisions):

	sentences reachable from a revision but absent from sentences_ht
master	0 revisions
this PR	615 revisions

Only 4 restores fire on that article; the resulting sentences then persist
through paragraph carry-forward for hundreds of revisions. Master holds
this invariant at exactly zero, so it's a real structural change, not a
pre-existing quirk.

Registering sentence_reused alongside the unmatched sentences (same
place, same value/splitted clearing) takes the violation count back to
0 with the restore path still firing.

To be clear about severity: I could not make this produce a wrong
answer. I built that fix and replayed the six most restore-heavy
articles with and without it — Jesse Owens, Albert Einstein, España,
日本, Brasil, Москва — and the final attribution is byte-identical:
0 differing origins across 381,716 aligned tokens. So today this is
latent, not a live bug, and I don't consider it a blocker.

It would surface only when a partially-restored sentence is later deleted
and reinserted, at which point the lookup consults the stale
sentence_prev rather than what's actually live. Given the fix is ~8
lines and restores the invariant exactly, I'd still take it as cheap
insurance — but it's your call, and the evidence says nothing is broken
right now.

For what it's worth the path is very rare anyway — 42 firings in 22,964
calls across the corpus — so if the restoration is hard to make correct,
dropping it would cost almost nothing.

Not a defect, but worth knowing

Duplicate token_ids within a single revision's token stream are
pre-existing on master (6,344 of 6,461 revisions on Jesse Owens,
identical counts on both branches). I chased this before realizing it
predates the PR. Flagging it so nobody else does — and because it means
"no duplicate tokens" isn't available as an invariant to test against.

[QZGao]

Indeed. I should add more test cases to the existing suite.

[QZGao]

Now all the changes are covered by the current test cases.

[QZGao]

@ragesoss You can review it now. I don't plan to put the fix to "Yeah Yeah Yeahs"-article issue (https://phabricator.wikimedia.org/T434097) in this PR. When I find the proper fix that does not regress, I'll put it in a separate PR.

[ragesoss]

Review notes — analysis run with Claude Code

How to read this. These findings come from an AI-assisted review session. They are not
to be automatically trusted or blindly acted upon. Each one is an observation that warrants
independent verification; some are likely to be wrong or to miss context the author has.
Read them, check the ones that matter, and use or discard them on your own judgment. The
empirical claims below are stated with the commands that produced them so they can be
rerun rather than taken on faith.

Scope of what was examined. The ~460 lines of code diff across WikiWho/wikiwho.py and
WikiWho/utils.py (the remaining +146k is fixtures); the test suite at the PR head; and a
per-token attribution replay comparing master (38f4117) against the PR head (7de9ae5) on
the repository's own revision fixtures.

Coverage limitation, stated up front. splatoon_3_revisions.json (39.5 MB) and
through_the_looking-glass_revisions.json (28 MB) were not replayed. analyse_article
holds the token graph for every revision resident simultaneously, and replaying those two
exhausted 31 GiB of RAM plus swap on the review machine. All drift numbers below therefore
cover only these six fixtures: adam_himebauch, luis_donaldo_colosio_riojas,
the_tell-tale_brain, one_thing_leads_2_another, american_goldfinch,
nintendo_switch (6.3 MB total, ~70 MB peak RSS for the whole suite). This bounds items 1
and 4 below; item 2's bisect does not depend on the excluded fixtures.

---

1. Nine tokens lose provenance to a later revision, on the citation this PR adds a test for

Comparing final-snapshot origin_rev_id per token, master vs PR head, across the six
fixtures: 73 content tokens gain an older origin, and 9 get a newer one. All 9 are on
Luis Donaldo Colosio Riojas, and they are exactly three template date fields:

token	master	PR head
4 june 2024 (|term_start2=)	#67 (1234010057)	#89 (1347221443) — the snapshot
23 march 2019 (|date=)	#31 (1028090673)	#57 (1222367802)
8 june 2021 (|date=)	#30 (1028074078)	#57 (1222367802)

Each of these exact substrings is present continuously in the wikitext from its first
appearance through the snapshot — verified against the fixture; 8 June 2021 appears in
60/60 consecutive revisions from #30, 23 March 2019 in 59/59 from #31, 4 June 2024 in
24/24 from #66. None is ever deleted and re-added, so a snapshot-or-later origin looks
incorrect.

One qualification that cuts against master as well: 4 June 2024 first appears at rev
#66 (1227424767), while master credits #67. Master is off by one revision there too —
so this is not a case of master being right and the PR head wrong, but of the PR head
moving the origin 22 revisions later, to the snapshot.

Rev 1222367802 is worth looking at closely, because it only reorders the fields of the
citation — |date= moves to the front:

prev (1220166917): {{cite news|url=…|title=Colosio, Vizcaíno, …|publisher=…|date=8 June 2021}}
curr (1222367802): {{cite news |date=8 June 2021 |title=Colosio, Vizcaíno, … |url=… |publisher=…}}

This is the same citation whose |title= the PR adds a passing test for
(colosio_citation_title_move_across_punctuation, asserting origin 1028074078). Within one
template and one revision, the title's provenance is fixed and the date's is lost.

2. Bisected to one line, with a mechanism

Reverting each of the change's four independent parts individually on top of the PR head
isolates it. The reproduction needs only the failing transition — revisions [56:58] of
the fixture, two revisions rather than 90 — which makes this cheap to rerun:

reverted on top of the PR head	the three dates
utils.py paragraph split	still regressed
_can_partially_restore_historical_sentence disabled	still regressed
_recover_unique_template_field_words disabled	still regressed
the _copy_safe_moved_run / coverage rewrite	restored
…of that, only the per-offset gate (_unique_moved_run_coverage → _has_unique_content_window)	still regressed
…of that, only the threshold, 4 → 3	restored, but breaks a PR test — see item 3

The responsible line is in _copy_safe_moved_run:

-    if len(content_core) >= WORD_MATCH_MOVE_MIN_INFO_TOKENS:          # master: 3
+    if len(content_core) >= WORD_MATCH_MOVE_MIN_ANCHOR_INFO_TOKENS:   # PR head: 4

The change hoists weak-anchor rejection up out of the per-offset check into the
copy-safety gate. On master, a 3-token content core passed this gate and was discriminated
downstream in _has_unique_content_window, which carried an explicit exception for
digit-bearing runs of exactly 3:

if any(any(char.isdigit() for char in token) for token in needle):
    return (_count_subsequence_cached(count_text_prev, needle, count_state) == 1 and ...)

Hoisting the gate dropped that exception. Template date fields are its casualty:
| date = 8 June 2021 has _longest_content_core of exactly ('8','june','2021') — three
informative tokens — and contains no [[, so the new _link_anchor_bounds fallback does
not apply either. The whole moved run is rejected before any of its tokens reaches the new
_unique_moved_run_coverage check, and the tokens are recreated at the current revision.

3. The conflict is real — a straight 4 → 3 revert is not the fix

Worth flagging because it is the non-obvious part.
test_reference_markup_does_not_strengthen_a_weak_move_anchor, added by this PR, uses
["across","the","country",".","<","ref","name","=",":","0",">"]. Its longest content core
is across the country — also exactly three informative tokens. Lowering the threshold
back to 3 restores the dates and makes that test fail (assert 16 is None). The digit
heuristic on master is precisely what separated these two cases; the ≥4 threshold cannot
distinguish them.

A candidate patch that satisfies both is below. It keeps the ≥4 gate and adds an OR-branch
for a ≥3 core that is either preceded by a |name= template-field opener or contains a
digit. Neither disjunct suffices alone — the field anchor misses | term_start2 = 4 June 2024 (its moved run does not include the |), and the digit test alone leaves ~15 Adam
Himebauch tokens attributed later than the field anchor achieves. Uniqueness in both prev
and curr is still required, so the existing safety property is unchanged.

This patch was written by Claude Code and has not been reviewed or run by a human.
Treat it as a description of one workable shape for the fix, not as a proposed commit.

def _template_field_anchor(run, content_core):
    """True when content_core sits directly after a `|name=` template field opener."""
    core_length = len(content_core)
    for start in range(len(run) - core_length + 1):
        if tuple(run[start:start + core_length]) != content_core:
            continue
        if start < 3 or run[start - 1] != '=':
            continue
        index = start - 2
        while index >= 0 and _is_informative_move_token(run[index]):
            index -= 1
        if index >= 0 and index < start - 2 and run[index] == '|':
            return True
    return False

and the gate in _copy_safe_moved_run becomes:

 def _copy_safe_moved_run(count_text_prev, count_text_curr, text_curr, curr_start, length, count_state):
     run = text_curr[curr_start:curr_start + length]
     content_core = _longest_content_core(run)
-    if len(content_core) >= WORD_MATCH_MOVE_MIN_ANCHOR_INFO_TOKENS:
+    if (len(content_core) >= WORD_MATCH_MOVE_MIN_ANCHOR_INFO_TOKENS or
+            (len(content_core) >= WORD_MATCH_MOVE_MIN_INFO_TOKENS and
+             (_template_field_anchor(run, content_core) or
+              any(char.isdigit() for token in content_core for char in token)))):
         return (_count_subsequence_cached(count_text_prev, content_core, count_state) == 1 and
                 _count_subsequence_cached(count_text_curr, content_core, count_state) == 1)

With that applied, all nine tokens return to master's origins, and per-token drift over the
six fixtures becomes 100 content tokens older, 0 newer (versus 73 older / 9 newer at the
PR head) — so it also recovers 27 further improvements rather than merely undoing the
regression. Every hand-written assertion in tests/ passes, including the weak-anchor test.

4. The golden fixtures cannot detect a regression in this PR

*_golden.json were regenerated from the PR head, so they encode its output exactly. Built
from each version's ww.tokens using the same fields test_token_authorship_matches_golden
compares:

fixture	master	PR head	with the item-3 patch
adam_himebauch	8341 tokens, mismatch	exact match (8336)	8285, mismatch
luis_donaldo_colosio_riojas	6886, mismatch	exact match (6908)	6859, mismatch
the_tell-tale_brain	5019, mismatch	exact match (4980)	exact match
one_thing_leads_2_another	1621, mismatch	exact match (1616)	exact match

Because the goldens match the PR head byte-for-byte, they will accept whatever the PR head
does, including item 1. Note the direction of the patch's mismatches: it creates fewer
tokens (Colosio 6908 → 6859), i.e. it retains more identities rather than recreating them —
which is the stated goal of the series, but registers as a golden failure. Per-token origin
direction (older vs. newer) discriminates here where the goldens do not.

Separately, five revision fixtures are present but unused by ARTICLES:
through_the_looking-glass, japan_cup, american_goldfinch, nintendo_switch,
2026_canvas_security_incident. Generating goldens for them would widen coverage; note
that the first of those is where most of the PR's newer-origin drift appeared in the
earlier, RAM-limited pass, and it has no golden today.

5. Other observations from reading the diff

Items (a)–(e) were checked by direct execution against the PR head; (f) is a code-reading
observation only.

(a) WORD_MATCH_HISTORICAL_MIN_EVIDENCE_RATIO is unreachable. With MIN_AVAILABLE=24
and MAX_OCCUPIED=2, len(available) < 4 * occupied_count cannot be true once the first
two gates pass — brute-forcing the reachable space returns no qualifying pairs. The
docstring lists "outnumber by at least 4:1" as an independent criterion, but it is implied.
Either drop it or lower MIN_AVAILABLE.

(b) _has_template_name_spacing_change also matches link pipes. The filter is
key[:2] == ('wikitext', '|') with no constraint on key[2], so
('wikitext','|','link', ('British','Columbia'), 0, ('BC',)) qualifies. Confirmed:
[[British Columbia|BC]] → [[BritishColumbia|BC]] returns True.
_pipe_key_changed_only_by_template_spacing has the same gap. Constraining key[2] to the
template-field key kinds would close both.

(c) The same gate fires when nothing changed. It tests the cross-product of name forms,
so a revision pair where both sides contain both spacing variants returns True.
Confirmed: "{{single chart|switzerland|62}} and {{singlechart|austria|7}}" compared against
itself returns True. Comparing per-name form sets (prev_forms[c] != curr_forms[c]) would
avoid this.

(d) Cost when it fires. _recover_unique_template_field_words indexes n-grams over the
whole token lists (prev_spans = [(0, len(text_prev))]), ignoring the candidate spans that
keep _recover_moved_word_runs bounded. Measured on synthetic diff regions, with and
without a template-name-spacing change present to trigger the pass:

n=  500 tokens    0.45 ms  ->   2.64 ms
n= 1500 tokens    1.36 ms  ->   8.54 ms
n= 3000 tokens    2.67 ms  ->  17.16 ms     (~6.4x for this stage)

End-to-end the effect was modest in an earlier pass over the full fixture set (~+4% total;
Colosio +52%, japan_cup +20%) — those aggregate figures were not re-measured here, and
japan_cup is outside the six fixtures used above. Items (b) and (c) make this pass fire in
cases where it has nothing to recover.

(e) utils.py: a line-start |}} is still split.
split_into_paragraphs("{{cite web\n|url=x\n|}}\nnext") returns
['{{cite web\n|url=x\n|}', '}\nnext'] — the same bug class the |} fix addresses, when the
template close begins a line. A (?!\}) lookahead would cover it; the only construct that
costs is a table close followed immediately by } on the same line, which is not valid
table syntax. Also, the other patterns in this file are module-level compiled (regex_dot,
regex_url, regex_cjk) while the new re.sub is inline, and this runs per revision.

(f) sentence_reused is never inserted into self.sentences_ht, and its .value /
.splitted are never cleared. Every other retained sentence gets value = '' /
splitted = None after hash-table insertion, specifically to keep pickles small; this one
retains the full sentence string indefinitely. More substantively: since only the original
sentence_prev remains in sentences_ht, a later delete-and-readd of that sentence looks up
a word list whose token objects now live in a different sentence of the current revision.
Worth confirming that is intended rather than incidental.

6. Smaller notes

• Threshold coupling in _recover_moved_word_runs: needs_link_context uses
  < MIN_INFO_TOKENS (3) while _copy_safe_moved_run's gate uses
  >= MIN_ANCHOR_INFO_TOKENS (4). Since _link_anchor_bounds guarantees ≥2 informative
  tokens before [[, the link-window path is reachable only for a content core of exactly
  2; a core of 3 passes copy-safety via the link branch but is then routed through
  _unique_moved_run_coverage. Two constants doing different jobs three lines apart would
  benefit from a comment.
• An undocumented tightening. The old _has_unique_content_window returned True
  unconditionally for non-informative tokens, so punctuation and markup inside an accepted
  run always kept provenance. _unique_moved_run_coverage requires them to fall inside a
  unique window, so edge punctuation can now lose it. The description presents item 5 as
  purely more permissive; in this respect it is stricter.
• _can_partially_restore_historical_sentence builds removal_revisions under
  if word.outbound and then separately evaluates all(word.outbound …); one pass would do.
• Pre-existing, not introduced here: the for matched_word in matched_words_prev: loop
  (~line 1400) uses word_prev, left over from the preceding loop, rather than
  matched_word, so those words never get last_rev_id updated. Worth a separate issue.
• Downstream note for wikiwho_api: the paragraph-split change alters paragraph and
  sentence hashes, so the first revision processed against an existing pickle will not match
  paragraphs containing an inline |} and will fall through to word-level matching. Since
  pickles carry no algorithm-version invalidation, a rebuild rather than an incremental
  upgrade seems safer.

7. What the replay shows the change doing

Stated neutrally, for the six fixtures measured: 73 content tokens move to an older origin
and 9 to a newer one; token counts drop on three of four goldens (more identities retained);
the |} handling is correct wikitext semantics for the inline case; count_state is shared
across both recovery passes, reusing the subsequence cache; and the new tests are
unit-level, each targeting a single rule. test_reference_markup_does_not_strengthen_a_weak_move_anchor
is the negative test that constrains the anchor gate — and, per item 3, the constraint that
makes the fix for item 1 non-trivial.

---

Drafted across two Claude Code sessions (~3.5 h wall clock, 5 human messages). Sage
initiated the review, and redirected its method after the first session exhausted the
machine's memory replaying the two largest fixtures — which is why item 1's reproduction was
narrowed to a two-revision slice. Sage read this comment in full before posting but did not
independently reproduce any finding, and did not run or review the patch in item 3, which
Claude Code wrote. The empirical claims — drift counts, the bisect, the golden comparisons,
and items 5a–5e — were produced by scripts Claude Code wrote and ran in-session, and have not
been checked by a second party; item 5d's end-to-end percentages are carried over from the
first session and were not re-measured. Items 5f and 6 are code-reading observations,
unverified by execution. The two largest fixtures were never replayed at all.

(Comment written by Claude Code.)

[QZGao]

Am looking into this new review. While some are indirectly addressed by #6, I believe there are points that are worthy of immediate fix in #5. Therefore I will push another commit to #5 soon.

[QZGao]

Changelist of 7f8604b:

1. Structure-aware three-token moved runs

   • Allows a unique three-informative-token run to qualify as moved text only when it is explicitly anchored by raw template-field syntax: | field = value.
   • Retains the normal four-token threshold elsewhere.
   • Avoids a broad “contains a digit” exception that could exchange lineages between unrelated citation dates.

   This supports short values such as marina y cerqueda while rejecting weak prose anchors such as across the country.

2. Narrow template-field rename recovery

   Adds a specialized recovery pass for cases such as:

   |term_start=4 June 2024
           ↓
   |termstart2=4 June 2024
   

   The recovery is deliberately restrictive:

   • Exactly three informative value tokens.
   • Value must contain a digit.
   • Value must be unique in both complete revisions.
   • Both occurrences must have template-field structure.
   • Normalized field names must be equivalent after removing underscores and trailing numeric suffixes.
   • Raw field names must actually differ.
   • Existing higher-confidence matches cannot be displaced.
   • Same-name citation fields are excluded.

3. Preserve partially restored sentences for future reuse

   • Tracks sentences reconstructed from partially available historical identities separately.
   • Registers those restored sentences in sentences_ht.
   • Inserts the newly restored representation before stale historical versions.
   • Clears temporary sentence text/split data consistently.

   This prevents a successfully repaired sentence from losing its recovered identities during another later delete-and-reinsert cycle.

[Copilot]

Pull request overview

Improves WikiWho’s token attribution across complex wikitext edits and historical sentence restoration.

Changes:

• Refines moved-text and template-field recovery.
• Adds conservative partial historical-sentence restoration.
• Corrects table-ending parsing and expands regression fixtures.

Reviewed changes

Copilot reviewed 4 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
.gitignore	Ignores macOS metadata.
WikiWho/wikiwho.py	Extends matching and restoration logic.
WikiWho/utils.py	Refines table-ending parsing.
tests/test_regression.py	Adds targeted regression tests.
tests/authorship_cases.json	Adds authorship expectations.
tests/fixtures/one_thing_leads_2_another_revisions.json	Adds revision-history fixture.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[QZGao]

Okay. Those are all resolved. Let's ask Copilot again for review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[QZGao]

Hmmm. It seems that GitHub's "Student Pro" plan no longer has any AI quota... 😓

[Copilot]

Pull request overview

Copilot reviewed 4 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

WikiWho/wikiwho.py:719

• For a validated run, this issues roughly eight distinct subsequence queries per token. Each cache miss scans both full revisions in _count_subsequence, and adjacent windows generally have different keys, so a long moved block is still Θ(run length × revision length) and can reintroduce quadratic runtime on the large articles this matcher is intended to handle. Please pre-index 3–10-token window frequencies once (under the existing window cap) or otherwise bound the coverage scan.

            if (_count_subsequence_cached(count_text_prev, needle, count_state) == 1 and
                    _count_subsequence_cached(count_text_curr, needle, count_state) == 1):

[MusikAnimal]

OK by me, but… I wholly rely on the AI for the reviews of this codebase. I am grateful for the fixes, but I think once we consider things "stable" we should refrain from further non-necessary changes. The algorithm is just too fragile, it seems.

If @ragesoss is happy this I will deploy it

[QZGao]

I think once we consider things "stable" we should refrain from further non-necessary changes.

Indeed. I also don't plan to modify it further as the current #6 edition is the best balance I can find between time and space consumption. Unless there are new reports on correctness cases, we can mark the current edition the "stable version" and call it a day.

[ragesoss]

I will review it this week.

[ragesoss]

Re-review notes — analysis run with Claude Code

How to read this. These findings come from an AI-assisted review session. They are not to
be automatically trusted or blindly acted upon. Each one is an observation that warrants
independent verification; some are likely to be wrong or to miss context the author has. Read
them, check the ones that matter, and use or discard them on your own judgment. The empirical
claims are stated with enough detail to be rerun rather than taken on faith.

Scope. The five commits since the previous review (7de9ae5 → e0cc079), re-verified
against master (38f4117). Every item from the previous review was rechecked by execution
rather than by reading the changelog.

The coverage limitation from both previous reviews is gone, and it was self-inflicted.
splatoon_3 and through_the_looking-glass were excluded before because replaying them
exhausted the review machine's RAM. That diagnosis was wrong. Measured with an RSS watcher at
all three refs, using analyse_article(revs) — the same whole-list call the test suite makes:

	master 38f4117	7de9ae5	e0cc079
splatoon_3 (41 MB, 1078 revs)	0.13 GB / 2.5 s	0.13 GB / 2.6 s	0.18 GB / 2.6 s

through_the_looking-glass peaks at 0.13 GB, and the complete suite over all ten fixtures
peaks at 0.38 GB in 9.8 s. So this PR does not change the memory profile, and neither fixture
was ever expensive; the earlier exhaustion came from loading multi-gigabyte parity histories
into a Python list (parsed dicts cost ~2.1× the on-disk JSON), not from analyse_article. Both
articles are included in the numbers below, and both change the picture materially.

---

Status of the previous review's items

#	item	status
1–3	three template dates lose provenance on Colosio	fixed, verified two ways
4	goldens cannot detect a regression in this PR	goldens unchanged in kind; authorship_cases.json now supplies the missing oracle
5a	WORD_MATCH_HISTORICAL_MIN_EVIDENCE_RATIO unreachable	not addressed
5b	template-spacing gate also matched link pipes	fixed, test added
5c	same gate fires when nothing changed	not addressed
5d	cost when it fires	shape unchanged; frequency now measured (below)
5e	line-start |}} split mid-token	fixed; the construct the earlier comment worried about does not occur in 71 k revisions
5f	restored sentences never entered sentences_ht	fixed, verified; invariant back to exactly 0
6	threshold coupling undocumented	comment + rename added
6	edge punctuation can now lose provenance	not addressed
6	matched_words_prev loop uses word_prev	still present on both branches; pre-existing, separate issue

Items 1–3. On the two-revision slice (revs[56:58], the field-reordering edit 1222367802)
8 June 2021 and 23 March 2019 are back to their master origins. Over the full 90-revision
history, 4 June 2024 now resolves to #66 — one revision earlier than master's #67, so
the off-by-one the previous review noted against master is also gone. The remaining
direction-changed token on that article, 2024 in | term_start = 1 September 2024, moves
from master's #66 to #67; 1 September 2024 first appears at #67, so the new value is the
correct one and master was carrying a 2024 token over from the previous 4 June 2024.

Item 4. The goldens are still regenerated from the head — all four available goldens are an
exact match at e0cc079 and a mismatch on master — so they remain a self-consistency check,
not an independent oracle. But authorship_cases.json now is one, and it is the more useful
artifact: it runs against master unchanged (it imports only the public API), where 4 of its
28 cases fail — Tell-Tale Brain's restore case, two Colosio cases, and the One Thing Leads 2
Another template-spacing case. More to the point, the three new Colosio cases fail at
7de9ae5 and pass at e0cc079, so the suite would now catch a reintroduction of exactly the
reported regression. (test_regression.py cannot run against master at all — it imports
private helpers that do not exist there — so authorship_cases.json is the only part of the
suite usable as a cross-branch oracle. Worth preserving that property.)
The five revision fixtures that had no golden are now covered by hand-written cases too.

Item 5f. Measuring sentence objects reachable from a revision but absent from
sentences_ht:

	master	7de9ae5	e0cc079
Jesse Owens (6,461 revs)	0	615	0
The Tell-Tale Brain	0	48	0
eight other articles	0	0	0

The invariant is restored exactly.

Item 5e. The (?!\}) lookahead was measured against real wikitext — 71,057 revisions
across 10 repository fixtures and 16 parity histories in 11 languages:

line-start form	occurrences	effect of the change
|} not followed by }	230,416	unchanged — still splits
|}}	11,360	no longer split mid-token (the fix)
|}}} or longer	0	—

Every sampled |}} is a template close with no table open (typically an infobox whose last
parameter line is empty). This corrects a claim in the earlier review comment, which
asserted the lookahead was wrong because it would break a genuine |}}} table-close-plus-
template-close: that construct does not appear anywhere in this corpus, while the case the
lookahead fixes appears 11,360 times. The earlier "0 occurrences in 6,358 revisions" figure for
|}} was simply a corpus too small and too English to contain it — it is common in he, ru, hi
and es.

Items 5a and 5c reproduce unchanged. Brute-forcing the reachable
(available, occupied) space still yields no pair where the ratio gate is the deciding one.
_has_template_name_spacing_change(keys, keys) still returns True when compared against
itself for "{{single chart|switzerland|62}} and {{singlechart|austria|7}}", because the gate
compares the cross-product of name forms rather than per-name form sets — so a revision pair
where both sides contain both spellings enters the expensive path with nothing to recover.

---

What the change does to attribution

Per-token origin on the final revision, master vs e0cc079, adjudicated by
introduced-vs-inherited: for each token whose origin differs, does the credited revision's
token stream contain the token's 7-token context window while its parent's does not? A revision
that introduces the window is a defensible origin; one that merely inherits it is not. This
deliberately avoids a distance-to-origin metric, which biases origin late and begs the question.

fixture	decisive	master	this PR	PR share	ambiguous
adam_himebauch	35	11	24	68.6 %	3
luis_donaldo_colosio_riojas	15	4	11	73.3 %	5
one_thing_leads_2_another	6	4	2	33.3 %	2
splatoon_3	96	30	66	68.8 %	41
through_the_looking-glass	130	48	82	63.1 %	100
american_goldfinch, nintendo_switch, the_tell-tale_brain	0	—	—	—	32
total	282	97 (34.4 %)	185 (65.6 %)		183

Net improvement, consistent with the 72.4 % measured over the wider 17-article corpus in the
first review comment. Two things are worth saying plainly about it:

The raw directional metric flatters the PR. Counting only whether origin moved older or
newer gives 341 older / 124 newer over these eight fixtures — and on the six small fixtures
alone it gives 97 older / 1 newer, which reads like a clean sweep. It is not: of the 97
decisive cases master wins, most are cases where this PR moved origin older to a revision
that does not contain the token's context. Older is not automatically better. The adjudicated
column is the honest one.

The losing cases cluster on repeated citation dates. Of the sampled master-wins, 15 of 26
have a context window inside a template field, mostly | date = … / | archive-date = … in
{{cite}} templates on Splatoon 3, where several citations carry dates a few days apart. The
commit message for 7f8604b explicitly names this hazard as the reason a bare digit-bearing
exception was rejected — the hazard appears to be present anyway, via some other path, at
roughly 30 tokens on that one article. Worth a look, though on the same fixture the PR still
wins 66–30 overall.

---

New observations

(a) The three-token structural exception has an arbitrary boundary. _template_field_anchor
walks backwards from the = over informative tokens only, and every non-alphanumeric token
stops it. Since split_into_tokens breaks term_start into term, _, start, any field name
containing _ or - fails the check. Recovery of a moved three-token date value, same shape as
the PR's own test_validated_move_recovers_tokens_across_punctuation:

|date=              YES        |term_start=        no
|title=             YES        |birth_date=        no
|df=                YES        |access-date=       no
|publication date=  YES        |archive-date=      no
                               |death_date=        no

This is not a regression — master recovers none of the nine shapes — but it excludes the
most common infobox and citation field spellings on enwiki while admitting |df=. Note also
that the sibling helper added in the same commit, _template_field_before_content, explicitly
allows _ (or tokens[index] == '_') but not -, so the two helpers draw three different
boundaries between them.

Worth flagging, since _template_field_anchor was taken essentially verbatim from the previous
review comment: that code was written by Claude Code and was labelled there as never having
been run or reviewed by a human. It is now in the PR. It does what the comment claimed for the
case it was aimed at, but its edges were not thought through, and the review comment should not
have been treated as a source of committable code.

(b) The two template-field recovery passes almost never fire. Instrumented over 22,586
revisions (16 articles, 8 wikis), counting assignments actually made:

pass	calls	calls that assigned	assignments	where
_recover_unique_template_field_words	11,212	1	9	one_thing_leads_2_another only
_recover_unique_short_numeric_template_fields	11,212	2	5	luis_donaldo_colosio_riojas only

This is an improvement on the previous measurement (0 assignments in 86,676 revisions) — the
first pass is no longer dead — but ~300 lines of specialised machinery plus five module
constants, invoked 11,212 times, produce 14 assignments, all on the two articles that motivated
them and none on the other 14 including a 6,461-revision article. That may be the intended
cost/benefit for correctness work; it is worth deciding deliberately rather than by accretion.

(c) A silent-disable guard. _recover_unique_short_numeric_template_fields opens with

run_length = WORD_MATCH_MOVE_MIN_INFO_TOKENS
if run_length != 3:
    return

Today that is a no-op. If WORD_MATCH_MOVE_MIN_INFO_TOKENS is ever changed, the entire pass
turns itself off with no test failing and no warning. An assertion or an explicit 3 would
fail loudly instead.

(d) _has_unique_link_move_window is now strictly broader, at a cost. Replacing the sparse
(10, 8, 6, 4, 3) sizes with every size from the run length down to
WORD_MATCH_MOVE_MIN_LINK_WINDOW = 6 adds sizes 9 and 7 and drops 4 and 3 — and 4 and 3 could
never have qualified, since _link_anchor_bounds needs at least two tokens before [[, two
inside, and both brackets, i.e. six. So nothing is lost and sizes 9 and 7 are gained, which
matches the Copilot finding. The cost is roughly 1.7× the subsequence probes on that path,
mitigated by the shared cache. The new constant silently encodes _link_anchor_bounds's
2 + 1 + 2 + 1 minimum; if that function's minimums change, the two will desync quietly.

(e) Nothing runs these tests automatically. gh pr checks 5 reports no checks, and the head
contains no .github/, .gitlab-ci.yml, tox or other CI configuration. The hand-written
authorship_cases.json suite is now the main safety net for this algorithm, and it only runs
when someone remembers to run it locally. Given how much of this series' correctness rests on
that file, a workflow that runs pytest tests/ on push looks like the highest-value remaining
change in the PR — it takes 10 s and 0.38 GB. Minor snag for whoever sets it up: in an
environment where pytest-django is installed, pytest tests/ aborts during fixture setup and
needs -p no:django.

---

Smaller notes

• The new re.sub in split_into_paragraphs is still an inline pattern while every other regex
  in utils.py is compiled at module level, and this runs once per revision.
• _can_partially_restore_historical_sentence still builds removal_revisions under
  if word.outbound and then separately evaluates all(word.outbound …); one pass would do.
• The span-ordering assumption in the new pass (if span_start >= prev_start + run_length: break) is safe: SequenceMatcher.get_opcodes() yields spans in ascending order, so
  move_prev_spans is sorted and non-overlapping. Noting it because it is load-bearing and
  unstated.
• Downstream, for wikiwho_api: the paragraph-split change moved again in this round, so
  paragraph and sentence hashes change again. Since pickles carry no algorithm-version
  invalidation, a rebuild rather than an incremental upgrade still looks like the safer
  deployment.

---

Drafted in a Claude Code session (~1 h, 3 human messages). Sage asked for the re-review and set
the memory constraint it had to work under, and read this comment in full before posting, but did
not independently reproduce any finding. Every empirical claim above — the adjudicated
attribution split, the sentences_ht invariant counts, the 71 k-revision wikitext scan, the pass
instrumentation, and the memory figures — was produced by scripts Claude Code wrote and ran
in-session, unchecked by a second party. No code in this comment has been run or reviewed by a
human; the previous review comment carried the same warning about the snippet that became
_template_field_anchor, which is finding (a) above.

(Comment written by Claude Code.)

[ragesoss]

Follow-up: the attribution measurement, on a wide corpus this time

Same caveat as before — these numbers came out of an AI-assisted session, they are observations
rather than conclusions, and they warrant independent checking before anyone leans on them.

The 65.6% figure in the previous comment came from the eight fixtures in this repository, ~3,300
revisions and almost entirely enwiki. That is a thin basis for a claim about an algorithm that
runs on every wiki, and the previous comment should have said so more plainly. The wider corpus
used in the very first review comment (72.4% favourable) was measured at e5a556ba — two heads
and two substantial reworks of the moved-run rules ago — so it did not describe the current code
either.

Re-run against the current head, e0cc079 vs master (38f4117): 16 articles, 11 wikis,
38,110 revisions, full histories replayed end to end under both versions, final token streams
aligned by value (token counts differ, so positional comparison misleads), and every token whose
origin changed adjudicated by introduced-vs-inherited — does the credited revision's token
stream contain the token's 7-token context window while its parent's does not?

	decisive cases	credits the introducing revision
master	1,673	480 (28.7%)
this PR		1,193 (71.3%)

Combined with the eight repository fixtures: 1,378 of 1,955 decisive cases (70.5%). So the
improvement measured at e5a556ba survived the rework intact — that was the open question, and
the answer is yes.

Every article that produced a decisive case, so the small samples are visible rather than
filtered out:

article	decisive	master	this PR	PR share
ar/4287	644	110	534	82.9%
ja/4821051	440	186	254	57.7%
hi/59	342	76	266	77.8%
he/325	65	23	42	64.6%
en/83780967	55	31	24	43.6%
es/972	36	18	18	50.0%
en/349784	29	10	19	65.5%
ru/71	25	7	18	72.0%
de/2552494	20	12	8	40.0%
pt/404	8	1	7	87.5%
en/46827	5	2	3	60.0%
simple/27263	4	4	0	0.0%
total	1,673	480	1,193	71.3%

Three articles go to master and one ties, all on small samples — the same pattern as the first
review, where two small samples went the other way. Nothing here looks language-specific: the
two largest non-Latin-script corpora, Arabic and Hindi, are also the most favourable. The four
remaining histories (en/79023819, zh/1686258, en/81357403, en/24544) produced no origin
divergence at all.

Two things this does not say.

It does not say the change is regression-free. Across both corpora it moves roughly 577 tokens
onto a revision that does not contain their context — real attribution that master gets right
and this does not, at a ratio of about 1 to 2.5 against. The largest single concentration is
ja/4821051 (186 of 440 decisive cases). If anything in this PR deserves another look before
merge, that is where the evidence points, along with the repeated {{cite}} date fields on
Splatoon 3 noted previously.

It also does not cover everything. About 48% of drifted tokens were ambiguous either way —
context window present in both candidate revisions or neither, which is the expected profile for
duplicated tokens, markup and citation internals — so the adjudicated set is the decidable
minority, not the whole diff. Seven histories were truncated at 4,000 revisions and ar/4287's
drifted tokens were sampled to 1,500 to bound the work.

On reproducing it. The whole run took 8 minutes at 0.21 GB peak RSS, single process, no
parallelism. Two earlier passes at this review skipped fixtures believing they were too expensive
to replay; that was wrong, and the correction is worth passing on, because it is the difference
between measuring an algorithm change on 3,300 revisions and on 38,000. The only thing that
actually costs memory here is loading a revision history into a Python list — parsed dicts run
about 2.1× the on-disk JSON, so a multi-gigabyte history needs tens of gigabytes before analysis
starts. Feeding revisions one at a time instead:

for line in open(history_path, encoding='utf-8'):
    ww.analyse_article([json.loads(line)])   # not analyse_article(revs)

keeps a 10,000-revision replay of a large article at ~0.2 GB. analyse_article itself is cheap,
and this PR does not change that.

Deployment note repeated from the previous comment because it bears on timing rather than
correctness: paragraph and sentence hashes moved again in this round, and pickles carry no
algorithm-version invalidation, so a rebuild rather than an incremental upgrade looks like the
safer rollout.

---

Drafted in a Claude Code session (~1.5 h, 6 human messages). Sage directed the review, set the
memory and concurrency limits the measurement had to run under, asked specifically for the
wide-corpus gap to be closed rather than accepting the narrower figure, and read this comment in
full before posting — but did not independently reproduce any of these numbers. The replay,
alignment, adjudication and resource measurements were produced by scripts Claude Code wrote and
ran in-session, unchecked by a second party.

(Comment written by Claude Code.)

[ragesoss]

@MusikAnimal I'm happy with it now.