Optimize the diff algorithm, part 4

[QZGao]

The matcher introduced in #5 improves several attribution cases, but it still cannot reliably preserve token identity when identical text occurs in multiple structural locations, as raised by @ragesoss in https://phabricator.wikimedia.org/T434097 on the Yeah Yeah Yeahs (EP) article. The failure mode is :

• The lead title originates in revision 1625540 (2003).
• Between 2015 and 2016, the article contains the same bold title twice:
  • once in the lead;
  • once in an infobox chronology field.
• During the subsequent article restructuring and removal of the infobox duplicate, the matcher follows the wrong occurrence lineage.
• The surviving lead title is consequently attributed to revision 748288963 instead of retaining token IDs 5, 6, and 7 from revision 1625540.

This is an occurrence-assignment problem. The existing matcher operates over flattened article-wide word arrays. Its matching stages—edge matching, SequenceMatcher, positional fallback, moved-run recovery, and construct-boundary recovery—assign token identities as they execute. When two identical occurrences compete, an early stage may consume the wrong previous token identity. Later evidence cannot reliably distinguish:

• an unmatched token;
• a correctly matched duplicate;
• a duplicate already assigned to the wrong occurrence.

The matcher therefore needs to consider competing evidence before finalizing token reuse, and it needs paragraph and sentence occurrence information to distinguish structurally separate copies.

A second authentic case, Bradshaw, West Virginia, defines an important boundary for the fix. Two different templates contain the textual value September 2012. When the unref template is removed, the remaining bias value must retain its own origin, revision 510854404, rather than inherit the older unref token from revision 510852310. Identical surrounding text must not override distinct structural ownership.

Therefore this #6 extends from #5 with the following matcher algorithm changes:

Proposed fixes and major algorithm changes

1. Separate match discovery from final token assignment

All word-matching stages now submit proposals to a central candidate ledger.

Each candidate records:

• the proposed current-to-previous token pairs;
• its evidence source;
• its confidence tier;
• the amount of contextual support;
• displacement;
• structural occurrence paths where applicable.

The ledger maintains a provisional compatibility view for candidate generators that depend on earlier matches, but these provisional assignments do not determine the final result.

After all applicable evidence has been collected, one resolver produces the final one-to-one token mapping.

The evidence order is explicit:

1. exact common-prefix and common-suffix matches;
2. full-revision-unique moved runs;
3. structurally anchored gap matches;
4. edited-construct boundary matches;
5. article-wide SequenceMatcher matches;
6. bounded positional fallback.

Structurally supported matches therefore outrank generic article-wide sequence alignment, while verified globally unique moved text can still outrank a merely local structural correspondence.

2. Preserve explicit structural occurrence identity

The structural matcher uses token slots carrying:

• token value;
• article-wide offset;
• paragraph occurrence index;
• sentence occurrence index;
• word offset within the sentence;
• the persistent Word object when one already exists.

Paragraph and sentence hashes are not treated as unique occurrence identifiers because identical paragraphs and sentences may legitimately appear multiple times.

The previous revision is read from its persistent hierarchy. The current revision reuses the hierarchy already produced during parsing. If the hierarchy is incomplete, inconsistent, or aliases paragraph, sentence, or word objects in a way that prevents reliable occurrence identification, structural matching fails closed to the original tokenizer-based representation.

3. Gate structural analysis on actual duplicate competition

Structural matching is not run for every edited revision.

A conservative residual preflight first checks whether the unresolved portions of both revisions share a potentially duplicated window. If they do not, the matcher retains the existing result without constructing full structural context.

Only revisions with a genuine unresolved duplicate competition proceed to structural anchor discovery.

This keeps the new behavior targeted at the class of problems it is designed to solve.

4. Establish structural correspondence with globally unique anchors

For relevant revisions, the matcher indexes informative windows of 10, 8, 6, and 4 tokens.

An anchor is usable only when its exact contextual key occurs once in the complete previous revision and once in the complete current revision. These globally unique anchors associate paragraph occurrences without relying on paragraph hashes.

Compatible overlapping anchors are merged into longer segments. Paragraph pairs are accepted only when they have sufficient lexical support and are uniquely preferred from both directions.

Paragraph starts and ends become virtual anchors only for unambiguous one-to-one paragraph correspondences. Paragraph splits and merges do not receive broad virtual boundaries merely because one correspondence is larger than another.

This is what separates the lead occurrence of Yeah Yeah Yeahs from the identical infobox occurrence.

5. Align only the gaps between certified anchors

Once a paragraph correspondence has been established, the matcher runs a bounded exact LCS inside the gaps between its anchors.

The alignment score applies deterministic priorities:

1. maximize the number of matched tokens;
2. preserve longer contiguous runs;
3. prefer informative context;
4. minimize displacement;
5. select previous occurrences left-to-right when the textual evidence is genuinely tied.

The left-to-right rule preserves established behavior for cases such as said said becoming said, where text alone cannot determine which historical occurrence survived.

Gap alignment is capped at 50,000 cells. Larger or insufficiently supported gaps do not receive structural matches.

6. Require duplicate and boundary evidence for structural runs

A run found inside a certified gap is not automatically accepted. Therefore, The matcher verifies that:

• the relevant ambiguity window occurs more than once in both complete revisions;
• the window is wholly contained in the still-available residual portion;
• the proposed run has sufficient informative content;
• the run does not improperly cross a template-field boundary;
• its paragraph correspondence is uniquely certified.

These checks prevent textual continuity from transferring identity across unrelated templates, as in the Bradshaw case.

7. Resolve structural conflicts as complete runs

Structural candidates are accepted atomically rather than as independent token edges.

A structural candidate is rejected if any part conflicts with stronger evidence or an already selected structural run. This prevents the resolver from constructing a mixed mapping that no candidate generator actually proposed.

Structural and lower-tier candidates are grouped into connected conflict components through their current and previous token endpoints. Components larger than 512 endpoints are considered too broad to disambiguate safely and retain the established mapping.

The final structural result is also rejected if it:

• preserves fewer previous identities than the established mapping; or
• produces only an isolated one- or two-token cardinality gain.

This keeps the structural layer conservative outside well-supported duplicate-lineage cases.

8. Add compact and targeted structural indexing

The production path avoids allocating a full structural object for every complete-revision token.

It instead builds compact documents containing:

• token values;
• paragraph ranges;
• the sentence offsets required for residual alignment;
• lazily constructed contextual keys;
• informative-token prefix sums.

Anchor discovery begins with keys from residual-bearing paragraphs and then verifies those exact keys over both complete revisions. When the targeted set becomes too broad, the matcher uses the complete exact scan instead.

Only one anchor width is retained at a time. Exact multi-pattern automata are used above measured document-size thresholds, with a 250,000-symbol limit to prevent unusually broad edits from creating an excessive transient object graph.

These routing choices affect cost only. Tuple scanning, automaton scanning, and the original slot-based fallback produce the same occurrence states and candidate evidence.

9. Add optional native acceleration for exact hot loops

An optional CPython extension accelerates the most frequently executed exact operations:

• token and paragraph splitting;
• adjacent structural-document construction;
• contextual document indexing;
• subsequence occurrence counting;
• duplicate-window discovery;
• unique-anchor extraction;
• residual-window extraction;
• bounded LCS.

The native extension does not decide token lineage. Candidate generation, evidence ordering, and conflict resolution remain in Python.

Every native kernel has an exact Python implementation. The extension is optional on Python 3, so a compilation or import failure falls back to the tested Python path rather than preventing WikiWho from being installed.

The native implementation changes constant factors—principally Python object creation, attribute access, and interpreter dispatch—without changing the matcher’s asymptotic bounds or correctness rules.

Minor notable changes

• iter_rev_tokens() now uses occurrence counters scoped separately to the revision’s paragraph level and each paragraph’s sentence level. This correctly traverses repeated paragraph and sentence hashes without repeated list counting.
• The current structural representation reuses the already-parsed revision hierarchy and only re-tokenizes the complete current wikitext as a fail-closed fallback.
• Shared paragraphs are scanned once when constructing adjacent structural documents.
• Structural occurrence counts saturate once a window is known to be duplicated, because the matcher only needs to distinguish zero, one, and multiple occurrences.
• The optional extension is configured from setup.py and uses the same token-symbol alphabet as the Python tokenizer.
• tests/test_structural_native.py directly compares every native operation with its Python oracle.
• Authentic revision fixtures were added for:
  • Yeah Yeah Yeahs (EP), covering the complete 359-revision history;
  • Bradshaw, West Virginia, covering the structurally distinct template values.
• The Yeah Yeah Yeahs assertions pin token IDs 5, 6, and 7 and origin revision 1625540:
  • before the duplicate-lineage loss;
  • through the 2016 restructuring;
  • after the infobox duplicate disappears;
  • in the latest cached revision.
• The Bradshaw assertion pins origin revision 510854404, ensuring that the structural matcher does not transfer identity from the removed unref template.

Test results

All Python tests used the canonical Python 3.9 environment.

Correctness

Test surface	Result
Local suite with native extension	75/75 passed
Pure-Python fallback suite	57/57 passed; native-only tests skipped
Yeah Yeah Yeahs lineage checkpoints	4/4 passed
Bradshaw structural-boundary checkpoint	1/1 passed
Complete regression evaluation	Approximately 7,200 records; zero differences
Evaluation summaries	All identical
Article analyses	All completed; zero failures
Locked stress histories	All completed with unchanged revision and token counts

The complete 359-revision Yeah Yeah Yeahs (EP) history retains token IDs 5, 6, and 7, originating in revision 1625540, through the article restructuring, removal of the infobox duplicate, and latest cached revision.

Native/Python differential validation

Component	Coverage
Tokenizer	Approximately 200,000 randomized and adversarial inputs
Paragraph splitter	Approximately 200,000 randomized and adversarial inputs
Authentic parser corpus	Approximately 280 histories and 29,000 revisions
Parser-produced content	Approximately 12 million sentences and 640 million input characters
Exact subsequence counter	Approximately 200,000 randomized inputs
Structural kernels	Direct and randomized comparisons with Python implementations
Scan routing	Tuple, automaton, targeted, and complete-scan paths
Fallback behavior	Persistent, new, shared, and inconsistent hierarchy cases

Runtime and memory

Stress surface	Current result, approximately	Change from previous validated matcher
Complete locked stress set	57 seconds	+4.3% runtime
Google Play prefix	46 seconds / 171 MiB	+5.9% runtime, +2.8% traced memory
Splatoon 3 prefix	4 seconds / 39 MiB	−1.1% runtime, +21.6% traced memory
Through the Looking-Glass prefix	3 seconds / 28 MiB	+0.1% runtime, +8.9% traced memory
Full Google Play history	69 seconds / 208 MiB	+7.5% runtime

The full Google Play run processed approximately 4,000 fixture records, accepted approximately 3,700 revisions, and produced approximately 135,000 token objects.

Traced-memory comparisons are approximate because the reference values aggregate repeated runs, while the current values are primary-run peaks. A separate process-level audit found that native acceleration reduced runtime by approximately 31% while increasing maximum RSS by approximately 1.9%, with no indication of a large hidden native allocation.

---

This #6 should be merged after #5, as the former is an extension of the latter.

(Since both of our maintainers mainly use LLMs to review the code now, I have this write-up, composed with help from GPT-5.6 Sol Ultra, for your LLM agents to quickly get the idea.)

Note: After merging #5 into main, there expect to have merge conflicts, as #5 has one more commit made since #6's divergence. Still, the conflicts are rather easy to resolve. Merge conflicts are resolved.

[Copilot]

Pull request overview

Improves token attribution when identical text appears in competing structural locations, with optional native acceleration.

Changes:

• Adds candidate-ledger resolution and structure-aware duplicate matching.
• Introduces native acceleration with Python fallbacks.
• Adds authentic lineage fixtures and differential regression tests.

Reviewed changes

Copilot reviewed 5 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
WikiWho/wikiwho.py	Implements structural matching and candidate resolution.
WikiWho/utils.py	Fixes table parsing and occurrence traversal.
WikiWho/_structural_native.c	Adds native structural-matching kernels.
setup.py	Configures the optional extension.
tests/test_structural_native.py	Compares native and Python behavior.
tests/test_regression.py	Expands matcher regression coverage.
tests/authorship_cases.json	Adds authentic attribution cases.
.gitignore	Ignores macOS metadata.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[QZGao]

Done. I have also merged matcher-fix into matcher-fix-2, so now #6 can be properly merged right after #5 without conflicts.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tests/authorship_cases.json:152

• This regression only checks the 2012 token, so an incorrect transfer of September from the removed unref template would still pass. The stated Bradshaw boundary requires the complete September 2012 value to retain revision 510854404; include both tokens in the assertion.

        "focus": ["2012"],
        "expected": {
          "values": ["2012"],
          "origin_rev_ids": [510854404]
        }

WikiWho/utils.py:47

• The comment describes |}} as the matched template ending, but the negative lookahead deliberately excludes that sequence and matches only line-start |} table closers. Update it so future changes do not invert this parsing rule.

    # Treat a line-start `|}}` as a template ending. It is ambiguous with a
    # table close followed by a literal `}`, which this fast path cannot parse.

[ragesoss]

Review notes — analysis run with Claude Code

How to read this. These findings come from an AI-assisted review session. They are not to
be automatically trusted or blindly acted upon. Each one is an observation that warrants
independent verification; some are likely to be wrong or to miss context the author has. Read
them, check the ones that matter, and use or discard them on your own judgment.

Scope. Reviewed as git diff e0cc079 origin/pr6, i.e. #6's own contribution on the
assumption that #5 lands first: +3,548 lines in WikiWho/wikiwho.py (2,026 → 5,150 lines),
the 3,989-line _structural_native.c, two new fixtures, and ~1,000 lines of tests.
Everything below was produced by execution against master (38f4117), #5's head (e0cc079)
and this head (b7d9000), not by reading the changelog.

One method note for anyone re-running this. The extension's .so is Python-version tagged,
so building it under 3.9 and then replaying under a different interpreter loads nothing and
silently compares the pure-Python path against itself. That mistake produced a vacuous
"identical" result in this session before it was caught. Any native-vs-Python comparison should
assert wikiwho._structural_native is not None before it starts.

---

The change decomposes cleanly, and that is the most useful thing here

component	effect on output	runtime cost
the candidate-ledger / resolver rewrite (most of the Python diff)	none	0–3%
the single _propose_structural_word_matches() call	all of the behaviour change	+17% to +98% in pure Python
the C extension	none	buys that cost back to roughly #5 parity

With that one call skipped, this branch is byte-identical to #5 on all twelve fixtures —
every field compared, including token_id, inbound and outbound:

2026_canvas_security_incident  identical (9517)    one_thing_leads_2_another  identical (1616)
adam_himebauch                 identical (8309)    splatoon_3                 identical (37774)
american_goldfinch             identical (10041)   the_tell-tale_brain        identical (4980)
bradshaw,_west_virginia        identical (4397)    through_the_looking-glass  identical (26902)
japan_cup                      identical (8740)    yeah_yeah_yeahs_(ep)       identical (14685)
luis_donaldo_colosio_riojas    identical (6869)    nintendo_switch            identical (1090)

So restructuring every existing matching stage to submit proposals to a ledger is verifiably
output-neutral, and it costs nothing measurable. That is worth stating plainly because it is the
part of the diff most likely to worry a reviewer, and it is the part that needs the least
scrutiny. Review attention belongs on the structural layer.

Verified claims

Native and Python agree exactly. This is the branch's central correctness claim and it holds
under independent testing:

surface	scale	divergences
full replay, all 12 fixtures, every token field	~135,000 tokens	0
tokenizer + paragraph splitter on real wikitext	16,748 revisions, 628 M characters, 12 wikis (ar he hi ja zh ru de es pt fr simple en)	0
boundary + randomised fuzz	60,892 inputs, every CJK range edge ±1, all TOKEN_SYMBOLS, adversarial |} / {{ / {| forms	0

The C range table in is_cjk_token_character matches regex_cjk interval for interval.

T434097 is genuinely fixed, and the boundary holds. Running this branch's
authorship_cases.json against #5's engine — test_authorship_cases.py imports only the public
API, so it runs cross-branch — the three lead-title cases fail at #5 (token_ids
[2676, 2677, 2678]) and pass here ([5, 6, 7], origin 1625540). The Bradshaw case passes on
both branches, so the September 2012 boundary was not broken to get there.

The C is careful. Every size multiplication is guarded (capacity > SIZE_MAX / sizeof(T),
capacity > SIZE_MAX / 2 before doubling), realloc results go to a temporary so a failure
cannot leak the old block, allocation failures set PyErr_NoMemory and unwind, and the LCS cell
cap is enforced before allocating (curr_len > max_cells / prev_len), which doubles as the
overflow guard. No leak signal: twelve repeated replays in one process drift +0.6 MB with native
versus +0.3 MB pure-Python. Peak RSS is within ~2 MB of the pure-Python path on every fixture,
so there is no hidden native allocation.

optional=True fails open as advertised. Appending #error to the .c and rebuilding from
clean: build_ext warns, setup.py exits 0, no .so is produced, and analyse_article
works pure-Python. Confirmed rather than assumed.

Attribution impact is small. Over 30,327 revisions and 16 articles in 11 wikis, aligning the
final token streams by value: 267 origin differences in 281,694 aligned tokens (0.09%), and
19 inbound/outbound differences.

---

Concerns

1. The pure-Python fallback is much slower than #5, and a build failure is silent.

Best of three, same fixtures, same interpreter:

fixture	#5	#6 pure-Python	#6 native
adam_himebauch	0.19 s	0.30 s (+58%)	0.19 s (±0%)
yeah_yeah_yeahs_(ep)	0.87 s	1.72 s (+98%)	0.85 s (−2%)
splatoon_3	4.12 s	5.49 s (+33%)	3.89 s (−6%)
through_the_looking-glass	2.44 s	2.85 s (+17%)	2.38 s (−2%)

The "+4.3% runtime" in the description therefore appears to describe the native path against
some earlier native baseline, not this branch against #5. Measured against #5, the accelerated
path is about break-even and the fallback is 17–98% slower. Isolating it with the gate described
below shows the entire cost is the structural layer — the ledger rewrite is free.

That matters operationally because of how the two interact: optional=True reduces a build
failure to a warning inside build_ext output with a zero exit status, and the import site
catches ImportError silently. A deploy onto a host without a working compiler, or with a
Python that does not match a prebuilt .so, gets the slow path with nothing in the logs. A
one-line logging.warning at import when _structural_native is unavailable would make that
visible, and stating the fallback cost in the description would set expectations.

2. Origin changes outside the targeted class adjudicate as a coin flip.

Of the 267 origin differences, 84 were decidable by introduced-vs-inherited — does the credited
revision's token stream contain the token's 7-token context window while its parent's does not?

	decisive cases	credits the introducing revision
#5	84	45 (53.6%)
#6		39 (46.4%)

Per article: en/83780967 30–28 to #6, en/349784 3–1 to #6, ar/4287 2–1 to #6, hi/59 3–6 to #5,
ja/4821051 1–5 to #5, he/325 0–4 to #5. Small samples throughout, and 0.09% of tokens is a
narrow blast radius — but the drift outside the duplicate-lineage case it targets is not an
improvement. That is worth knowing when deciding how much of this to attribute to the new
evidence ordering versus to the specific fix.

3. token_id renumbering is broad, though benign in kind.

15% of aligned tokens (42,265) get a different token_id. Nearly all of it is a constant
offset rather than reassignment — ar/4287 shows delta 9 on 6,540 tokens, hi/59 delta 9 on
23,348, en/349784 deltas 20 and 13, ja/4821051 delta 4 — because this branch creates a handful
fewer tokens early in each history and everything after shifts. A small residue of large deltas
(ar/4287 delta 28,300 on 9 tokens; he/325 delta 21,106 on 4) is genuine lineage recovery, which
is the point. Nothing wrong here, but any consumer that persists token_id across the upgrade
will see them all move, which is worth pairing with the pickle-rebuild note from #5.

---

A kill switch looks cheap and worth having

The structural layer enters through exactly one call before ledger.resolve(), so it gates
trivially. Adding an environment check around that single call and re-running the suite:

• structural layer on: 33 of 33 authorship cases pass;
• structural layer off: the 3 Yeah Yeah Yeahs cases revert to Optimize the diff algorithm, part 3 #5's answer, all 30 others pass;
• byte-identical to Optimize the diff algorithm, part 3 #5 on all twelve fixtures, at Optimize the diff algorithm, part 3 #5's runtime.

Given this is a ~7,500-line change to production attribution with no CI, being able to disable
the structural layer — and separately to force the pure-Python path — without a redeploy seems
like cheap insurance for whoever is on the other end of a bad day.

The gate used for these measurements was written by Claude Code and has not been run or
reviewed by a human beyond this session. It is offered as evidence that the layer is separable,
not as a proposed commit.

Smaller notes

• Two compiler warnings are false positives. gcc -Wall -Wextra reports sentence_start
  and sentence_length as possibly uninitialised at _structural_native.c:2739/519. Every
  early goto done in append_sentence_values sets status to 0 or −1 and the caller bails on
  <= 0, so both are always set when it returns 1. gcc cannot see that correlation; initialising
  them at declaration would silence it. The remaining 13 warnings are -Wunused-parameter on
  Python C API signatures, and one -Wmissing-field-initializers on a sentinel.
• tracemalloc cannot see this extension. The kernels use raw
  malloc/calloc/realloc rather than PyMem_*, so the "traced memory" columns in the
  description are blind to native allocation by construction. The process-level RSS audit
  mentioned in the description is the right instrument, and independent RSS measurement here
  agrees with it — worth noting so nobody re-derives the worry from the traced figures.
• One fail-closed path in an otherwise fail-open design. configure_token_symbols runs at
  import and raises TypeError if TOKEN_SYMBOLS is not a tuple, but the import site catches
  only ImportError. If that constant's type ever changes, WikiWho.wikiwho becomes unimportable
  rather than falling back. Broadening to except Exception around both the import and the
  configure call would close it.
• iter_rev_tokens is a stricter contract now. The rewrite indexes by occurrence
  unconditionally where the old code fell back to index 0 for single-object hashes, so a hash
  appearing in ordered_paragraphs more often than it has stored objects becomes an IndexError
  instead of a silent duplicate. Checked across 3,925 revisions of all twelve fixtures: the
  invariant holds everywhere, old and new yield identical streams, and neither raises. Failing
  loudly is arguably the better behaviour; flagged only because it is a public entry point.
• japan_cup is a red herring, noted to save the next reader the detour. Its 16 changed
  outbound values are two textually identical 8-token copies (* 4 - y - o & up, both
  originating in 24427622) swapping which identity is recorded as removed at rev #61 versus
  #136. Both branches record one of each; the article genuinely drops one copy at #61 and the
  other at #136, and text alone cannot say which is which. The left-to-right preference stated in
  the description is the more principled convention. Rev #97's absence from the in/out lists is
  vandalism flagging, not a gap.
• Still no CI. With C in the tree this now also means nothing catches a platform build break,
  and the pure-Python and native paths can drift apart without anyone noticing. The suite runs in
  ~10 s.

What this review did not cover

The 3,989 lines of C were not audited line-by-line for reference-count correctness; valgrind
was unavailable on the review machine, so the evidence for memory safety is differential testing,
a repeated-replay leak probe, and RSS comparison rather than instrumentation. One platform only
(x86-64 Linux, gcc 15.2, CPython 3.9.13) — no other Python versions, compilers, or architectures,
which is exactly where an optional C extension is most likely to surprise. The wide corpus was
capped at 3,000 revisions per history. The ~1,000 new lines of test code were read for what they
assert, not reviewed closely.

---

Drafted in a Claude Code session (~1 h on this review, within a longer session of ~2.5 h and 9
human messages). Sage directed the review, set the memory and concurrency limits every
measurement ran under, and read this comment in full before posting, but did not independently
reproduce any finding. The replays, differential and fuzz testing, adjudication, build tests and
performance measurements were produced by scripts Claude Code wrote and ran in-session, unchecked
by a second party. No code in this comment has been reviewed or run by a human beyond that.

(Comment written by Claude Code.)

[QZGao]

I agree with Claude Code that the existing cases do not demonstrate the improvements yet. Therefore I'm adding ten more test cases.

[QZGao]

1. Yoshiminosato Station: URL moved out of repeated citations

Target:

| website = {{Official website|1=http://www.nankai.co.jp/traffic/station/yoshiminosato}}

Revision sequence:

• 754998474: no station URL.

• 777311072: JaventheAlderick adds the station source repeatedly in citations as:

  http://www.nankai.co.jp/traffic/station/yoshiminosato.html

• 840729939: Nyamo Kurosawa consolidates the repeated citations and reuses that URL stem in the infobox website field and external links. The .html suffix is removed.

• The test examines the shared URL tokens through yoshiminosato, excluding the removed .html.

Under WikiWho’s token-reuse semantics, those tokens came from 777311072 and were moved/reused during the cleanup. Both older branches instead reset every URL token to 840729939, attributing the existing URL content to the cleanup editor.

2. 天草市立二浦小学校: duplicate CJK characters in a newly added name

Target—the second occurrence of:

早浦亀浦両村組合立 白石小学校

The test specifically selects the first 浦 and the 村.

Revision sequence:

• 37043401: the original article contains several separate school names, such as 早浦小学校, 亀浦小学校, and 二浦村立白石小学校. It therefore contains many identical 浦 and 村 character tokens, but not the combined target name.
• 102587470: still does not contain the combined narrative occurrence.
• 106452677: Ackeyyama introduces the combined name in the former-name field and in a new narrative history sentence.

Because WikiWho tokenizes this Japanese text character by character, the old matchers borrow the matching 浦 and 村 from unrelated names in the original 37043401 article. The targeted narrative occurrence was actually introduced in 106452677.

3–5. Jewish Sports Review: unchanged citation fields reset by a dash edit

The three targets come from the same citation:

|title=Jewish sports legends: the International Jewish Hall of Fame
|publisher=Books.google.com
|date=
|accessdate=May 12, 2010

They test three kinds of tokens:

1. The field name publisher.
2. The value Books.google.com.
3. The field names date and accessdate.

Revision sequence:

• 361652961: the article contains a bare Google Books reference.

• 361653377: Epeefleche runs Reflinks, converting it into a structured cite web reference and introducing all these fields.

• 361653642: the reference remains unchanged.

• 361653919: the same editor changes the title punctuation from:

  ... - Google Books

  to:

  ... – Google Books

The selected publisher, hostname, and date-field tokens do not change in 361653919. Nevertheless, both older matchers rematch the citation around the changed dash and assign all selected fields to 361653919. Their correct origin remains 361653377.

These are three tests because they cover a template field key, a field value, and neighboring empty/nonempty field keys.

6. Richard Thompson: moved award name is wrongly treated as newly written

Target text:

Marsh Award for Marine and Freshwater Conservation

Revision sequence:

• 933675168, by Lopifalko, creates the article and introduces the phrase in two places:

  *2016: [[Marsh Award for Marine and Freshwater Conservation]] from ...

  It also occurs inside a citation title. Both copies therefore have the same origin revision and editor.

• The phrase remains present exactly twice through every one of the 16 revisions used by the test.

• 954140295 still has the linked award name:

  *2016: [[Marsh Award for Marine and Freshwater Conservation]] from ...

• 954149685, by Duncan.Hull, reorganizes the article, moves the award bullet under “Awards and honours”, and removes the link brackets:

  *2016: Marsh Award for Marine and Freshwater Conservation from ...

The seven words themselves are unchanged. Duncan.Hull authored the restructuring and unlinking, but not those words.

Correct attribution on matcher-fix-2:

marsh         → 933675168
award         → 933675168
for           → 933675168
marine        → 933675168
and           → 933675168
freshwater    → 933675168
conservation  → 933675168

Both master and matcher-fix incorrectly attribute all seven tokens to 954149685, thus crediting Duncan.Hull rather than Lopifalko.

Why this is adjudicable:

• The complete seven-word span survives verbatim.
• Its semantic role remains the name of the same award in the same bullet.
• Only its location and surrounding link markup change.
• Although the phrase occurs twice, both copies were introduced in 933675168. Choosing either earlier copy leads to the same origin revision and editor. There is therefore no origin-level coin flip.

7. Meeting at Night: retained phrase is reset during a lead rewrite

Target text, including quotation marks:

"Night" and "Morning"

It tokenizes into seven tokens:

"  night  "  and  "  morning  "

Revision sequence:

• 575833625, by Solomon7968, creates the article. The lead says:

  Browning wrote the poem in two parts as "Night" and "Morning",

  The same phrase also occurs later in the article:

  the two love poems "Night" and "Morning" as complementary

  Thus both occurrences originate in the same revision.

• The exact phrase remains in both positions through 577765467.

• 577768395, by Bjenks, rewrites the lead sentence:

  Before:

  Browning wrote the poem in two parts as "Night" and "Morning",

  After:

  The original poem appeared in ... (1845) in which "Night" and "Morning" were two sections.

  The surrounding sentence is new, but the quoted phrase is preserved verbatim and still names the same two original sections.

• Later revisions—including 577773380 and the test snapshot 744563761—leave this phrase unchanged. Across all 32 revisions in the test history, it consistently occurs exactly twice.

Correct attribution on matcher-fix-2:

"        → 575833625
night    → 575833625
"        → 575833625
and      → 575833625
"        → 575833625
morning  → 575833625
"        → 575833625

Both master and matcher-fix incorrectly assign all seven tokens to 577768395, crediting Bjenks instead of Solomon7968.

Why this is adjudicable:

• The entire quoted phrase is identical before and after the rewrite.
• It continues to perform the same semantic role in the lead.
• It does not move between interchangeable structural slots.
• The second copy also originated in 575833625, so even an occurrence-level ambiguity could not produce a competing origin revision.

8. Johannes Westö: a new infobox field steals HJK from an older field

Target—the first HJK, which is part of the link target:

| youthclubs1 = [[HJK Helsinki|HJK]]
                  ^^^

Revision sequence:

• 300557875: Lomdiff creates the article with HJK in currentclub and clubs.

• 387490690: it still has clubs = [[HJK]], but no youth-club field.

• 390661514: Etzo introduces:

  | youthclubs = [[HJK Helsinki|HJK]]

• 427065194: a bot structurally renames the parameter to youthclubs1.

• The link persists to snapshot 957571858.

Although identical HJK text existed elsewhere in the infobox, the youth-club occurrence is a new fact in a new field introduced by 390661514. Both older branches incorrectly take its first HJK from the unrelated 300557875 occurrence.

9–10. 1925 Seanad election: a new table row steals boilerplate from old rows

Target row:

[[P. J. Brady]] elected at a by-election to replace [[Stephen O'Mara]]

The tests separately select:

• elected at a by-election
• to replace

Revision sequence:

• 326404585: the original article contains several other rows using exactly the same boilerplate:

  William Cummins elected at a by-election to replace ...
  Thomas Foran elected at a by-election to replace ...
  Douglas Hyde elected at a by-election to replace ...

• 326593076: those existing rows survive a copyedit; there is no P. J. Brady row.

• 326650275: Spleodrach adds the new P. J. Brady row.

The repeated wording in the Brady row is a new occurrence introduced with that row. Both older branches match it to one of the unrelated boilerplate runs from article creation and assign 326404585.

The username is the same for both revisions, but the occurrence-level revision provenance is wrong.

(These test cases are mined from real Wikipedia article revisions by GPT-5.6 Sol.)