closed_tokens.py builds DISPATCH from multiple per-class token tables. When a token appears in two classes (e.g. "since" is both a preposition and a conjunction), the first class registered wins. This is tested and documented behavior (Tests/test_backend.py::TestClosedTokens explicitly asserts the first-class-wins rule for "since", "until", "as", and "that").
The gap is diagnostic, not correctness: the collision is resolved silently. A contributor who adds a token to a second class (or adds a new class that overlaps an existing one) receives no warning. The later entry is dropped at build time with no log message, no assertion, and no way to discover the conflict without reading the code or grepping DISPATCH output.
Fix: In the DISPATCH build loop, log or raise a warning when an existing key is about to be overwritten:
if token in DISPATCH:
logging.warning(
"closed_tokens: %r already registered as %r; %r entry ignored",
token, DISPATCH[token]["class"], cls_name
)
This preserves the first-class-wins behavior while making collisions observable during development.
closed_tokens.pybuildsDISPATCHfrom multiple per-class token tables. When a token appears in two classes (e.g. "since" is both a preposition and a conjunction), the first class registered wins. This is tested and documented behavior (Tests/test_backend.py::TestClosedTokensexplicitly asserts the first-class-wins rule for "since", "until", "as", and "that").The gap is diagnostic, not correctness: the collision is resolved silently. A contributor who adds a token to a second class (or adds a new class that overlaps an existing one) receives no warning. The later entry is dropped at build time with no log message, no assertion, and no way to discover the conflict without reading the code or grepping DISPATCH output.
Fix: In the
DISPATCHbuild loop, log or raise a warning when an existing key is about to be overwritten:This preserves the first-class-wins behavior while making collisions observable during development.