rules-v2: midnight ends, tanker hours, bilingual completions; refuse unreadable LLM values

[baz8080]

PR D of the fixes from the retroactive review. The LLM prompt and PROMPT_VERSION are unchanged. The LLM makes the same midnight mistake; the prompt fix is baz8080/uisce#102, deferred by the owner.

Rules (rules.py, now rules-v2)

• Midnight. "from 2pm until midnight on 11 September" was read as 11 Sep 00:00, before the works began. Now:
  • a start earlier on the same day ends at D+1 00:00;
  • after a start the day before, only a literal "12am on D" reads D 00:00 ("from 9pm on 21 May until 12am on 22 May");
  • the word "midnight" after a start the day before abstains, as does no start at all.
• Tanker hours. An until inside a sentence about an alternative water supply, tanker, bowser, water station, bottled water or standpipe makes the rules abstain. It was emitted as the works' end for 239696 to 239698. Word boundaries keep "wastewater station" out of the match.
• Bilingual completions. An English completion below an Irish block is read when it is the same day's update. The Irish date comes from its header when that parses, else from the date the block opens with. Irish alone still abstains. This closes 244845, 244925 and 245025.
• "16:59pm" (an hour from 13 to 23 plus "pm") now reads as the 24-hour time.

LLM values and records (inference.py, build.py, eval_rules_shadow.py)

• Model values build.py couldn't read. Before, one of these failed every CI build.
  • Values with only one possible meaning are normalised: "9:30" becomes "09:30", "24:00" becomes 00:00 on the next day, and a window closing at "24:00" closes at "00:00".
  • Anything else fails the case loudly. The case keeps its previous record, and the failure shows in the run's count.
• readable_latest and current_states are the one reading of the JSONL, shared by inference, build and the shadow eval.
  • A case whose newest record can't be read is redone.
  • A readable older record is still published meanwhile.
• Stale records. build.py warns about a stale published record using inference's own selection and state map.
• One transaction. The DROP, CREATE and INSERTs run together, so a failure can no longer leave inferred_cases empty.
• The shadow eval compares against the model's own records only.

Review

First round. A high-effort code review confirmed the RULES_VERSION bump is safe and found 10 things. Among them were three places where rules-v2 guessed instead of abstaining: tanker hours under a revising update, midnight after a daytime start, and an Irish completion read against an older English update.

Second round. A medium review found that my first-round fixes had gone too far in places:

• Unreadable replies. Storing an unreadable reply as not_found could silently replace a good published record. Reverted: the case fails loudly again.
• The midnight cutoff. An 18:00 cutoff still guessed, so it was replaced by the literal "12am" rule above.
• The Irish date check. It missed the "10.15rn" header form.
• The shadow eval. It compared the rules with rules-v1 records.
• Smaller fixes: type guards, a shared state map, and reusing the open connection.

Measured (release DB and committed JSONL)

rules against the model's own records	rules-v1	rules-v2
answered (of 10,870)	9,970 (91.7%)	10,082 (92.8%)
agree	9,969	10,065
disagree	1	17

The 17 disagreements, read in full:

• 15 are midnight rows where the LLM is wrong.
• 1 is a known digit transposition by the LLM (237463).
• 1 has Irish and English headers that disagree with each other (244190).

Labelled replays:

• Round 1: 73/73.
• 2026-08-21 round: 113/113.
• Round 2: 110/111. The one miss is 231853, whose label accepted the midnight bug. The CSV is left as recorded.

rules-v2 abstains on 4 of the 2,691 cases whose latest record is rules-v1: 243084, 244538, 244584 and 244720. CI's --rules-only run can't replace those, so their v1 answer stands, listed in the stale warning, until the LLM residue is run by hand.

JSONL growth: the version bump re-stamps about 2,700 rules-v1 cases with the same answer on the first CI build after merge, taking the file from 19.8 MB to about 21.1 MB.

Notes

• rules-vs-llm-end-times.md: a dated rules-v2 section, plus the review amendments.
• CLAUDE.md: a rules-v2 row.
• roadmap.md: the prompt v4 entry, pointing to issue Prompt v4: the LLM reads "until midnight on D" as the start of D #102.

Tests

• New and updated tests across test_rules.py, test_inference.py and test_build.py, including one for each reviewer probe text.
• uv run ruff check and uv run pytest pass: 734 tests, merged with current main.

🤖 Generated with Claude Code

https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ