Skip to content

Prompt v4: the LLM reads "until midnight on D" as the start of D #102

Description

@baz8080

Deferred by the owner on 2026-09-24. Found by the retroactive review, and fixed in the rules only by rules-v2 (baz8080/uisce#100).

The problem

This line in PROMPT in src/uisce/inference.py:

  • 12 noon (12:00pm) is 12:00. 12 midnight (12:00am) is 00:00.

makes gemma read "from 2pm until midnight on 11 September" as 11 Sep 00:00. That end falls before the works begin, and in some cases before the notice was published.

Affected records (2026-09-23 release)

  • 16 published latest LLM records read a same-day "from X until midnight on D" as D 00:00. All are closed cases:
    • 231693 to 231700, which are eight pins of one event;
    • 231812, 231853, 232292, 232318, 238423, 240587, 240993;
    • 244597, whose record is stale: the text says "until midnight 23 September", with no "on".
  • Five more have no start time and were read as D 00:00: 232219, 232957, 233026, 234332 and 239927. rules-v2 now abstains on these. 232957 and 234332 end before their own publication.

Proposed replacement for the line

  • 12 noon (12:00pm) is 12:00. Midnight (12am) is 00:00 on the day it begins, so "until midnight on 11 September" ends at 00:00 on 12 September: report local_date 2026-09-12. Only when the works start on the day before ("from 9pm on 21 May until 12am on 22 May") is the end the date as written, 2026-05-22 at 00:00.

Order of work

  1. Change the line and bump PROMPT_VERSION to 4, in one commit.
  2. Run uv run uisce-infer by hand with LM Studio.
    • The bump makes every record stale, including the rules records, so rules-v2 answers everything it covers first. That replaces gemma records for about 10,137 cases, 17 of them with a different answer.
    • The LLM gets the residue: 754 cases, or 612 distinct descriptions on the 2026-09-23 release. That is about 40 minutes at roughly 3.6 s a case.
  3. Before committing the JSONL, run the end-time eval: uv run uisce-eval-replay --csv <round> on all three labelled rounds.
    • Round 2 labels 231853 as correct at 30 April 00:00, so a pv4 that reads it right will score a miss there.
    • Adjudicate that row in notes/end-time-eval.md, never in the CSV.

Optional, same bump

The prompt could also say that an alternative supply's hours are not the works' end. 239696 to 239698 read a tanker's 23:59 as the end. If this goes in, measure it separately from the midnight change.

The full context is in notes/roadmap.md under "Decisions waiting on the owner", once baz8080/uisce#100 merges, and in notes/rules-vs-llm-end-times.md, "rules-v2".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions