diff --git a/emrg/server/competition_prompt.md b/emrg/server/competition_prompt.md index ade0be64..16b0f72b 100644 --- a/emrg/server/competition_prompt.md +++ b/emrg/server/competition_prompt.md @@ -400,7 +400,7 @@ it does. | Network timeout / platform unavailable | Record in memory (blocked = network unavailable), end the round. **Do not retry** — a repeated call that already failed is not a route, and hammering it is what this row forbids. A *different* route to the same reading is the blocker rule's first case, and is worth one attempt if it cannot touch the score | | Rules text unobtainable (login wall, render failure) | **Reject by default** (§3.5) — record as `rejected` with reason "rules text unavailable, online-only status cannot be verified" | | Full offline-requirement ambiguity (staged rounds) | Read conservatively (§3.6) → do not enter | -| Needs a human action (real-name, SMS, card authorization) | Write into `blocked` with what the host must do; **do not retry, do not work around it** | +| Needs a human action (real-name, SMS, card authorization) | **Check it in the real browser first** — the platform's own page shows whether the record already exists, and a logged-out probe is not evidence about the account. Then write into `blocked` naming the step that blocks and what the host must do; **do not retry, do not work around it** | | Submission fails / score unavailable | Record the exact failure; if no score can be obtained, stop iterating and record `blocked` — never iterate without a score anchor | | Compute infeasible (GPU-heavy) | Mark `rejected` with reason "not solvable on the available compute (no NVIDIA GPU)" | | Deadline passed | Freeze (Phase E), record the final standing, archive (Phase F) | diff --git a/emrg/server/open_source_prompt.md b/emrg/server/open_source_prompt.md index fa5b46e8..9ff98cc1 100644 --- a/emrg/server/open_source_prompt.md +++ b/emrg/server/open_source_prompt.md @@ -278,6 +278,23 @@ Failing → fix, re-test, until green; a failure you cannot fix does not get hid **A suite that could not be run is not a suite that failed.** `command not found`, an exec-format error or an interpreter that is simply absent says nothing about the project — the shell never started the runner, so no test was collected. Record it as a skip: name the command and its exact error in the closing summary, state that the suite did not run, and finish the cycle without a verdict about that tree. Read as a failure it invents a defect the project does not have; read as green it claims a run that never happened. +**The same rule covers the host's side of the ledger.** Before this cycle concludes that a step is +**the host's own** — a CLA or contributor agreement, an account status, a repository setting, a +maintainer-only page — check that state in the real browser: those pages render the account's actual +record, while an API or CLI probe answers about a logged-out view and is not evidence about the +account. A conclusion that only the host can act is a claim like any other, and it is the expensive +kind, because it spends the host's time on work the cycle could have done. + +An agreement signed years ago, a verification already completed, a setting already carried — each is +invisible from the outside and decisive from the browser. So the working order is: **drive the browser, +read the actual record, and only then decide.** Where automation genuinely cannot perform the step — +a password only the host knows, a 2FA prompt, a legal-entity signature — the fallback **names that +step and why**, never a bare "this one is the host's". `§4` item 4 is where it is then written down. + +This is a rule about **capability**, and it is not a licence about **permission**: a step that would +spend money, change an account setting or sign anything as the host stays a stop-and-ask, whatever the +browser shows. + ##### 3.1.7 Commit and open the pull request Commit and branch names follow the project's conventions (§3.1.3); with none specified, use `fix/` and `: `. diff --git a/tests/test_host_blocked_browser_check.py b/tests/test_host_blocked_browser_check.py new file mode 100644 index 00000000..6a49534f --- /dev/null +++ b/tests/test_host_blocked_browser_check.py @@ -0,0 +1,297 @@ +"""One rule, two carriers: "this step is the host's" needs a browser check first. + +Why this file exists +-------------------- +A task prompt decides what a round can do for itself, and two of the shipped +templates can hand a step back to the host for a **capability** reason: the +open-source template (``open_source_prompt.md``, whose subject is platform account +state — a CLA record, an agreement signed years ago, a repository setting) and the +competition template (``competition_prompt.md``, whose ``blocked`` row is the shape +that records a wall). Neither used to require that the state be looked at before +the conclusion was written down, and the failure that permits has already happened +here: a round concluded a blocked CLA meant *only the host could sign it*, the host +asked it to look in the browser instead, and the real cause was an empty author +email on the commits — fixable from the CLI. The conclusion was wrong in the one +direction that spends the host's time. + +Two carriers, one requirement — so one arm on one carrier proves nothing about the +other. The tests below therefore do two things and the second is the point: +``test_each_carrier_states_every_part_of_the_rule`` asserts each carrier on its own +(a deletion reddens that carrier's case), and +``test_the_two_carriers_state_the_same_rule`` compares the two **both ways**, so an +edit that lands in one template and not the other is reported as the asymmetry it +is rather than as one missing phrase. + +What the clause region is, and the level it is read at +------------------------------------------------------ +Each carrier is read through a **marker-bounded extractor** rather than a whole-file +substring search, so "the rule is in the file" cannot be satisfied by a sentence +that wandered into an unrelated section. A region bound is no bound if it can widen +to EOF (a truncated end marker did exactly that once), so each clause is also held +to a fraction of its file. + +File-level reading is one level short of a round's prompt: a Jinja comment +(``{# … #}``) removes text from the render while the file keeps every character. +That shape was measured on this repo before (#1904: five load-bearing phrases still +in the file, every file-level assertion green, the render 2,301 characters shorter). +So every clause is extracted **twice** — from the file, and from the file with its +Jinja comments removed — and must survive both. Named limit: this is the comment +shape only; a ``{% if false %}`` block is not simulated here, and nothing in this +file shows that a round *follows* the rule, only that it is stated. + +The permission boundary is guarded in both directions too. The rule is about what +automation can do, never about what it may do: a step that spends money or changes +the host's account settings stays a stop-and-ask, and +``test_the_capability_rule_does_not_swallow_the_permission_stops`` fails if the +clause is rewritten into a licence — or if the iOS template's own explicit +exceptions are removed to "generalise" it. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +PROMPTS = Path(__file__).resolve().parent.parent / "emrg" / "server" + +OPEN_SOURCE = PROMPTS / "open_source_prompt.md" +COMPETITION = PROMPTS / "competition_prompt.md" +IOS_APP = PROMPTS / "ios_app_prompt.md" + +#: Where the clause starts and ends in `open_source_prompt.md` §3.1.6. Both are +#: single-phrase anchors so a reflow of the surrounding prose does not move them. +OPEN_SOURCE_START = "**The same rule covers the host's side of the ledger.**" +OPEN_SOURCE_END = "whatever the browser shows." + +#: The `blocked` shape in `competition_prompt.md`'s error-handling table. +COMPETITION_ROW_PREFIX = "| Needs a human action (real-name, SMS, card authorization)" + +#: The rule as parts a reader can point at. Each pattern accepts the wording either +#: carrier uses: the carriers state one requirement, not one sentence, and pinning a +#: single exact string per carrier would be a paste-lock rather than a guard. +RULE = ( + ( + "the account's real state is read in the real browser", + re.compile(r"\breal browser\b"), + ), + ( + "the probe without the login is described as logged-out", + re.compile(r"logged-out"), + ), + ( + "such a probe is stated not to be evidence about the account", + re.compile(r"not evidence about the account"), + ), + ( + "the fallback names the step that blocks", + re.compile(r"(?:names that step|the step that blocks)"), + ), +) + +#: The hand-back shape the competition row used to state, retired by this change. +RETIRED_HAND_BACK = "`blocked` with what the host must do" + +JINJA_COMMENT = re.compile(r"\{#.*?#\}", re.DOTALL) + + +def _normalised(text: str) -> str: + """Collapse whitespace, so a reflowed paragraph and a table row read alike.""" + return re.sub(r"\s+", " ", text).strip() + + +def _strip_jinja_comments(text: str) -> str: + return JINJA_COMMENT.sub(" ", text) + + +def _bound_clause(clause: str, source: str, whole: str) -> str: + """Hold an extracted clause inside its file, so a broken end marker cannot pass. + + A missing end marker would otherwise widen the region to EOF and let any later + sentence satisfy the rule. + """ + assert clause, f"{source}: the clause region came out empty" + assert len(clause) < len(_normalised(whole)) / 3, ( + f"{source}: the extracted clause is {len(clause)} chars of a " + f"{len(_normalised(whole))}-char file — the end marker did not bound it, so " + "this region is not the clause and proves nothing" + ) + return clause + + +def _open_source_clause(text: str) -> str: + flat = _normalised(text) + start = flat.find(OPEN_SOURCE_START) + assert start != -1, ( + f"open_source_prompt.md: the clause does not start with {OPEN_SOURCE_START!r}" + ) + end = flat.find(OPEN_SOURCE_END, start) + assert end != -1, ( + f"open_source_prompt.md: the clause no longer ends with {OPEN_SOURCE_END!r}" + ) + return _bound_clause(flat[start : end + len(OPEN_SOURCE_END)], "open_source_prompt.md", text) + + +def _competition_row(text: str) -> str: + for line in text.splitlines(): + if line.startswith(COMPETITION_ROW_PREFIX): + return _bound_clause( + _normalised(line), "competition_prompt.md (blocked row)", text + ) + raise AssertionError( + f"competition_prompt.md: no row starts with {COMPETITION_ROW_PREFIX!r} — the " + "host-blocked shape this rule is attached to is gone, so the rule is not " + "reached by the round that writes the row" + ) + + +#: One entry per carrier: the file, and the extractor that reads *the clause*. +CARRIERS = { + "open_source_prompt.md": (OPEN_SOURCE, _open_source_clause), + "competition_prompt.md": (COMPETITION, _competition_row), +} + + +def _clause(name: str) -> str: + """The clause, read at both levels: the committed file, and the render's text. + + Extraction is run on the file and again on the file with its Jinja comments + removed — a sentence inside ``{# … #}`` is in the file and not in the prompt, so + an extractor that only ever sees the raw file cannot tell those apart. + """ + path, extractor = CARRIERS[name] + assert path.is_file(), f"{path} is missing — a carrier cannot state the rule" + raw = path.read_text(encoding="utf-8") + clause = extractor(raw) + extractor(_strip_jinja_comments(raw)) + return clause + + +def _parts(name: str) -> set[str]: + clause = _clause(name) + return {label for label, pattern in RULE if pattern.search(clause)} + + +# ── the carriers are the two the issue names, and nothing silently drops one ── + + +def test_the_carriers_are_the_two_templates_the_rule_belongs_to(): + assert set(CARRIERS) == {"open_source_prompt.md", "competition_prompt.md"} + for name, (path, _extractor) in CARRIERS.items(): + assert path.is_file(), f"{name}: declared carrier missing at {path}" + + +# ── each carrier, on its own ──────────────────────────────────────────────── + + +@pytest.mark.parametrize("name", sorted(CARRIERS)) +@pytest.mark.parametrize("label,pattern", RULE, ids=[label for label, _ in RULE]) +def test_each_carrier_states_every_part_of_the_rule(name, label, pattern): + clause = _clause(name) + assert pattern.search(clause), ( + f"{name}: the clause no longer states that {label}. The clause reads:\n\n" + f"{clause}\n\n" + "A carrier that drops one part lets a round conclude a step is the host's " + "without checking it — which is the failure this rule exists to stop." + ) + + +# ── the two carriers, compared both ways ──────────────────────────────────── + + +def test_the_two_carriers_state_the_same_rule(): + open_source = _parts("open_source_prompt.md") + competition = _parts("competition_prompt.md") + + only_in_open_source = open_source - competition + only_in_competition = competition - open_source + + assert not only_in_open_source and not only_in_competition, ( + "the two carriers no longer state the same requirement.\n" + f" only in open_source_prompt.md: {sorted(only_in_open_source) or 'none'}\n" + f" only in competition_prompt.md: {sorted(only_in_competition) or 'none'}\n\n" + "Both templates hand a step back to the host, so both must carry the same " + "obligation: whichever side lost a part is the edit to finish." + ) + + +def test_the_competition_row_no_longer_states_a_bare_hand_back(): + clause = _clause("competition_prompt.md") + assert RETIRED_HAND_BACK not in clause, ( + "competition_prompt.md's blocked row is back to the bare hand-back shape " + f"({RETIRED_HAND_BACK!r}): it tells the round to record a wall without " + "looking at the account's actual state first." + ) + + +# ── the rule is about capability, never permission ────────────────────────── + + +def test_the_capability_rule_does_not_swallow_the_permission_stops(): + clause = _clause("open_source_prompt.md").lower() + for phrase in ("spend money", "account setting", "stop-and-ask"): + assert phrase in clause, ( + "open_source_prompt.md: the clause no longer names " + f"{phrase!r} as staying a stop-and-ask — without it the browser check " + "reads as a licence to act on the host's behalf." + ) + + +def test_the_ios_templates_own_exceptions_are_untouched(): + """The iOS template keeps its own two exceptions; this rule does not rewrite them. + + It is listed here because the cheapest way to "generalise" the rule is to delete + the exception it is not about. + """ + text = " ".join(IOS_APP.read_text(encoding="utf-8").split()) + # Read the exceptions as the two things they name, not as one exact sentence: + # the template bolds the word between them, so a sentence-level literal is a + # markup lock rather than a statement about the rule. + for phrase in ("money is spent", "account settings change"): + assert phrase in text, ( + f"ios_app_prompt.md: its stop-and-ask exception for {phrase!r} is gone. " + "Money and account settings are authorization boundaries, not capability " + "claims, and the browser rule does not touch them." + ) + + +# ── the reading itself is self-tested ─────────────────────────────────────── + + +def test_a_jinja_comment_over_the_clause_is_caught(): + """A comment hides the rule from the render while the file keeps every character. + + The device that catches it must be shown to fire, or a stripped-text reading + that silently stopped stripping would pass for ever. + """ + for name, (path, extractor) in CARRIERS.items(): + raw = path.read_text(encoding="utf-8") + assert extractor(raw), f"{name}: the extractor found nothing on the real file" + + clause = extractor(raw) + # Wrap the clause's own first words in a comment, the shape #1904 measured. + head = clause.split()[0] + index = raw.find(head) + assert index != -1 + hidden = raw[:index] + "{# " + raw[index:] + " #}" + # The file still carries the text … + extractor(hidden) + # … and the render does not, which is what the second reading catches. + with pytest.raises(AssertionError): + extractor(_strip_jinja_comments(hidden)) + + +def test_the_retired_hand_back_shape_is_absent_and_its_absence_is_meaningful(): + """Guard against a vacuous pass: an absence check needs the phrase to be real. + + A phrase that was never in the file is absent from it for ever, so "the retired + wording is gone" would be green on a template that never carried it. The subject + is named here instead: the competition template still speaks of what the host + must do, so the row this rule attaches to is the one being read. + """ + text = COMPETITION.read_text(encoding="utf-8") + assert "what the host must do" in text, ( + "competition_prompt.md no longer mentions what the host must do at all — the " + "blocked row this rule attaches to has been renamed or removed." + )