From 84461a91c39438faadb51b55165be0eab91b431a Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 10:31:38 +0200 Subject: [PATCH 1/9] Plan OCR 1.11.4 and 1.11.5 integration --- PLANS.md | 165 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) diff --git a/PLANS.md b/PLANS.md index 116f44f..8ec996f 100644 --- a/PLANS.md +++ b/PLANS.md @@ -5,3 +5,168 @@ before handoff or commit. Completed stable plans are indexed in [the execution-history archive](docs/engineering/execution_history/README.md). ## Active Work + +### Toolkit 0.9.1 — OCR 1.11.4 and 1.11.5 + +- **Status:** active +- **Plan Origin:** plan_mode_approved +- **Release classification:** release-required; stable delivery release-deferred +- **Target stable version:** 0.9.1 +- **Branch:** `codex/v0.9.1-ocr-1.11.5` + +#### Goal + +Integrate OCR 1.11.4 and 1.11.5 through a green published Draft PR, with exact +1.11.5 runtime support and precise diagnostic/DLP boundaries that preserve GitLab +summary publication. External configured qualification owns the later release decision. + +#### Requested Scope + +- Qualify #176 and #177 separately; retain adjacent semantic audits and evidence. +- Accept bounded private failure arguments without exposing them or degrading publication. +- Qualify serialized comment repair, Objective-C++ routing, grouped review and defaults. +- Update local OCR to checksum-verified 1.11.5, current docs, examples and changelog. +- Publish a green Draft with external qualification instructions; keep issues/milestone open. + +#### Requirement Traceability + +| Requirement | Outcome | Queue | Verification | +| --- | --- | --- | --- | +| REQ-001 | Exact OCR 1.11.5 with separate predecessor evidence | WQ-02, WQ-04 | assets, adjacent audits, real binary probes | +| REQ-002 | Bounded arguments never reach normalized/public diagnostics | WQ-03 | hostile parser and production projection tests | +| REQ-003 | Private arguments cannot change summary or publication DLP | WQ-03 | paired result/publication/summary regressions | +| REQ-004 | Comment repair and Objective-C++ routing are qualified | WQ-04 | deterministic real OCR gateway and rules previews | +| REQ-005 | Grouping, defaults, MCP and receipt boundaries remain correct | WQ-04 | compatibility and runtime tests | +| REQ-006 | Docs, decision flow, changelog and backlog reflect current behavior | WQ-05 | documentation checks and rendered Towncrier | +| REQ-007 | Green pushed Draft with truthful external qualification | WQ-06 | local/hosted gates and remote readback | + +#### Explicit Non-Goals + +No real LLM calls, merge, release PR, tags, package publication, or issue/milestone +closure. No new environment variables, MCP tools, receipt schemas, upstream repair +implementation, or runtime compatibility fallback. No OCR config/credentials or +user HOME changes. No separate Codex Security scan. + +#### Constraints + +Preserve zero runtime dependencies, Python 3.12–3.14, receipt v8 and action receipt +v3. Keep malformed additive diagnostics independent from authoritative review +publication. Preserve summary format, numeric tool/token reporting, DLP protection +of public findings/warnings/suggestions, and independent later-action decisions. +New tests receive purpose-focused docstrings and stay with existing owners. + +#### Inputs And Sources + +- Approved conversation plan and final DLP/summary acceptance clarification. +- GitHub #176/#177; hosted compatibility run 33962853525. +- Official adjacent comparisons 1.11.3→1.11.4 and 1.11.4→1.11.5. +- Canonical project principles, development/release guides, compatibility policy, + review decision flow and public configuration/security contracts. +- Engineering-workflow 0.9.1; repository ownership takes precedence over templates. + +#### User Decisions And Answers + +Delivery stops at a published green Draft. External configured environment owns +real model testing and confirmation. Local OCR may immediately advance to 1.11.5 +after plan materialization; all local binary checks use isolated HOME and no LLM. +Private failure arguments alone must not alter DLP counts, review status, findings +or GitLab summary. Public secrets remain subject to normal DLP filtering. + +#### Completed Baseline State + +Clean main at ae0a9ac4357b95349e73939d71b0a2a9e0e6d69d; toolkit 0.9.0 released, +next version 0.9.1, recommended OCR 1.11.3. Local OCR is 1.11.1. Both candidate +issues report compatible machine evidence; current parser rejects arguments as +an extra diagnostic field. Workflow audit found canonical owners and valid indexes. + +#### Current Work Queue + +| Queue | Status | Work | +| --- | --- | --- | +| WQ-01 | in_progress | Plan, milestone and branch ready; signed commit and Draft next | +| WQ-02 | done | Verified assets and safely updated local OCR 1.11.5 | +| WQ-03 | pending | Bounded failure arguments and DLP/summary regressions | +| WQ-04 | pending | Expanded real OCR qualification, adjacent evidence and final pins | +| WQ-05 | pending | Public docs, decision flow, changelog and backlog reconciliation | +| WQ-06 | pending | Final local gate, push, hosted checks and external Draft handoff | + +#### Locked Decisions + +- Optional arguments is an opaque string, including empty, capped at 32768 code + points and 131072 UTF-8 bytes under the existing whole-result bound. Wrong shape + or excess produces invalid diagnostic state, never loss of a valid review. +- Discard arguments before normalized details/rendering and publication DLP; + raw OCR artifacts remain private. Do not parse nested serialized payloads. +- Keep normal warning/DLP/later-action treatment of comment_args_repaired. +- Qualify both releases, promote only exact 1.11.5, and retain historical evidence. +- Current group terminology must distinguish group prompt ceiling from per-file + preselection filtering. Viewer marks carry no GitLab lifecycle authority. +- Existing scheduled discovery works and stays unchanged. + +#### Verification + +Focused parser/production projection tests compare equivalent results with/without +private arguments across summary, findings, DLP, token/tool counters and coverage. +Exercise malformed, oversized, secret/PII/control/Unicode arguments; genuine public +DLP filtering; verified-zero/nonzero/invalid/conflicting diagnostics; repaired and +rejected comment batches; rules .mm/.m; grouping and numeric/MCP contracts. +Before each signed logical commit: formatter, focused tests, complete diff and +trust-flow self-review, git diff --check. Once after implementation: quality/coverage +floors, lock, manifest/evidence, Towncrier, privacy and pinned Gitleaks. Hosted PR +owns OS/Python matrix, package checks, dependencies, Security and CodeQL. + +#### Latest Validation Results + +- 2026-09-06: current branch and version markers verified; plan approved; no edits + preceded this plan. Existing scheduled candidate run is successful. +- 2026-09-06: local Darwin arm64 OCR 1.11.5 verifies digest + c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9 against + GitHub and the verified upstream checksum file. Isolated version/help passed; + config, credentials and user HOME were preserved. Milestone v0.9.1 tracks #176/#177. + +#### Risks And Recovery + +Restore the old verified local binary if replacement validation fails. Failed probes +block promotion; retain their bounded evidence and classify the failing owner. +Never weaken summary/DLP/receipt gates to obtain green checks. Evidence-driven CI +fixes receive the same self-review and commit gates. Preserve temporary private +outputs until verification, then remove only task-owned files. + +#### Resume Point + +Finish WQ-01 coordination and WQ-02 local binary verification, then implement WQ-03. +After the initial Draft push, keep later implementation commits local until all +slices and final local validation complete. + +#### Plan Fidelity Check + +- [x] Requirements, sources, constraints and user decisions are preserved. +- [x] Every requirement maps to an ordered queue and verification owner. +- [x] Data/privacy boundaries, recovery and exact next action are explicit. + +#### Reconciliation Check + +- [x] Baseline and workflow owners verified. +- [ ] Code, tests, public docs, issues and Draft agree. +- [ ] Local/remote head and current hosted checks agree. + +#### Closure Gate + +- [ ] Implementation and required local/hosted checks complete. +- [ ] Draft open and mergeable; unresolved review threads absent. +- [ ] Issues and milestone open; external qualification checklist recorded. +- [ ] Worktree clean and resume point reflects external qualification. + +#### Post-Close Delivery + +External agent starts with exact Draft head/tree and checksum-verified OCR 1.11.5 +in isolated HOME, then runs production ocr review using configured LLM. Verify +MCP use, diagnostics/repair, grouping, findings/suggestions, privacy and cleanup. +Publish only bounded structural evidence, hashes and pass/fail. Owner confirmation +precedes the ordinary protected merge/release process. + +#### Handoff Notes + +Do not require intermediate OCR installation. Keep 1.11.4 audit distinct from final +1.11.5 support. Update issue checkboxes only for proven criteria. Stable release +remains deferred and the active plan remains available until later release closure. From 20b29f970e22000858b8bdd52b2cea5af6304ca1 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 10:35:51 +0200 Subject: [PATCH 2/9] Keep private OCR failure arguments outside publication DLP --- PLANS.md | 11 ++- changelog.d/176.bugfix.md | 1 + src/ocr_toolkit/review_runner.py | 17 ++++ tests/test_review_runner.py | 145 +++++++++++++++++++++++++++++++ 4 files changed, 171 insertions(+), 3 deletions(-) create mode 100644 changelog.d/176.bugfix.md diff --git a/PLANS.md b/PLANS.md index 8ec996f..0125de1 100644 --- a/PLANS.md +++ b/PLANS.md @@ -83,9 +83,9 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | Queue | Status | Work | | --- | --- | --- | -| WQ-01 | in_progress | Plan, milestone and branch ready; signed commit and Draft next | +| WQ-01 | done | Signed planning commit pushed; Draft PR #178 opened | | WQ-02 | done | Verified assets and safely updated local OCR 1.11.5 | -| WQ-03 | pending | Bounded failure arguments and DLP/summary regressions | +| WQ-03 | done | Bounded failure arguments and DLP/summary regressions | | WQ-04 | pending | Expanded real OCR qualification, adjacent evidence and final pins | | WQ-05 | pending | Public docs, decision flow, changelog and backlog reconciliation | | WQ-06 | pending | Final local gate, push, hosted checks and external Draft handoff | @@ -123,6 +123,11 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9 against GitHub and the verified upstream checksum file. Isolated version/help passed; config, credentials and user HOME were preserved. Milestone v0.9.1 tracks #176/#177. +- 2026-09-06: Draft #178 opened. Baseline real OCR 1.11.5 no-LLM contracts passed. + Diagnostic/parser/finalization/posting tests passed 236 tests and 94 subtests. + Paired clean/finding/warning/partial/filtered results produced identical persisted + projections, DLP state, console diagnostics and GitLab notes with/without private + arguments. Malformed and oversized arguments remain diagnostic-only degradation. #### Risks And Recovery @@ -134,7 +139,7 @@ outputs until verification, then remove only task-owned files. #### Resume Point -Finish WQ-01 coordination and WQ-02 local binary verification, then implement WQ-03. +Implement WQ-04 expanded real OCR probes and adjacent evidence promotion. After the initial Draft push, keep later implementation commits local until all slices and final local validation complete. diff --git a/changelog.d/176.bugfix.md b/changelog.d/176.bugfix.md new file mode 100644 index 0000000..98a9611 --- /dev/null +++ b/changelog.d/176.bugfix.md @@ -0,0 +1 @@ +Accept bounded private `failure_details.arguments` from OCR without treating otherwise valid failed-tool diagnostics as malformed. Arguments are omitted before normalized console output and publication DLP; they never enter finalized results, receipts, GitLab notes, or toolkit telemetry. Invalid diagnostics still preserve valid review findings and summary publication. Existing DLP checks on public findings, warnings, and suggestions and independent later-action restrictions remain in effect. diff --git a/src/ocr_toolkit/review_runner.py b/src/ocr_toolkit/review_runner.py index a7a107f..d6d2d55 100644 --- a/src/ocr_toolkit/review_runner.py +++ b/src/ocr_toolkit/review_runner.py @@ -184,6 +184,8 @@ MAX_TOOL_FAILURE_NAME_CHARS = 256 MAX_TOOL_FAILURE_PATH_CHARS = 4_096 MAX_TOOL_FAILURE_ERROR_CHARS = 32_768 +MAX_TOOL_FAILURE_ARGUMENT_CHARS = 32_768 +MAX_TOOL_FAILURE_ARGUMENT_BYTES = 131_072 MAX_TOOL_FAILURE_LOG_FIELD_CHARS = 500 @@ -900,6 +902,17 @@ def _safe_publication_warnings( return retained, len(value) - len(retained) +def _bounded_tool_failure_arguments(value: object) -> bool: + """Validate opaque private arguments without retaining or interpreting their content.""" + + if not isinstance(value, str) or len(value) > MAX_TOOL_FAILURE_ARGUMENT_CHARS: + return False + try: + return len(value.encode("utf-8")) <= MAX_TOOL_FAILURE_ARGUMENT_BYTES + except UnicodeEncodeError: + return False + + def _tool_failure_telemetry(value: object) -> ToolFailureTelemetry: """Parse additive OCR diagnostics without granting them result authority.""" @@ -954,8 +967,12 @@ def _tool_failure_telemetry(value: object) -> ToolFailureTelemetry: if not isinstance(detail, dict) or set(detail) not in ( {"tool_call_number", "tool_name", "error"}, {"tool_call_number", "tool_name", "file_path", "error"}, + {"tool_call_number", "tool_name", "arguments", "error"}, + {"tool_call_number", "tool_name", "file_path", "arguments", "error"}, ): return ToolFailureTelemetry(True, False, None, {}, ()) + if "arguments" in detail and not _bounded_tool_failure_arguments(detail["arguments"]): + return ToolFailureTelemetry(True, False, None, {}, ()) number = detail.get("tool_call_number") name = detail.get("tool_name") error = detail.get("error") diff --git a/tests/test_review_runner.py b/tests/test_review_runner.py index fad1f57..a4d3e1a 100644 --- a/tests/test_review_runner.py +++ b/tests/test_review_runner.py @@ -938,6 +938,30 @@ def test_generated_mr_receipt_is_validated_before_atomic_publication( {"failure": 1}, {"failure": True, "failure_by_tool": {}, "failure_details": []}, {"failure": -1, "failure_by_tool": {}, "failure_details": []}, + { + "failure": 1, + "failure_by_tool": {"file_read": 1}, + "failure_details": [ + { + "tool_call_number": 2, + "tool_name": "file_read", + "error": "Unavailable", + "arguments": {}, + } + ], + }, + { + "failure": 1, + "failure_by_tool": {"file_read": 1}, + "failure_details": [ + { + "tool_call_number": 2, + "tool_name": "file_read", + "error": "Unavailable", + "arguments": "x" * 32_769, + } + ], + }, { "failure": 1, "failure_by_tool": {"file_read": 1}, @@ -1077,6 +1101,127 @@ def test_valid_failure_details_are_console_only_and_keep_the_review_postable( assert "\x1b" not in diagnostics +@pytest.mark.parametrize("public_case", ["clean", "finding", "filtered", "warning", "partial"]) +@pytest.mark.parametrize( + "arguments", + [ + "", + '{"value":"private-argument-canary"}', + "person@example.invalid\x00\t\u202e", + "😀" * 32_768, + ], + ids=["empty", "private-value", "controls-pii", "utf8-limit"], +) +def test_private_failure_arguments_preserve_finalized_signal_and_gitlab_notes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, public_case: str, arguments: str +) -> None: + """Drop opaque arguments before DLP and preserve the complete posting projection.""" + + composition = MCPComposition( + payload={}, + capabilities=(MCPCapability("ocr_toolkit_evidence", BUILTIN_EVIDENCE_TOOLS, True),), + external_servers=(), + secret_values=(), + ) + notes: list[str] = [] + monkeypatch.setattr( + workflow, "collect_previous_bot_comment_refs", lambda _: snapshot.BotCommentRefs() + ) + monkeypatch.setattr(workflow, "get_diff_refs", lambda _: None) + monkeypatch.setattr( + workflow, + "post_review_note_bounded", + lambda _c, title, body, _t: notes.append(title + body) or {"id": len(notes)}, + ) + monkeypatch.setattr(workflow, "finalize_posting", lambda *_: True) + monkeypatch.setattr(workflow, "delete_previous_bot_comments_if_collected", lambda *_: None) + monkeypatch.setattr(workflow.secrets, "token_hex", lambda _: "a" * 32) + outputs = [] + for include_arguments in (False, True): + detail = {"tool_call_number": 2, "tool_name": "file_read", "error": "Unavailable"} + if include_arguments: + detail["arguments"] = arguments + payload = complete_review_payload( + tool_calls={ + "total": 2, + "by_tool": {"ocr_toolkit_evidence": 1, "file_read": 1}, + "failure": 1, + "failure_by_tool": {"file_read": 1}, + "failure_details": [detail], + } + ) + payload["token_usage"] = {"input_tokens": 100, "output_tokens": 20, "total_tokens": 120} + if public_case == "clean": + payload["comments"] = [] + elif public_case == "filtered": + payload["comments"][0]["content"] = "Contact person@example.invalid" + elif public_case == "warning": + payload["warnings"] = [ + {"code": "comment_args_repaired", "message": "Repaired serialized comments"} + ] + elif public_case == "partial": + payload["manifest"]["terminal_state"] = "partial" + coverage = payload["manifest"]["coverage"] + coverage["selected"].append({"item_id": "unfinished"}) + coverage["failed"].append( + {"item_id": "unfinished", "classification": "provider", "reason": "Unavailable"} + ) + payload["status"] = "partial" + result = tmp_path / f"result-{include_arguments}.json" + result.write_text(json.dumps(payload), encoding="utf-8") + log = io.StringIO() + with redirect_stderr(log): + _, filtered, publication = review_runner._finalize_ocr_result( + result, + composition, + DEFAULT_IDENTITY, + None, + SUMMARY_ACTION_COUNTS, + forbidden=("private-argument-canary",), + ) + persisted = json.loads(result.read_text(encoding="utf-8")) + serialized = result.read_text(encoding="utf-8") + notes.clear() + with patched_env(OCR_AUTO_APPROVE="false"): + assert workflow.post_results(gitlab_config(), persisted) == 0 + assert notes and "Technical details" in notes[-1] + assert "private-argument-canary" not in serialized + log.getvalue() + "".join(notes) + assert "failure_details" not in serialized + assert filtered is (public_case == "filtered") + outputs.append((serialized, publication, log.getvalue(), list(notes))) + assert outputs[0] == outputs[1] + + +@pytest.mark.parametrize( + "arguments", + [None, [], {}, 1, True, "x" * 32_769, "\ud800"], + ids=["null", "array", "object", "integer", "bool", "oversized", "surrogate"], +) +def test_invalid_private_failure_arguments_are_only_diagnostic_degradation( + arguments: object, +) -> None: + """Reject malformed or oversized private data without raising from the parser.""" + + telemetry = review_runner._tool_failure_telemetry( + { + "total": 1, + "by_tool": {"file_read": 1}, + "failure": 1, + "failure_by_tool": {"file_read": 1}, + "failure_details": [ + { + "tool_call_number": 1, + "tool_name": "file_read", + "error": "Unavailable", + "arguments": arguments, + } + ], + } + ) + assert telemetry.present and not telemetry.valid + assert telemetry.details == () + + def test_local_review_also_keeps_signal_when_failure_diagnostics_are_malformed( tmp_path: Path, ) -> None: From 02ed911e0897da81d913d6414e3e7fa1e3982e80 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 11:00:41 +0200 Subject: [PATCH 3/9] Refactor forward-only OCR qualification and adopt 1.11.5 --- PLANS.md | 25 +- README.md | 10 +- changelog.d/176.maintenance.md | 1 + changelog.d/176.refactor.md | 1 + changelog.d/176.rules.md | 1 + changelog.d/177.maintenance.md | 1 + compatibility/evidence/ocr-1.11.4.json | 257 ++++++++++ compatibility/evidence/ocr-1.11.5.json | 257 ++++++++++ compatibility/ocr-support.json | 110 ++++- docs/configuration.md | 20 +- docs/development.md | 22 + examples/gitlab/README.md | 2 +- examples/gitlab/ocr-review.gitlab-ci.yml | 6 +- scripts/ocr_compat.py | 586 +++++++++++++---------- scripts/ocr_compat_history.py | 147 ++++++ src/ocr_toolkit/preflight.py | 2 +- tests/test_environment_contract.py | 6 +- tests/test_ocr_compat.py | 202 ++++++-- tests/test_review_runner.py | 2 +- tests/test_runtime_helpers.py | 2 +- 20 files changed, 1323 insertions(+), 337 deletions(-) create mode 100644 changelog.d/176.maintenance.md create mode 100644 changelog.d/176.refactor.md create mode 100644 changelog.d/176.rules.md create mode 100644 changelog.d/177.maintenance.md create mode 100644 compatibility/evidence/ocr-1.11.4.json create mode 100644 compatibility/evidence/ocr-1.11.5.json create mode 100644 scripts/ocr_compat_history.py diff --git a/PLANS.md b/PLANS.md index 0125de1..531be61 100644 --- a/PLANS.md +++ b/PLANS.md @@ -39,6 +39,7 @@ summary publication. External configured qualification owns the later release de | REQ-005 | Grouping, defaults, MCP and receipt boundaries remain correct | WQ-04 | compatibility and runtime tests | | REQ-006 | Docs, decision flow, changelog and backlog reflect current behavior | WQ-05 | documentation checks and rendered Towncrier | | REQ-007 | Green pushed Draft with truthful external qualification | WQ-06 | local/hosted gates and remote readback | +| REQ-008 | Forward-only live qualification, isolated historical readback and stable tests | WQ-04R | no live version branches, frozen-history validation, CLI and probe regressions | #### Explicit Non-Goals @@ -86,8 +87,9 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-01 | done | Signed planning commit pushed; Draft PR #178 opened | | WQ-02 | done | Verified assets and safely updated local OCR 1.11.5 | | WQ-03 | done | Bounded failure arguments and DLP/summary regressions | -| WQ-04 | pending | Expanded real OCR qualification, adjacent evidence and final pins | -| WQ-05 | pending | Public docs, decision flow, changelog and backlog reconciliation | +| WQ-04 | done | Expanded real OCR qualification, adjacent evidence and final pins | +| WQ-04R | done | Forward-only live qualification, frozen historical readback and maintenance instructions | +| WQ-05 | in_progress | Public docs, decision flow, changelog and backlog reconciliation | | WQ-06 | pending | Final local gate, push, hosted checks and external Draft handoff | #### Locked Decisions @@ -102,6 +104,12 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index - Current group terminology must distinguish group prompt ceiling from per-file preselection filtering. Viewer marks carry no GitLab lifecycle authority. - Existing scheduled discovery works and stays unchanged. +- User requested a complete qualification-layer refactor during implementation. + Live probes target the current consumed contract without historical execution + branches or patch-specific fixtures. Historical evidence keeps its original + validation semantics in a separate owner. Promotion-policy tests use frozen + baselines; current pin tests alone assert the current version. Update canonical + development/compatibility instructions rather than adding duplicate agent rules. #### Verification @@ -128,6 +136,17 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. Paired clean/finding/warning/partial/filtered results produced identical persisted projections, DLP state, console diagnostics and GitLab notes with/without private arguments. Malformed and oversized arguments remain diagnostic-only degradation. +- 2026-09-06: expanded real-binary no-LLM suites passed for OCR 1.11.4 and 1.11.5. + Native/serialized/repaired two-comment batches preserve fields and anchors; + suspect truncation is rejected with raw arguments confined to private diagnostics. + Objective-C++ and both .m routing modes passed. Historical Linux asset proofs + remain linked to run 33962853525; new contract evidence explicitly names Darwin. +- 2026-09-06: full live qualification refactor removed historical execution branches + and text/grouping fallbacks, separated frozen evidence readback, and made + promotion validate current contracts before writes. Generic tests now use frozen + baselines. Complete real-binary JSON proof is identical before/after refactoring + for both releases. Focused suite passed 389 tests and 109 subtests; documentation + and metadata suite passed 163 tests. Canonical maintenance instructions updated. #### Risks And Recovery @@ -139,7 +158,7 @@ outputs until verification, then remove only task-owned files. #### Resume Point -Implement WQ-04 expanded real OCR probes and adjacent evidence promotion. +Finish WQ-05 public contract review, then perform WQ-06 final local/hosted gates. After the initial Draft push, keep later implementation commits local until all slices and final local validation complete. diff --git a/README.md b/README.md index 5d0d7c8..8c7f3cd 100644 --- a/README.md +++ b/README.md @@ -31,11 +31,11 @@ python -m pip install open-code-review-toolkit ``` The toolkit does not bundle OCR. Download the platform asset for exact -[Open Code Review 1.11.3](https://github.com/alibaba/open-code-review/releases/tag/v1.11.3), +[Open Code Review 1.11.5](https://github.com/alibaba/open-code-review/releases/tag/v1.11.5), verify it before installation against the [compatibility manifest](compatibility/ocr-support.json), and place the verified binary on `PATH`. The current Linux amd64 digest is -`9726204ac81baee153fd65b1ff357c380f73e9d8091c4a73c3c9fb541b5164cb`; Darwin arm64 is -`515cd92ce300b62dea2bcaf53e910cdf57b681e1d58bbd0a9e8e49617bca52fa`. The manifest owns +`53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95`; Darwin arm64 is +`c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9`. The manifest owns the other platform hashes. Verify the installation without contacting an LLM provider: @@ -45,14 +45,14 @@ ocr --version ocr-ci --help ``` -`ocr --version` must report `open-code-review v1.11.3`; `ocr-ci --help` must exit +`ocr --version` must report `open-code-review v1.11.5`; `ocr-ci --help` must exit successfully. `ocr-ci preflight` is an operational configuration check, not the installation smoke test. The exact recommended OCR release and its verified asset checksums live in the [versioned compatibility manifest](compatibility/ocr-support.json). CI should pin that release and checksum before execution. The [versioned compatibility policy](docs/compatibility.md) records tested assets and evidence and describes the conservative Dependabot-like qualification workflow for later upstream releases. Review output defaults to English. `OCR_REVIEW_LANGUAGE` accepts another explicit language name when a project needs localized review output; for example, `OCR_REVIEW_LANGUAGE=Russian`. -The current OCR 1.11.3 integration defaults `OCR_REVIEW_EFFORT` to `medium` for two review rounds. `low` and `high` are explicit one- and three-round alternatives; see the [configuration reference](docs/configuration.md#review-effort) for cost, budget, and precedence boundaries. +The current OCR 1.11.5 integration defaults `OCR_REVIEW_EFFORT` to `medium` for two review rounds. `low` and `high` are explicit one- and three-round alternatives; see the [configuration reference](docs/configuration.md#review-effort) for cost, budget, and precedence boundaries. Stable distributions are published to [PyPI](https://pypi.org/project/open-code-review-toolkit/) and mirrored as checksum-listed, provenance-attested assets in the corresponding [GitHub Release](https://github.com/xeonvs/open-code-review-toolkit/releases). Development snapshots are published only to TestPyPI. diff --git a/changelog.d/176.maintenance.md b/changelog.d/176.maintenance.md new file mode 100644 index 0000000..89ff8b6 --- /dev/null +++ b/changelog.d/176.maintenance.md @@ -0,0 +1 @@ +Qualify OCR 1.11.4 as the separately recorded predecessor to the final 1.11.5 target. Verify native comment-batch recovery and its repair warning through a deterministic gateway, while preserving finding fields, anchors, suggestions and rejection of suspect truncation. Upstream site and plugin-distribution changes do not add toolkit capabilities. See the separate diagnostic-parser fix and Objective-C++ Rules entries. diff --git a/changelog.d/176.refactor.md b/changelog.d/176.refactor.md new file mode 100644 index 0000000..c2decf4 --- /dev/null +++ b/changelog.d/176.refactor.md @@ -0,0 +1 @@ +Separate forward-only live OCR qualification from frozen historical evidence validation. Every candidate now runs the same current behavioral suite; legacy grouping/preview execution fallbacks are removed, promotion validates current contract evidence before writing, and generic promotion tests no longer track the latest runtime pin. Existing recorded evidence remains readable and unchanged. This changes repository qualification tooling, not the toolkit runtime API. diff --git a/changelog.d/176.rules.md b/changelog.d/176.rules.md new file mode 100644 index 0000000..c77aaa0 --- /dev/null +++ b/changelog.d/176.rules.md @@ -0,0 +1 @@ +Route Objective-C++ `.mm` files through OCR's built-in Objective-C rules. Content-dependent MATLAB/Objective-C routing for `.m` remains qualified. diff --git a/changelog.d/177.maintenance.md b/changelog.d/177.maintenance.md new file mode 100644 index 0000000..410af7c --- /dev/null +++ b/changelog.d/177.maintenance.md @@ -0,0 +1 @@ +Target checksum-verified OCR 1.11.5 as the only supported runtime for toolkit 0.9.1; install it directly without an intermediate OCR release. Separately qualify the 1.11.4 and 1.11.5 changes. Grouped-review naming now describes concurrency and prompt ceilings per group; per-file large-diff filtering, default medium effort, inherited completion cap, max-tools, result/manifest and receipt contracts remain unchanged. The OCR binary includes the grpc update; VS Code dependency updates and browser-only viewer fixed/ignored marks add no toolkit behavior or GitLab lifecycle authority. diff --git a/compatibility/evidence/ocr-1.11.4.json b/compatibility/evidence/ocr-1.11.4.json new file mode 100644 index 0000000..f029d1e --- /dev/null +++ b/compatibility/evidence/ocr-1.11.4.json @@ -0,0 +1,257 @@ +{ + "assets": [ + { + "name": "opencodereview-darwin-amd64", + "sha256": "09a57a9022071c19f82cec4d793bb8d93d3b308e0207bdd6b9ad823cbb1e5d17", + "size": 57803552 + }, + { + "name": "opencodereview-darwin-arm64", + "sha256": "36a29bb525f59ad88f39f6fb4a8bd76a67df5ed95399c92f3d10143274c91b15", + "size": 54844930 + }, + { + "name": "opencodereview-linux-amd64", + "sha256": "7133465774aff6330fd8d25eecc57a28e5852bc55d1ae55fd00443ebc4c6b023", + "size": 56058018 + }, + { + "name": "opencodereview-linux-arm64", + "sha256": "981964ac4466f1ed20129ae46b63a31a929219c1228a7a94c345cb4c9b8155d3", + "size": 52822178 + }, + { + "name": "opencodereview-windows-amd64.exe", + "sha256": "e9300d32877a1512e9571a5776c9173c2f7eccee58d2f7ef4aa54011bd70b9fb", + "size": 57519616 + }, + { + "name": "opencodereview-windows-arm64.exe", + "sha256": "e08fc1d3896509b1589046f6180b92b921cd540c113faff60a98d4f79198450d", + "size": 53482496 + }, + { + "name": "sha256sum.txt", + "sha256": "2dccf4dccaabcb0be94e43766316acbf7c8ab6f7309001575ee9e0cd4862a273", + "size": 572 + } + ], + "classification": "human-review-required", + "classification_reasons": [ + "release notes contain a material or ambiguous compatibility signal" + ], + "comparison_version": "1.11.3", + "contracts": { + "comment_arguments_probe": { + "comments_per_batch": 2, + "failure_arguments_preserved": true, + "intact_cases": [ + "array", + "serialized", + "repaired" + ], + "repair_warning": true, + "result": "passed", + "suspect_batch_rejected": true + }, + "comment_thinking_probe": { + "additive_field_preserved": true, + "posting_exposes_thinking": false, + "result": "passed" + }, + "completion_cap_probe": { + "explicit": 4096, + "inherited": 16384, + "result": "passed", + "wire_field": "max_completion_tokens" + }, + "language_rule_probe": { + "excluded_extensions": [ + ".svh" + ], + "extensions": [ + ".cjs", + ".cxx", + ".hxx", + ".mjs", + ".mm", + ".pug", + ".sv", + ".v", + ".vh", + ".vhd", + ".vhdl" + ], + "m_routing": "matlab_and_objective_c", + "result": "passed", + "rule_source": "system_builtin", + "selected": 11 + }, + "numeric_cli_probe": { + "max_tokens_budget": { + "cases": { + "invalid_below": { + "effective": null, + "input": -1, + "outcome": "rejected" + }, + "minimum": { + "effective": 1, + "input": 1, + "outcome": "accepted" + }, + "omitted": { + "effective": "unlimited", + "input": null, + "outcome": "accepted" + }, + "representative": { + "effective": 30000, + "input": 30000, + "outcome": "accepted" + }, + "sentinel": { + "effective": "unlimited", + "input": 0, + "outcome": "accepted" + } + }, + "maximum": null, + "owner": "ocr-cli" + }, + "max_tools": { + "cases": { + "invalid_below": { + "effective": null, + "input": -1, + "outcome": "rejected" + }, + "minimum": { + "effective": 100, + "input": 50, + "outcome": "accepted" + }, + "minimum_minus_one": { + "effective": 100, + "input": 49, + "outcome": "normalized", + "reported_normalization": 50 + }, + "omitted": { + "effective": 100, + "input": null, + "outcome": "accepted" + }, + "representative": { + "effective": 101, + "input": 101, + "outcome": "accepted" + }, + "sentinel": { + "effective": 100, + "input": 0, + "outcome": "accepted" + } + }, + "maximum": null, + "owner": "ocr-template-or-higher-cli", + "reported_minimum": 50 + }, + "result": "passed" + }, + "optional_capabilities": [ + "llm_result_identity", + "per_run_model_override", + "per_run_provider_override", + "review_effort", + "semantic_grouping" + ], + "preview_probe": { + "format": "json", + "path": "example.py", + "result": "passed", + "session_store_created": false + }, + "required_review_flags": [ + "--audience", + "--background-file", + "--effort", + "--format", + "--from", + "--max-tokens-budget", + "--max-tools", + "--preview", + "--rule", + "--to" + ], + "result_contract_probe": { + "additive_fields_allowed": true, + "comment_fields": [ + "category", + "content", + "end_line", + "existing_code", + "path", + "severity", + "start_line", + "thinking" + ], + "manifest_schema": "ocr.run-manifest/v1", + "normalized_outcome": "clean", + "result": "passed" + }, + "review_budget_probe": { + "budget": 30000, + "completed": 2, + "failed_budget": 1, + "grouping_requests": 0, + "grouping_strategy": "per_file", + "partial_findings_preserved": true, + "result": "passed", + "selected": 3 + }, + "semantic_grouping_probe": { + "default_effort": "medium", + "files": 4, + "filter_requests": 1, + "grouping_completion_cap": 16384, + "grouping_requests": 1, + "main_requests": 3, + "prior_finding_semantics": "filter_survivors_as_confirmed", + "recheck_instruction_requests": 3, + "result": "passed", + "review_rounds": 2 + }, + "small_change_grouping_probe": { + "grouping_requests": 0, + "high_churn": "per_file", + "low_churn": "bundle_all", + "result": "passed", + "single_file": "per_file", + "threshold_files": 4 + }, + "target_rule_selection_probe": { + "format": "json", + "from_to_unchanged": true, + "path": "synthetic-template.ocrfixture", + "result": "passed", + "source_exclusion": "unsupported_ext", + "target_selected": true + }, + "version_probe": "passed" + }, + "published_at": "2026-09-04T12:38:24Z", + "qualification_provenance": { + "asset_verification_run": "https://github.com/xeonvs/open-code-review-toolkit/actions/runs/33962853525", + "contract_binary_sha256": "36a29bb525f59ad88f39f6fb4a8bd76a67df5ed95399c92f3d10143274c91b15", + "contract_platform": "darwin-arm64" + }, + "release_changes": "## 🚀 Features\n\n- feat(tool): include arguments in failure details (#1152)\n\n## 🐛 Bug Fixes\n\n- fix(rules): route Objective-C++ files to Objective-C rules (#1166)\n- fix(tool): repair serialized code_comment arguments instead of losing the batch (#1153)\n\n## Other Changes\n\n- Keep repeated documentation headings independently addressable (#1151)\n- ci: add plugin distribution contract guardrails (#1117)\n\n**Full Changelog**: https://github.com/alibaba/open-code-review/compare/v1.11.3...v1.11.4", + "release_notes_sha256": "81f1a6a8b329e688b2575f69b779dc9593e407763783ffbff47cdcacb86febff", + "result": "compatible", + "schema_version": 3, + "tag": "v1.11.4", + "tested_baseline_version": "1.11.3", + "upstream_repository": "alibaba/open-code-review", + "version": "1.11.4" +} diff --git a/compatibility/evidence/ocr-1.11.5.json b/compatibility/evidence/ocr-1.11.5.json new file mode 100644 index 0000000..4b68705 --- /dev/null +++ b/compatibility/evidence/ocr-1.11.5.json @@ -0,0 +1,257 @@ +{ + "assets": [ + { + "name": "opencodereview-darwin-amd64", + "sha256": "4b7cde2c86d6f6967b3fd43c9401326e11c63cdb336252b2e5a528591fe474b2", + "size": 57815904 + }, + { + "name": "opencodereview-darwin-arm64", + "sha256": "c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9", + "size": 54844978 + }, + { + "name": "opencodereview-linux-amd64", + "sha256": "53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95", + "size": 56070306 + }, + { + "name": "opencodereview-linux-arm64", + "sha256": "ec10e788463ad405a7868d3803cb52d64fe8defb1934e70007165b3be3f222b2", + "size": 52822178 + }, + { + "name": "opencodereview-windows-amd64.exe", + "sha256": "667164fc0d58f711dbb04c4aa4f7c601174d10bb25f4041f6626e69f7e0e420e", + "size": 57533952 + }, + { + "name": "opencodereview-windows-arm64.exe", + "sha256": "34c31dd218e47ba9ab8bc5d6fd57d2fa39fd70e33aa6b890ab0f38922f9ba166", + "size": 53496320 + }, + { + "name": "sha256sum.txt", + "sha256": "0519c13b03d69dd6c4aa5470a8eb52727f5e3fd2ce43e5887d8a67cdf107aa1a", + "size": 572 + } + ], + "classification": "human-review-required", + "classification_reasons": [ + "release notes contain a material or ambiguous compatibility signal" + ], + "comparison_version": "1.11.4", + "contracts": { + "comment_arguments_probe": { + "comments_per_batch": 2, + "failure_arguments_preserved": true, + "intact_cases": [ + "array", + "serialized", + "repaired" + ], + "repair_warning": true, + "result": "passed", + "suspect_batch_rejected": true + }, + "comment_thinking_probe": { + "additive_field_preserved": true, + "posting_exposes_thinking": false, + "result": "passed" + }, + "completion_cap_probe": { + "explicit": 4096, + "inherited": 16384, + "result": "passed", + "wire_field": "max_completion_tokens" + }, + "language_rule_probe": { + "excluded_extensions": [ + ".svh" + ], + "extensions": [ + ".cjs", + ".cxx", + ".hxx", + ".mjs", + ".mm", + ".pug", + ".sv", + ".v", + ".vh", + ".vhd", + ".vhdl" + ], + "m_routing": "matlab_and_objective_c", + "result": "passed", + "rule_source": "system_builtin", + "selected": 11 + }, + "numeric_cli_probe": { + "max_tokens_budget": { + "cases": { + "invalid_below": { + "effective": null, + "input": -1, + "outcome": "rejected" + }, + "minimum": { + "effective": 1, + "input": 1, + "outcome": "accepted" + }, + "omitted": { + "effective": "unlimited", + "input": null, + "outcome": "accepted" + }, + "representative": { + "effective": 30000, + "input": 30000, + "outcome": "accepted" + }, + "sentinel": { + "effective": "unlimited", + "input": 0, + "outcome": "accepted" + } + }, + "maximum": null, + "owner": "ocr-cli" + }, + "max_tools": { + "cases": { + "invalid_below": { + "effective": null, + "input": -1, + "outcome": "rejected" + }, + "minimum": { + "effective": 100, + "input": 50, + "outcome": "accepted" + }, + "minimum_minus_one": { + "effective": 100, + "input": 49, + "outcome": "normalized", + "reported_normalization": 50 + }, + "omitted": { + "effective": 100, + "input": null, + "outcome": "accepted" + }, + "representative": { + "effective": 101, + "input": 101, + "outcome": "accepted" + }, + "sentinel": { + "effective": 100, + "input": 0, + "outcome": "accepted" + } + }, + "maximum": null, + "owner": "ocr-template-or-higher-cli", + "reported_minimum": 50 + }, + "result": "passed" + }, + "optional_capabilities": [ + "llm_result_identity", + "per_run_model_override", + "per_run_provider_override", + "review_effort", + "semantic_grouping" + ], + "preview_probe": { + "format": "json", + "path": "example.py", + "result": "passed", + "session_store_created": false + }, + "required_review_flags": [ + "--audience", + "--background-file", + "--effort", + "--format", + "--from", + "--max-tokens-budget", + "--max-tools", + "--preview", + "--rule", + "--to" + ], + "result_contract_probe": { + "additive_fields_allowed": true, + "comment_fields": [ + "category", + "content", + "end_line", + "existing_code", + "path", + "severity", + "start_line", + "thinking" + ], + "manifest_schema": "ocr.run-manifest/v1", + "normalized_outcome": "clean", + "result": "passed" + }, + "review_budget_probe": { + "budget": 30000, + "completed": 2, + "failed_budget": 1, + "grouping_requests": 0, + "grouping_strategy": "per_file", + "partial_findings_preserved": true, + "result": "passed", + "selected": 3 + }, + "semantic_grouping_probe": { + "default_effort": "medium", + "files": 4, + "filter_requests": 1, + "grouping_completion_cap": 16384, + "grouping_requests": 1, + "main_requests": 3, + "prior_finding_semantics": "filter_survivors_as_confirmed", + "recheck_instruction_requests": 3, + "result": "passed", + "review_rounds": 2 + }, + "small_change_grouping_probe": { + "grouping_requests": 0, + "high_churn": "per_file", + "low_churn": "bundle_all", + "result": "passed", + "single_file": "per_file", + "threshold_files": 4 + }, + "target_rule_selection_probe": { + "format": "json", + "from_to_unchanged": true, + "path": "synthetic-template.ocrfixture", + "result": "passed", + "source_exclusion": "unsupported_ext", + "target_selected": true + }, + "version_probe": "passed" + }, + "published_at": "2026-09-05T04:23:21Z", + "qualification_provenance": { + "asset_verification_run": "https://github.com/xeonvs/open-code-review-toolkit/actions/runs/33962853525", + "contract_binary_sha256": "c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9", + "contract_platform": "darwin-arm64" + }, + "release_changes": "## 🚀 Features\n\n- feat(viewer): mark comments as fixed/ignored and hide them (#1101)\n\n## 🐛 Bug Fixes\n\n- fix(deps): bump grpc, fast-uri and browserslist to patch high-severity advisories (#1169)\n\n## 🔧 Refactoring\n\n- refactor: retire per-file naming left over from grouped review (#1168)\n\n**Full Changelog**: https://github.com/alibaba/open-code-review/compare/v1.11.4...v1.11.5", + "release_notes_sha256": "09b53e15c3bf994ebabaedcfa044e053b583126e77e2165f1d29c0e48bac3de7", + "result": "compatible", + "schema_version": 3, + "tag": "v1.11.5", + "tested_baseline_version": "1.11.3", + "upstream_repository": "alibaba/open-code-review", + "version": "1.11.5" +} diff --git a/compatibility/ocr-support.json b/compatibility/ocr-support.json index ede7f00..5386221 100644 --- a/compatibility/ocr-support.json +++ b/compatibility/ocr-support.json @@ -1,6 +1,6 @@ { - "monitoring_floor": "1.11.3", - "recommended_version": "1.11.3", + "monitoring_floor": "1.11.5", + "recommended_version": "1.11.5", "releases": [ { "assets": [ @@ -1505,6 +1505,112 @@ "release_url": "https://github.com/alibaba/open-code-review/releases/tag/v1.11.3", "status": "tested", "version": "1.11.3" + }, + { + "assets": [ + { + "name": "opencodereview-darwin-amd64", + "sha256": "09a57a9022071c19f82cec4d793bb8d93d3b308e0207bdd6b9ad823cbb1e5d17", + "size": 57803552 + }, + { + "name": "opencodereview-darwin-arm64", + "sha256": "36a29bb525f59ad88f39f6fb4a8bd76a67df5ed95399c92f3d10143274c91b15", + "size": 54844930 + }, + { + "name": "opencodereview-linux-amd64", + "sha256": "7133465774aff6330fd8d25eecc57a28e5852bc55d1ae55fd00443ebc4c6b023", + "size": 56058018 + }, + { + "name": "opencodereview-linux-arm64", + "sha256": "981964ac4466f1ed20129ae46b63a31a929219c1228a7a94c345cb4c9b8155d3", + "size": 52822178 + }, + { + "name": "opencodereview-windows-amd64.exe", + "sha256": "e9300d32877a1512e9571a5776c9173c2f7eccee58d2f7ef4aa54011bd70b9fb", + "size": 57519616 + }, + { + "name": "opencodereview-windows-arm64.exe", + "sha256": "e08fc1d3896509b1589046f6180b92b921cd540c113faff60a98d4f79198450d", + "size": 53482496 + }, + { + "name": "sha256sum.txt", + "sha256": "2dccf4dccaabcb0be94e43766316acbf7c8ab6f7309001575ee9e0cd4862a273", + "size": 572 + } + ], + "capabilities": [ + "llm_result_identity", + "per_run_model_override", + "per_run_provider_override", + "review_effort", + "semantic_grouping" + ], + "evidence": "compatibility/evidence/ocr-1.11.4.json", + "evidence_sha256": "0ffd6f968f27a7e67a7abf034db3689292e546866ec38949f94e11304067e3cf", + "human_conclusion": "Reviewed adjacent source changes and verified hosted assets plus expanded Darwin no-LLM contracts: private failure arguments require toolkit parser adaptation; lossless comment repair preserves batch fields and rejects suspect truncation; Objective-C++ uses Objective-C rules. Site and plugin changes are upstream-only. Diagnostics remain independent from publication; no LLM quality claim.", + "published_at": "2026-09-04T12:38:24Z", + "release_url": "https://github.com/alibaba/open-code-review/releases/tag/v1.11.4", + "status": "tested", + "version": "1.11.4" + }, + { + "assets": [ + { + "name": "opencodereview-darwin-amd64", + "sha256": "4b7cde2c86d6f6967b3fd43c9401326e11c63cdb336252b2e5a528591fe474b2", + "size": 57815904 + }, + { + "name": "opencodereview-darwin-arm64", + "sha256": "c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9", + "size": 54844978 + }, + { + "name": "opencodereview-linux-amd64", + "sha256": "53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95", + "size": 56070306 + }, + { + "name": "opencodereview-linux-arm64", + "sha256": "ec10e788463ad405a7868d3803cb52d64fe8defb1934e70007165b3be3f222b2", + "size": 52822178 + }, + { + "name": "opencodereview-windows-amd64.exe", + "sha256": "667164fc0d58f711dbb04c4aa4f7c601174d10bb25f4041f6626e69f7e0e420e", + "size": 57533952 + }, + { + "name": "opencodereview-windows-arm64.exe", + "sha256": "34c31dd218e47ba9ab8bc5d6fd57d2fa39fd70e33aa6b890ab0f38922f9ba166", + "size": 53496320 + }, + { + "name": "sha256sum.txt", + "sha256": "0519c13b03d69dd6c4aa5470a8eb52727f5e3fd2ce43e5887d8a67cdf107aa1a", + "size": 572 + } + ], + "capabilities": [ + "llm_result_identity", + "per_run_model_override", + "per_run_provider_override", + "review_effort", + "semantic_grouping" + ], + "evidence": "compatibility/evidence/ocr-1.11.5.json", + "evidence_sha256": "18b56303554d3d8eee89f3f8db17c3d491f5f2bb990f5ffa412d64db685d1ada", + "human_conclusion": "Reviewed adjacent source changes and verified hosted assets plus expanded Darwin no-LLM contracts. Grouped-review renaming preserves consumed runtime behavior; grpc updates the binary dependency, while VS Code packages and browser-only viewer marks do not add toolkit surfaces. Exact 1.11.5 is the deployment target; configured LLM qualification remains pending.", + "published_at": "2026-09-05T04:23:21Z", + "release_url": "https://github.com/alibaba/open-code-review/releases/tag/v1.11.5", + "status": "tested", + "version": "1.11.5" } ], "schema_version": 1, diff --git a/docs/configuration.md b/docs/configuration.md index ec81643..cac230e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,7 +22,7 @@ These are the complete supported toolkit-owned runtime inputs. `Required` is sco | `OCR_LLM_VALIDATE_MODEL` | Operator / `ocr-ci preflight` | No | `false` | `true` validates through `/models`; `auto` may use the offline allowlist; false values skip validation. | | `OCR_LLM_MODELS_URL` | Operator / `ocr-ci preflight` | No | Derived from `OCR_LLM_URL` | Explicit absolute credential-free HTTPS metadata URL when validation is enabled or inference query parameters make derivation ambiguous. | | `OCR_LLM_ALLOWED_MODELS` | Operator / `ocr-ci preflight` | No | Empty list | Comma-separated exact model identifiers for offline or `auto` validation. | -| `OCR_TELEMETRY_ENABLED` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR telemetry configuration; OCR 1.11.3 spans may include path-derived group keys, model-produced labels, and local grouping decisions. | +| `OCR_TELEMETRY_ENABLED` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR telemetry configuration; OCR 1.11.5 spans may include path-derived group keys, model-produced labels, and local grouping decisions. | | `OCR_TELEMETRY_CONTENT_LOGGING` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR content logging; keep disabled for private review data. | | `OCR_TELEMETRY_EXPORTER` | Operator / `ocr-ci configure` | No | Empty string | Exporter name written only when telemetry is enabled. | | `OCR_TELEMETRY_OTLP_ENDPOINT` | Operator / `ocr-ci configure` | No | Unset | OTLP endpoint written only when telemetry is enabled and the value is non-empty. | @@ -60,13 +60,13 @@ If `OCR_LLM_EXTRA_BODY` already owns that field, an exactly equal JSON integer i The toolkit does not derive this value from `/models.max_completion_tokens`. That metadata is a model capability boundary, not an account spending limit or proof of how a gateway reserves request cost. -The inherited value is version-owned and therefore changes with a qualified OCR upgrade. The toolkit observed `max_completion_tokens=58888` from OCR 1.9.10 and `16384` from OCR 1.10.0 through 1.11.3 when the variable was unset. Grouping requests use the same template-owned cap; an explicit toolkit override still applies to every protocol request. Deployments that require an invariant gateway-specific cap must set `OCR_LLM_MAX_COMPLETION_TOKENS` explicitly rather than depending on an OCR default. +The inherited value is version-owned and therefore changes with a qualified OCR upgrade. The toolkit observed `max_completion_tokens=58888` from OCR 1.9.10 and `16384` from OCR 1.10.0 through 1.11.5 when the variable was unset. Grouping requests use the same template-owned cap; an explicit toolkit override still applies to every protocol request. Deployments that require an invariant gateway-specific cap must set `OCR_LLM_MAX_COMPLETION_TOKENS` explicitly rather than depending on an OCR default. ### Review effort -`OCR_REVIEW_EFFORT` defaults to `medium` and is written to OCR's root `effort` configuration key. OCR 1.11.3 maps `low`, `medium`, and `high` to one, two, and three review rounds and scales its 15-minute subtask base to 15, 30, or 45 minutes. The environment is operator-owned; merge-request text cannot change it. An explicit caller `--effort` passed after `ocr-ci review --` has normal OCR CLI precedence over the generated config, while an unknown environment value fails configuration before preview or model execution. +`OCR_REVIEW_EFFORT` defaults to `medium` and is written to OCR's root `effort` configuration key. OCR 1.11.5 maps `low`, `medium`, and `high` to one, two, and three review rounds and scales its 15-minute subtask base to 15, 30, or 45 minutes. The environment is operator-owned; merge-request text cannot change it. An explicit caller `--effort` passed after `ocr-ci review --` has normal OCR CLI precedence over the generated config, while an unknown environment value fails configuration before preview or model execution. -OCR 1.11.3 may present filter-surviving comments to a later round as previously confirmed, but the toolkit does not accept that wording as validation. Its mandatory background prefix travels with every main request and requires prior/filter-surviving findings to remain unverified until current code, tests, or trusted evidence support them. Survival cannot change severity, suppress or resolve a finding, authorize approval, or enter a receipt as independent validation. +OCR 1.11.5 may present filter-surviving comments to a later round as previously confirmed, but the toolkit does not accept that wording as validation. Its mandatory background prefix travels with every main request and requires prior/filter-surviving findings to remain unverified until current code, tests, or trusted evidence support them. Survival cannot change severity, suppress or resolve a finding, authorize approval, or enter a receipt as independent validation. Effort controls review depth, not the prompt/context ceiling, per-call completion cap, aggregate token budget, or per-round tool limit. Semantic grouping and filtering can add requests even at `low`; higher effort can add further rounds until OCR stops early, reaches a coverage/budget boundary, or completes the configured depth. @@ -95,15 +95,19 @@ These names belong to `examples/gitlab/ocr-review.gitlab-ci.yml`; they are shell | Variable | Source / owner | Required | Exact default | Behavior | | --- | --- | --- | --- | --- | -| **`OCR_VERSION`** | Example pipeline | Yes | `v1.11.3` | Checksum-pinned recommended OCR binary release for toolkit 0.9.0. | -| **`OCR_SHA256`** | Example pipeline | Yes | `9726204ac81baee153fd65b1ff357c380f73e9d8091c4a73c3c9fb541b5164cb` | Expected Linux AMD64 OCR binary digest. | +| **`OCR_VERSION`** | Example pipeline | Yes | `v1.11.5` | Checksum-pinned recommended OCR binary release for toolkit 0.9.1. | +| **`OCR_SHA256`** | Example pipeline | Yes | `53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95` | Expected Linux AMD64 OCR binary digest. | | **`OCR_TOOLKIT_VERSION`** | Example pipeline | Yes | `0.9.0` | Exact toolkit wheel release installed by the current published example. | | **`OCR_TOOLKIT_CHECKSUMS_URL`** | Example pipeline | Yes | Release URL derived from `OCR_TOOLKIT_VERSION` | Toolkit `SHA256SUMS` URL. | | `OCR_TOOLKIT_WHEEL` | Example shell | Computed | `open_code_review_toolkit-${OCR_TOOLKIT_VERSION}-py3-none-any.whl` | Exact wheel filename selected from the release. | | `OCR_TOOLKIT_WHEEL_SHA256` | Example shell | Computed | Matching value from `SHA256SUMS` | Digest checked before installing the toolkit wheel. | -| `OCR_MAX_TOOLS` | Example pipeline / OCR CLI | No | `0` | OCR 1.11.3 uses template default `100`; `1-49` reports normalization to `50` but remains effectively `100`, and only a value above `100` raises the cap. | +| `OCR_MAX_TOOLS` | Example pipeline / OCR CLI | No | `0` | OCR 1.11.5 uses template default `100`; `1-49` reports normalization to `50` but remains effectively `100`, and only a value above `100` raises the cap. | | `OCR_MAX_TOKENS_BUDGET` | Example pipeline / OCR CLI | No | `0` | Non-negative aggregate OCR token ceiling; `0` is unlimited. | +During Draft qualification, install the toolkit artifact built from the reviewed +commit. The example's published toolkit-version pin is advanced by the later +release PR; published toolkit 0.9.0 does not support the new OCR runtime. + ## Dynamic adapter and MCP inputs | Variable | Source / owner | Required | Exact default | Behavior | @@ -177,7 +181,7 @@ finish and OCR accounts the provider-reported input plus output tokens. `OCR_MAX_TOOLS` is independent of all three token controls. Leave the example default at `0` so OCR uses its embedded template limit of `100` per subtask. -OCR 1.11.3 reports values `1-49` as normalized to the minimum `50`, but both +OCR 1.11.5 reports values `1-49` as normalized to the minimum `50`, but both that target and explicit `50` remain below the template default and therefore remain effectively `100`. Use `101` or greater only when deliberately raising the loop cap. OCR 1.11.0 corrected stale help text for this already-qualified diff --git a/docs/development.md b/docs/development.md index 536fb69..9dfe987 100644 --- a/docs/development.md +++ b/docs/development.md @@ -25,6 +25,28 @@ For package, executable-integration, or release-machinery changes, install the w GitHub Actions storage is repository-owned infrastructure. Pull-request CI restores setup-uv caches but does not save branch-specific entries; protected-main publication may refresh shared dependency state. CodeQL TRAP caching and the separately controlled v4 overlay-database mode are disabled, so the small repository receives a full analysis without per-run CodeQL cache writes. Workflow artifacts use a seven-day handoff window. The weekly **Actions storage maintenance** workflow grants `actions: write` only to its cleanup job and deletes all CodeQL caches, non-main or superseded setup-uv caches, superseded Gitleaks caches, artifacts older than seven days, ordinary logs older than 14 days, and release/TestPyPI logs older than 30 days. It deletes completed TestPyPI preview runs after 14 days, TestPyPI development and ordinary completed runs after 30 days, and stable Release runs after 60 days; deleting a run removes that run's metadata, logs, and check metadata, so a workflow run is never removed before its separately promised log window. Active and newer runs remain untouched. The scheduled collector reads a closed 74-day UTC window in daily shards, retaining a fail-closed ten-page limit per day instead of applying that limit to the aggregate run history. Scheduled log cleanup uses a bounded 14-day retry window so immutable run history does not get scanned and retried forever. Manual dispatch is a dry run unless `execute` is selected; the same plan is available locally with `python scripts/actions_cleanup.py`, requires `--execute` for deletion, and accepts `--include-all-old-logs` for a deliberate one-time historical log cleanup. +## Maintaining OCR qualification + +The live suite in `scripts/ocr_compat.py` qualifies the current toolkit-consumed +OCR contract for every candidate. Do not add release-number branches, old parser +fallbacks, or patch-specific fixture copies to live probes. Version checks belong +to binary identity, release ordering, classification and promotion, not to choosing +which behavioral checks execute. Change a probe only when a consumed upstream +contract actually changes, and document the concrete before/after behavior. + +`scripts/ocr_compat_history.py` owns frozen validation of evidence from before the +forward-only suite boundary. It never launches OCR and must not depend on current +numeric defaults, language inventories or live fixtures. Preserve historical +evidence bytes. Generic promotion tests use a frozen baseline; only current-pin +tests assert the latest supported version. Current promotion validates all required +contract evidence before writing any pin or evidence file. + +Qualification uses one existing deterministic gateway and bounded observations. +Do not copy OCR's implementation, add another configuration framework, or turn +unconsumed upstream wording/features into mandatory checks. Prove preservation +with real no-LLM OCR runs and adversarial verifier tests. Separate those proofs +from model-quality qualification in a configured external environment. + ## Planning and documentation lifecycle `PLANS.md` contains complete active or blocked repository work, including release classification, target version, service boundaries, validation, and exact resume state. Before a logical commit, update the plan and every directly affected status-bearing document to describe post-commit truth. A milestone closes only after current implementation and tests prove its own outcome; reconcile the roadmap table and diagram, backlog, strategy, and README without deleting unfinished adjacent scope. diff --git a/examples/gitlab/README.md b/examples/gitlab/README.md index 84553d2..fb272db 100644 --- a/examples/gitlab/README.md +++ b/examples/gitlab/README.md @@ -2,7 +2,7 @@ [`ocr-review.gitlab-ci.yml`](ocr-review.gitlab-ci.yml) is the complete checksum-pinned pipeline. It defaults to identity-only review, explicitly keeps `OCR_GITLAB_TARGET_PROTECTION_MODE: "required"`, and contains the install, preflight, configure, review, and post lifecycle. Copy one mode file below into the pipeline's top-level `variables` mapping, or include exactly one file after reviewing its trust boundary. -The pipeline pins OCR 1.11.3 and explicitly sets `OCR_REVIEW_EFFORT=medium`, which permits two review rounds. Use `low` for one round when latency/cost is the priority, or `high` for three only after accepting the additional provider work. OCR uses a 15-minute subtask base and scales it to 15/30/45 minutes for low/medium/high; the example's 45-minute job timeout accommodates every preset without adding another environment control. Semantic grouping and filtering are OCR behavior shared by every context mode; they do not change which merge-request text a mode admits. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; only values above `100` raise the per-subtask cap. `OCR_MAX_TOKENS_BUDGET` and the optional completion cap remain independent controls. +The pipeline pins OCR 1.11.5 and explicitly sets `OCR_REVIEW_EFFORT=medium`, which permits two review rounds. Use `low` for one round when latency/cost is the priority, or `high` for three only after accepting the additional provider work. OCR uses a 15-minute subtask base and scales it to 15/30/45 minutes for low/medium/high; the example's 45-minute job timeout accommodates every preset without adding another environment control. Semantic grouping and filtering are OCR behavior shared by every context mode; they do not change which merge-request text a mode admits. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; only values above `100` raise the per-subtask cap. `OCR_MAX_TOKENS_BUDGET` and the optional completion cap remain independent controls. | Mode | Recipe | MR text admitted | External access | Automatic approval | | --- | --- | --- | --- | --- | diff --git a/examples/gitlab/ocr-review.gitlab-ci.yml b/examples/gitlab/ocr-review.gitlab-ci.yml index a2e0f37..867a86f 100644 --- a/examples/gitlab/ocr-review.gitlab-ci.yml +++ b/examples/gitlab/ocr-review.gitlab-ci.yml @@ -6,10 +6,10 @@ default: image: python:3.12-slim variables: - OCR_VERSION: "v1.11.3" + OCR_VERSION: "v1.11.5" OCR_TOOLKIT_VERSION: "0.9.0" OCR_TOOLKIT_CHECKSUMS_URL: "https://github.com/xeonvs/open-code-review-toolkit/releases/download/v${OCR_TOOLKIT_VERSION}/SHA256SUMS" - OCR_SHA256: "9726204ac81baee153fd65b1ff357c380f73e9d8091c4a73c3c9fb541b5164cb" + OCR_SHA256: "53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95" OCR_POST_MODE: "draft" OCR_STRICT_POSTING: "true" # Secure default: reject an unprotected GitLab target before OCR. @@ -46,7 +46,7 @@ open_code_review: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' when: on_success - when: never - # OCR 1.11.3 scales the 15-minute subtask base by low/medium/high rounds (15/30/45). + # OCR scales the 15-minute subtask base by low/medium/high rounds (15/30/45). timeout: 45m script: - apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git diff --git a/scripts/ocr_compat.py b/scripts/ocr_compat.py index ea3033f..563ed44 100644 --- a/scripts/ocr_compat.py +++ b/scripts/ocr_compat.py @@ -26,6 +26,8 @@ from pathlib import Path from typing import Any, NoReturn, TypeVar +import ocr_compat_history as history + ROOT = Path(__file__).resolve().parents[1] DEFAULT_MANIFEST = ROOT / "compatibility" / "ocr-support.json" PREFLIGHT = ROOT / "src" / "ocr_toolkit" / "preflight.py" @@ -64,6 +66,7 @@ ) REQUIRED_REVIEW_FLAGS = { "--audience", + "--effort", "--background-file", "--format", "--max-tokens-budget", @@ -118,6 +121,21 @@ }, "result": "passed", } + +CURRENT_LANGUAGE_RULES = { + "views/page.pug": "**/*.pug", + "rtl/module.v": "**/*.{v,sv,vh}", + "rtl/include.vh": "**/*.{v,sv,vh}", + "rtl/module.sv": "**/*.{v,sv,vh}", + "rtl/entity.vhd": "**/*.{vhd,vhdl}", + "rtl/entity.vhdl": "**/*.{vhd,vhdl}", + "src/module.mjs": "**/*.{ts,js,tsx,jsx,mjs,cjs}", + "src/module.cjs": "**/*.{ts,js,tsx,jsx,mjs,cjs}", + "native/source.cxx": "**/*.{cpp,cc,cxx,hpp,hxx}", + "native/header.hxx": "**/*.{cpp,cc,cxx,hpp,hxx}", + "native/object.mm": "**/*.mm", +} + REQUIRED_ASSETS = { "opencodereview-darwin-amd64", "opencodereview-darwin-arm64", @@ -283,6 +301,96 @@ def load_json(path: Path) -> dict[str, Any]: return value +def _validate_current_contracts(value: object) -> None: + """Require the live consumed contract, independent of candidate release numbering.""" + + if not isinstance(value, dict): + _fail("current qualification contracts are missing") + flags = value.get("required_review_flags") + capabilities = value.get("optional_capabilities") + if ( + not isinstance(flags, list) + or not REQUIRED_REVIEW_FLAGS.issubset(flags) + or not isinstance(capabilities, list) + or not {"review_effort", "semantic_grouping"}.issubset(capabilities) + ): + _fail("current qualification omitted required CLI capabilities") + extensions = sorted(Path(path).suffix for path in CURRENT_LANGUAGE_RULES) + expected = { + "numeric_cli_probe": CURRENT_NUMERIC_CLI_CONTRACT, + "review_budget_probe": { + "budget": 30_000, + "completed": 2, + "failed_budget": 1, + "partial_findings_preserved": True, + "result": "passed", + "selected": 3, + "grouping_requests": 0, + "grouping_strategy": "per_file", + }, + "semantic_grouping_probe": { + "default_effort": "medium", + "filter_requests": 1, + "grouping_requests": 1, + "main_requests": 3, + "result": "passed", + "review_rounds": 2, + "grouping_completion_cap": 16_384, + "files": 4, + "prior_finding_semantics": "filter_survivors_as_confirmed", + "recheck_instruction_requests": 3, + }, + "small_change_grouping_probe": { + "grouping_requests": 0, + "high_churn": "per_file", + "low_churn": "bundle_all", + "result": "passed", + "single_file": "per_file", + "threshold_files": 4, + }, + "language_rule_probe": { + "excluded_extensions": [".svh"], + "extensions": extensions, + "result": "passed", + "rule_source": "system_builtin", + "selected": len(extensions), + "m_routing": "matlab_and_objective_c", + }, + "completion_cap_probe": { + "explicit": 4_096, + "inherited": 16_384, + "result": "passed", + "wire_field": "max_completion_tokens", + }, + "comment_arguments_probe": { + "result": "passed", + "intact_cases": ["array", "serialized", "repaired"], + "comments_per_batch": 2, + "suspect_batch_rejected": True, + "failure_arguments_preserved": True, + "repair_warning": True, + }, + } + for name, contract in expected.items(): + if value.get(name) != contract: + _fail(f"current qualification contract disagrees: {name}") + if value.get("version_probe") != "passed" or value.get("preview_probe") != { + "format": "json", + "path": "example.py", + "result": "passed", + "session_store_created": False, + }: + _fail("current qualification omitted version or isolated JSON preview proof") + result = value.get("result_contract_probe") + if ( + not isinstance(result, dict) + or result.get("result") != "passed" + or result.get("manifest_schema") != "ocr.run-manifest/v1" + or result.get("normalized_outcome") != "clean" + ): + _fail("current qualification omitted the review-result contract") + + def validate_manifest(manifest: dict[str, Any], root: Path = ROOT) -> None: """Validate the versioned OCR support contract and evidence linkage.""" @@ -361,95 +469,10 @@ def validate_manifest(manifest: dict[str, Any], root: Path = ROOT) -> None: evidence = load_json(evidence_path) if evidence.get("version") != version or evidence.get("result") != "compatible": _fail(f"evidence does not qualify {version} as compatible") - if _version(version) >= (1, 9, 5): - contracts = evidence.get("contracts") - required_flags = ( - contracts.get("required_review_flags") if isinstance(contracts, dict) else None - ) - budget_probe = ( - contracts.get("review_budget_probe") if isinstance(contracts, dict) else None - ) - if not isinstance(required_flags, list) or "--max-tokens-budget" not in required_flags: - _fail(f"evidence does not qualify the review budget flag for {version}") - expected_budget_probe: dict[str, object] = { - "budget": 30_000, - "completed": 2, - "failed_budget": 1, - "partial_findings_preserved": True, - "result": "passed", - "selected": 3, - } - if _version(version) >= (1, 11, 1): - expected_budget_probe.update( - {"grouping_requests": 0, "grouping_strategy": "per_file"} - ) - if budget_probe != expected_budget_probe: - _fail(f"evidence does not qualify partial review budget behavior for {version}") - if _version(version) >= (1, 10, 0): - contracts = evidence.get("contracts") - required_flags = ( - contracts.get("required_review_flags") if isinstance(contracts, dict) else None - ) - capabilities = ( - contracts.get("optional_capabilities") if isinstance(contracts, dict) else None - ) - if not isinstance(required_flags, list) or "--effort" not in required_flags: - _fail(f"evidence does not qualify the review effort flag for {version}") - if not isinstance(capabilities, list) or not { - "review_effort", - "semantic_grouping", - }.issubset(capabilities): - _fail(f"evidence does not qualify effort and grouping for {version}") - expected_grouping_probe = { - "default_effort": "medium", - "filter_requests": 1, - "grouping_requests": 1, - "main_requests": 3, - "result": "passed", - "review_rounds": 2, - } - if _version(version) >= (1, 10, 2): - expected_grouping_probe["grouping_completion_cap"] = 16_384 - if _version(version) >= (1, 11, 1): - expected_grouping_probe.update( - { - "files": 4, - "prior_finding_semantics": "filter_survivors_as_confirmed", - "recheck_instruction_requests": 3, - } - ) - if contracts.get("semantic_grouping_probe") != expected_grouping_probe: - _fail(f"evidence does not qualify semantic grouping behavior for {version}") - if _version(version) >= (1, 11, 1) and contracts.get("small_change_grouping_probe") != { - "grouping_requests": 0, - "high_churn": "per_file", - "low_churn": "bundle_all", - "result": "passed", - "single_file": "per_file", - "threshold_files": 4, - }: - _fail(f"evidence does not qualify small-change grouping behavior for {version}") - expected_language_probe = { - "excluded_extensions": [".svh"], - "extensions": _expected_language_rule_extensions(version), - "result": "passed", - "rule_source": "system_builtin", - "selected": len(_expected_language_rule_extensions(version)), - } - if ( - _version(version) >= (1, 11, 1) - and contracts.get("language_rule_probe") != expected_language_probe - ): - _fail(f"evidence does not qualify built-in language rules for {version}") - if contracts.get("completion_cap_probe") != { - "explicit": 4_096, - "inherited": 16_384, - "result": "passed", - "wire_field": "max_completion_tokens", - }: - _fail(f"evidence does not qualify the completion cap for {version}") - if contracts.get("numeric_cli_probe") != CURRENT_NUMERIC_CLI_CONTRACT: - _fail(f"evidence does not qualify numeric CLI boundaries for {version}") + if _version(version) < history.HISTORICAL_CUTOFF: + history.validate_contracts(version, _version(version), evidence, _fail) + else: + _validate_current_contracts(evidence.get("contracts")) evidence_assets = evidence.get("assets") if not isinstance(evidence_assets, list): _fail(f"evidence assets are missing for {version}") @@ -811,9 +834,9 @@ class _StubHandler(http.server.BaseHTTPRequestHandler): tokens_per_request = 2 grouping_tokens_per_request = 2 grouping_mode = "singletons" - grouping_inventory_version: str | None = None grouping_inventories: list[list[GroupingInventoryEntry]] = [] main_mode = "findings" + comment_mode = "default" completion_caps: list[object] = [] request_stages: list[str] = [] prior_finding_semantics: set[str] = set() @@ -833,9 +856,7 @@ def _message_contents(messages: list[Any]) -> list[str]: def _grouping_files(cls, messages: list[Any]) -> list[str]: """Extract one version-bound grouping inventory and retain structural evidence.""" - if cls.grouping_inventory_version is None: - _fail("grouping inventory version is not configured") - entries = parse_grouping_inventory(messages, cls.grouping_inventory_version) + entries = parse_grouping_inventory(messages) cls.grouping_inventories.append(entries) return [entry.path for entry in entries] @@ -973,6 +994,8 @@ def do_POST(self) -> None: ] } ) + if type(self).comment_mode != "default": + arguments = _comment_probe_arguments(type(self).comment_mode, path) function = {"name": "code_comment", "arguments": arguments} call_id = "call-comment" else: @@ -1028,8 +1051,8 @@ def _stub_gateway( tokens_per_request: int = 2, grouping_tokens_per_request: int = 2, grouping_mode: str = "singletons", - grouping_inventory_version: str | None = None, main_mode: str = "findings", + comment_mode: str = "default", ) -> Iterator[str]: """Serve deterministic responses with configurable real usage accounting.""" @@ -1043,9 +1066,11 @@ def _stub_gateway( _StubHandler.tokens_per_request = tokens_per_request _StubHandler.grouping_tokens_per_request = grouping_tokens_per_request _StubHandler.grouping_mode = grouping_mode - _StubHandler.grouping_inventory_version = grouping_inventory_version _StubHandler.grouping_inventories = [] _StubHandler.main_mode = main_mode + if comment_mode not in {"default", "array", "serialized", "repaired", "rejected"}: + _fail("stub comment mode is invalid") + _StubHandler.comment_mode = comment_mode _StubHandler.completion_caps = [] _StubHandler.request_stages = [] _StubHandler.prior_finding_semantics = set() @@ -1061,24 +1086,14 @@ def _stub_gateway( thread.join(timeout=5) -def parse_grouping_inventory(messages: list[Any], version: str) -> list[GroupingInventoryEntry]: - """Parse the exact grouping prompt shape qualified for one OCR release line.""" +def parse_grouping_inventory(messages: list[Any]) -> list[GroupingInventoryEntry]: + """Parse the current status-first inventory without legacy execution fallbacks.""" - parsed_version = _version(version) - if (1, 10, 0) <= parsed_version < (1, 11, 0): - pattern = re.compile( - r"(?P[^\r\n]{1,1000}) " - r"\((?PADDED|MODIFIED|DELETED|RENAMED), " - r"\+(?P0|[1-9][0-9]{0,9})/-(?P0|[1-9][0-9]{0,9})\)" - ) - elif parsed_version >= (1, 11, 0): - pattern = re.compile( - r"(?PADDED|MODIFIED|DELETED|RENAMED) " - r"(?P[^\r\n]{1,1000}) " - r"\(\+(?P0|[1-9][0-9]{0,9})/-(?P0|[1-9][0-9]{0,9})\)" - ) - else: - _fail(f"grouping inventory format is not qualified for OCR {version}") + pattern = re.compile( + r"(?PADDED|MODIFIED|DELETED|RENAMED) " + r"(?P[^\r\n]{1,1000}) " + r"\(\+(?P0|[1-9][0-9]{0,9})/-(?P0|[1-9][0-9]{0,9})\)" + ) prefix = "Group the following changed files:\n\n" suffix = "\n\nRespond with a JSON array:" @@ -1101,7 +1116,7 @@ def parse_grouping_inventory(messages: list[Any], version: str) -> list[Grouping for line in lines: match = pattern.fullmatch(line) if match is None: - _fail(f"OCR {version} emitted an invalid grouping inventory entry") + _fail("OCR emitted an invalid grouping inventory entry") path = match.group("path") insertions = int(match.group("insertions")) deletions = int(match.group("deletions")) @@ -1111,7 +1126,7 @@ def parse_grouping_inventory(messages: list[Any], version: str) -> list[Grouping or insertions > MAX_GROUPING_CHURN or deletions > MAX_GROUPING_CHURN ): - _fail(f"OCR {version} emitted an invalid grouping inventory value") + _fail("OCR emitted an invalid grouping inventory value") observed_paths.add(path) entries.append( GroupingInventoryEntry( @@ -1199,7 +1214,7 @@ def _validate_file_groups(value: Any, expected_paths: set[str] | None = None) -> _fail("candidate semantic grouping did not cover the expected paths") -def _budget_result_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _budget_result_probe(binary: Path, directory: Path) -> dict[str, object]: """Drive the real OCR review budget gate and validate its partial manifest.""" git_env = _isolated_probe_environment(directory / "budget-git-home") @@ -1229,7 +1244,6 @@ def _budget_result_probe(binary: Path, version: str, directory: Path) -> dict[st env = _isolated_probe_environment(home) with _stub_gateway( tokens_per_request=20_000, - grouping_inventory_version=version, ) as gateway_url: env.update( { @@ -1304,10 +1318,9 @@ def _budget_result_probe(binary: Path, version: str, directory: Path) -> dict[st if any(len(group.get("files", [])) != 1 for group in groups): _fail("budget-limited review did not preserve per-file group boundaries") grouping_requests = request_stages.count("grouping") - expected_grouping_requests = 0 if _version(version) >= (1, 11, 1) else 1 - if grouping_requests != expected_grouping_requests: + if grouping_requests != 0: _fail("budget-limited review emitted an unexpected grouping request count") - if expected_grouping_requests == 0 and grouping_inventories: + if grouping_inventories: _fail("budget-limited small-change review unexpectedly emitted a grouping inventory") result: dict[str, object] = { "budget": 30_000, @@ -1317,8 +1330,7 @@ def _budget_result_probe(binary: Path, version: str, directory: Path) -> dict[st "result": "passed", "selected": 3, } - if _version(version) >= (1, 11, 1): - result.update({"grouping_requests": 0, "grouping_strategy": "per_file"}) + result.update({"grouping_requests": 0, "grouping_strategy": "per_file"}) return result @@ -1618,7 +1630,7 @@ def _numeric_cli_probe( } -def _semantic_grouping_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _semantic_grouping_probe(binary: Path, directory: Path) -> dict[str, object]: """Drive a threshold-crossing group through grouping and medium review rounds.""" root = directory / "semantic-grouping-probe" @@ -1629,11 +1641,7 @@ def _semantic_grouping_probe(binary: Path, version: str, directory: Path) -> dic _run(["git", "init", "--initial-branch=main"], cwd=repo, env=git_env) _run(["git", "config", "user.name", "Synthetic Reviewer"], cwd=repo, env=git_env) _run(["git", "config", "user.email", "reviewer@example.com"], cwd=repo, env=git_env) - paths = ( - ("01-first.py", "02-second.py", "03-third.py", "04-fourth.py") - if _version(version) >= (1, 11, 1) - else ("first.py", "second.py") - ) + paths = ("01-first.py", "02-second.py", "03-third.py", "04-fourth.py") for path in paths: (repo / path).write_text("def value():\n return 1\n", encoding="utf-8") _run(["git", "add", *paths], cwd=repo, env=git_env) @@ -1654,7 +1662,6 @@ def _semantic_grouping_probe(binary: Path, version: str, directory: Path) -> dic env = _isolated_probe_environment(root / "review-home") with _stub_gateway( grouping_mode="combined", - grouping_inventory_version=version, ) as gateway_url: env.update( { @@ -1710,7 +1717,7 @@ def _semantic_grouping_probe(binary: Path, version: str, directory: Path) -> dic if len(grouping_inventories) != 1: _fail("semantic grouping review emitted an unexpected grouping inventory count") _require_exact_grouping_inventory(grouping_inventories[0], expected_inventory) - expected_grouping_cap = 16_384 if _version(version) >= (1, 10, 2) else 4_096 + expected_grouping_cap = 16_384 if len(completion_caps) != len(stages) or completion_caps[0] != expected_grouping_cap: _fail( "semantic grouping review emitted an unexpected grouping completion cap: " @@ -1724,24 +1731,20 @@ def _semantic_grouping_probe(binary: Path, version: str, directory: Path) -> dic "result": "passed", "review_rounds": 2, } - if _version(version) >= (1, 10, 2): - result["grouping_completion_cap"] = expected_grouping_cap - if _version(version) >= (1, 11, 1): - if prior_finding_semantics != {"filter_survivors_as_confirmed"}: - _fail("multi-round review did not expose the qualified prior-finding semantics") - if recheck_instruction_requests != 3: - _fail("multi-round review did not retain toolkit re-check guidance in every round") - result["prior_finding_semantics"] = "filter_survivors_as_confirmed" - result["recheck_instruction_requests"] = recheck_instruction_requests - result["files"] = len(paths) + result["grouping_completion_cap"] = expected_grouping_cap + if prior_finding_semantics != {"filter_survivors_as_confirmed"}: + _fail("multi-round review did not expose the qualified prior-finding semantics") + if recheck_instruction_requests != 3: + _fail("multi-round review did not retain toolkit re-check guidance in every round") + result["prior_finding_semantics"] = "filter_survivors_as_confirmed" + result["recheck_instruction_requests"] = recheck_instruction_requests + result["files"] = len(paths) return result -def _small_change_grouping_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _small_change_grouping_probe(binary: Path, directory: Path) -> dict[str, object]: """Prove OCR keeps below-threshold grouping local and deterministic.""" - if _version(version) < (1, 11, 1): - _fail("small-change grouping probe requires OCR 1.11.1 behavior") scenarios = ( ("single_file", ("single.py",), 1, "per_file"), ("low_churn", ("first.py", "second.py"), 1, "bundle_all"), @@ -1773,7 +1776,7 @@ def _small_change_grouping_probe(binary: Path, version: str, directory: Path) -> _run(["git", "commit", "-am", "small-change update"], cwd=repo, env=git_env) head = _run(["git", "rev-parse", "HEAD"], cwd=repo, env=git_env).strip() env = _isolated_probe_environment(root / "review-home") - with _stub_gateway(grouping_inventory_version=version) as gateway_url: + with _stub_gateway() as gateway_url: env.update( { "OCR_LLM_URL": gateway_url, @@ -1831,7 +1834,7 @@ def _small_change_grouping_probe(binary: Path, version: str, directory: Path) -> } -def _completion_cap_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _completion_cap_probe(binary: Path, directory: Path) -> dict[str, object]: """Observe the real OCR chat-completions output cap with and without an override.""" probe_root = directory / "completion-cap-probe" @@ -1839,7 +1842,7 @@ def _completion_cap_probe(binary: Path, version: str, directory: Path) -> dict[s git_env = _isolated_probe_environment(probe_root / "git-home") repo, base, head = _synthetic_repo(probe_root, git_env) observed: dict[str, int] = {} - inherited = 16_384 if _version(version) >= (1, 10, 0) else 58_888 + inherited = 16_384 for label, expected in (("inherited", inherited), ("explicit", 4_096)): env = _isolated_probe_environment(probe_root / f"{label}-home") with _stub_gateway() as gateway_url: @@ -1897,24 +1900,11 @@ def _completion_cap_probe(binary: Path, version: str, directory: Path) -> dict[s } -def _preview_file_selection(payload: dict[str, Any] | str, path: str) -> tuple[bool, object]: - """Return one preview file's selected state and closed exclusion reason.""" +def _preview_file_selection(payload: object, path: str) -> tuple[bool, object]: + """Return one JSON preview file's selected state and closed exclusion reason.""" - if isinstance(payload, str): - section: str | None = None - for raw_line in payload.splitlines(): - line = re.sub(r"\x1b\[[0-9;]*m", "", raw_line).strip() - if line.startswith("Will review ("): - section = "selected" - continue - if line.startswith("Excluded from review ("): - section = "excluded" - continue - if path not in line: - continue - exclusion = "unsupported_ext" if "(unsupported_ext)" in line else None - return section == "selected", exclusion - return False, None + if not isinstance(payload, dict): + _fail("target-rule preview must be a JSON object") files = payload.get("files") if not isinstance(files, list): _fail("target-rule preview emitted an invalid file manifest") @@ -1924,7 +1914,7 @@ def _preview_file_selection(payload: dict[str, Any] | str, path: str) -> tuple[b return records[0].get("will_review") is True, records[0].get("exclude_reason") -def _target_rule_selection_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _target_rule_selection_probe(binary: Path, directory: Path) -> dict[str, object]: """Prove the real OCR selector consumes target rules without changing its range.""" git_env = _isolated_probe_environment(directory / "rule-git-home") @@ -1958,9 +1948,7 @@ def _target_rule_selection_probe(binary: Path, version: str, directory: Path) -> ) ) - json_preview = _version(version) >= (1, 9, 0) - - def preview(home_name: str, *extra: str) -> dict[str, Any] | str: + def preview(home_name: str, *extra: str) -> dict[str, Any]: home = directory / home_name env = _isolated_probe_environment(home) command = [ @@ -1973,13 +1961,10 @@ def preview(home_name: str, *extra: str) -> dict[str, Any] | str: "--preview", *extra, ] - if json_preview: - command.extend(["--format", "json"]) + command.extend(["--format", "json"]) output = _run(command, cwd=repo, env=env) if os.path.lexists(home / ".opencodereview" / "sessions"): _fail("target-rule preview created a review session store") - if not json_preview: - return output try: payload = json.loads(output) except json.JSONDecodeError as exc: @@ -1999,7 +1984,7 @@ def preview(home_name: str, *extra: str) -> dict[str, Any] | str: if not target_selected or target_reason not in {None, ""}: _fail("real OCR did not select the synthetic file from target rules") return { - "format": "json" if json_preview else "text", + "format": "json", "from_to_unchanged": True, "path": target.name, "result": "passed", @@ -2008,16 +1993,7 @@ def preview(home_name: str, *extra: str) -> dict[str, Any] | str: } -def _expected_language_rule_extensions(version: str) -> list[str]: - """Return the canonical sorted extension projection for one OCR release.""" - - extensions = {".pug", ".sv", ".v", ".vh", ".vhd", ".vhdl"} - if _version(version) >= (1, 11, 2): - extensions.update({".cjs", ".cxx", ".hxx", ".mjs"}) - return sorted(extensions) - - -def _language_rule_probe(binary: Path, version: str, directory: Path) -> dict[str, object]: +def _language_rule_probe(binary: Path, directory: Path) -> dict[str, object]: """Prove consumed built-in language selection and rule ownership without an LLM.""" root = directory / "language-rule-probe" @@ -2028,30 +2004,8 @@ def _language_rule_probe(binary: Path, version: str, directory: Path) -> dict[st _run(["git", "init", "--initial-branch=main"], cwd=repo, env=git_env) _run(["git", "config", "user.name", "Synthetic Reviewer"], cwd=repo, env=git_env) _run(["git", "config", "user.email", "reviewer@example.com"], cwd=repo, env=git_env) - qualified_rules = { - "views/page.pug", - "rtl/module.v", - "rtl/include.vh", - "rtl/module.sv", - "rtl/entity.vhd", - "rtl/entity.vhdl", - } - exact_patterns: dict[str, str] = {} - if _version(version) >= (1, 11, 2): - qualified_rules.update( - { - "src/module.mjs", - "src/module.cjs", - "native/source.cxx", - "native/header.hxx", - } - ) - exact_patterns = { - "src/module.mjs": "**/*.{ts,js,tsx,jsx,mjs,cjs}", - "src/module.cjs": "**/*.{ts,js,tsx,jsx,mjs,cjs}", - "native/source.cxx": "**/*.{cpp,cc,cxx,hpp,hxx}", - "native/header.hxx": "**/*.{cpp,cc,cxx,hpp,hxx}", - } + exact_patterns = CURRENT_LANGUAGE_RULES + qualified_rules = set(exact_patterns) supported_paths = tuple(sorted(qualified_rules)) unsupported_path = "rtl/include.svh" paths = (*supported_paths, unsupported_path) @@ -2116,16 +2070,144 @@ def _language_rule_probe(binary: Path, version: str, directory: Path) -> dict[st if expected_pattern is not None and f"Pattern: {expected_pattern}\n" not in output: _fail(f"candidate resolved the wrong built-in language rule for {path}") extensions = sorted(Path(path).suffix for path in supported_paths) - expected_extensions = _expected_language_rule_extensions(version) + expected_extensions = sorted(Path(path).suffix for path in CURRENT_LANGUAGE_RULES) if extensions != expected_extensions: _fail("language probe paths disagree with the canonical extension projection") - return { + result: dict[str, object] = { "extensions": expected_extensions, "excluded_extensions": [".svh"], "result": "passed", "rule_source": "system_builtin", "selected": len(supported_paths), } + for name, source, heading in ( + ("objective.m", "#import \n", "Objective-C"), + ("matrix.m", "function y = sample(x)\ny = x + 1;\nend\n", "MATLAB"), + ): + (repo / name).write_text(source, encoding="utf-8") + output = _run([str(binary), "rules", "check", name], cwd=repo, env=env) + if "Pattern: **/*.m\n" not in output or heading not in output: + _fail("candidate changed content-dependent .m rule routing") + mm_output = _run([str(binary), "rules", "check", "native/object.mm"], cwd=repo, env=env) + if "Objective-C" not in mm_output: + _fail("candidate did not route Objective-C++ to Objective-C rules") + result["m_routing"] = "matlab_and_objective_c" + return result + + +def _comment_probe_records(path: str) -> list[dict[str, str]]: + """Build a bounded comment batch whose public fields must survive intact.""" + + return [ + { + "content": f'Use "guard" before step {number}.', + "existing_code": " return 2", + "suggestion_code": " return 3", + "path": path, + "severity": "low", + "category": "bug", + } + for number in (1, 2) + ] + + +def _comment_probe_arguments(mode: str, path: str) -> str: + """Serialize controlled correct, recoverable and suspect tool arguments.""" + + records = _comment_probe_records(path) + if mode == "array": + return json.dumps({"comments": records}) + serialized = json.dumps(records) + if mode == "repaired": + serialized = serialized.replace('\\"guard\\"', '"guard"') + elif mode == "rejected": + serialized = ( + '[{"content":"Use "guard","existing_code":" return 2",' + '"suggestion_code":" return 3","path":' + json.dumps(path) + "}]" + ) + return json.dumps({"comments": serialized}) + + +def _comment_arguments_probe(binary: Path, directory: Path) -> dict[str, object]: + """Cross OCR's real parser and failure serialization using a deterministic gateway.""" + + from ocr_toolkit.review_runner import _tool_failure_telemetry + + root = directory / "comment-arguments-probe" + root.mkdir() + repo, base, head = _synthetic_repo(root, _isolated_probe_environment(root / "git-home")) + for mode in ("array", "serialized", "repaired", "rejected"): + env = _isolated_probe_environment(root / f"{mode}-home") + with _stub_gateway(comment_mode=mode) as gateway: + env.update( + { + "OCR_LLM_URL": gateway, + "OCR_LLM_TOKEN": "synthetic-token", + "OCR_LLM_MODEL": "synthetic-model", + "OCR_LLM_PROTOCOL": "openai", + "OCR_TELEMETRY_ENABLED": "false", + } + ) + raw = _run( + [ + str(binary), + "review", + "--from", + base, + "--to", + head, + "--format", + "json", + "--audience", + "agent", + "--concurrency", + "1", + "--effort", + "low", + "--no-filter", + ], + cwd=repo, + env=env, + ) + payload = json.loads(raw) + comments = payload.get("comments") + warnings = payload.get("warnings", []) + if not isinstance(comments, list) or not isinstance(warnings, list): + _fail(f"comment arguments {mode}: invalid result shape") + if mode == "rejected": + calls = payload.get("tool_calls", {}) + telemetry = _tool_failure_telemetry(calls) + details = calls.get("failure_details", []) + if ( + comments + or not telemetry.valid + or telemetry.failed != 1 + or len(details) != 1 + or details[0].get("arguments") != _comment_probe_arguments(mode, "example.py") + ): + _fail("comment arguments rejection lost its closed failure evidence") + else: + expected = _comment_probe_records("example.py") + actual = [{key: comment.get(key) for key in expected[0]} for comment in comments] + if sorted(actual, key=lambda item: item["content"]) != expected: + _fail(f"comment arguments {mode}: changed or lost comment fields") + if any(comment.get("start_line") != 2 for comment in comments): + _fail(f"comment arguments {mode}: lost deterministic anchors") + repaired = [ + item + for item in warnings + if isinstance(item, dict) and item.get("type") == "comment_args_repaired" + ] + if len(repaired) != int(mode == "repaired"): + _fail(f"comment arguments {mode}: wrong repair warning state") + return { + "result": "passed", + "intact_cases": ["array", "serialized", "repaired"], + "comments_per_batch": 2, + "suspect_batch_rejected": True, + "failure_arguments_preserved": True, + "repair_warning": True, + } def run_contracts(binary: Path, version: str, directory: Path) -> dict[str, Any]: @@ -2134,35 +2216,28 @@ def run_contracts(binary: Path, version: str, directory: Path) -> dict[str, Any] binary.chmod(binary.stat().st_mode | stat.S_IXUSR) git_env = _isolated_probe_environment(directory / "git-home") repo, base, head = _synthetic_repo(directory, git_env) - version_output = _run([str(binary), "--version"], cwd=repo) + version_output = _run([str(binary), "--version"], cwd=repo, env=git_env) if re.search(rf"(?= (1, 10, 0): - required_review_flags.add("--effort") missing = sorted(flag for flag in required_review_flags if flag not in help_output) if missing: _fail(f"candidate review help is missing required flags: {', '.join(missing)}") preview_home = directory / "preview-home" preview_env = _isolated_probe_environment(preview_home) preview_command = [str(binary), "review", "--from", base, "--to", head, "--preview"] - json_preview = _version(version) >= (1, 9, 0) - if json_preview: - preview_command.extend(["--format", "json"]) + preview_command.extend(["--format", "json"]) preview = _run(preview_command, cwd=repo, env=preview_env) - if json_preview: - try: - preview_payload = json.loads(preview) - except json.JSONDecodeError as exc: - raise CompatibilityError("candidate JSON preview did not emit JSON") from exc - preview_files = preview_payload.get("files") if isinstance(preview_payload, dict) else None - if not isinstance(preview_files, list) or not any( - isinstance(item, dict) and item.get("path") == "example.py" for item in preview_files - ): - _fail("candidate JSON preview did not select the synthetic changed file") - elif "example.py" not in preview: - _fail("candidate preview did not select the synthetic changed file") + try: + preview_payload = json.loads(preview) + except json.JSONDecodeError as exc: + raise CompatibilityError("candidate JSON preview did not emit JSON") from exc + preview_files = preview_payload.get("files") if isinstance(preview_payload, dict) else None + if not isinstance(preview_files, list) or not any( + isinstance(item, dict) and item.get("path") == "example.py" for item in preview_files + ): + _fail("candidate JSON preview did not select the synthetic changed file") if os.path.lexists(preview_home / ".opencodereview" / "sessions"): _fail("candidate preview created a review session store") @@ -2217,10 +2292,9 @@ def run_contracts(binary: Path, version: str, directory: Path) -> dict[str, Any] comments = sample.get("comments") if not isinstance(comments, list) or len(comments) != 1 or not isinstance(comments[0], dict): _fail("candidate full review did not emit the synthetic comment") - if _version(version) >= (1, 10, 0): - _validate_file_groups(sample.get("groups"), {"example.py"}) - if review_stages != ["main", "main", "filter", "main"]: - _fail(f"default medium review emitted an unexpected stage sequence: {review_stages!r}") + _validate_file_groups(sample.get("groups"), {"example.py"}) + if review_stages != ["main", "main", "filter", "main"]: + _fail(f"default medium review emitted an unexpected stage sequence: {review_stages!r}") sample["future_additive_field"] = {"accepted": True} from ocr_toolkit.posting.comments import comment_line from ocr_toolkit.posting.formatting import ( @@ -2239,27 +2313,25 @@ def run_contracts(binary: Path, version: str, directory: Path) -> dict[str, Any] _fail("toolkit token summary rejected the candidate result contract") if "1 total" not in format_tool_calls_summary(sample.get("tool_calls")): _fail("toolkit tool-call summary rejected the candidate result contract") - thinking_probe: dict[str, Any] | None = None - if _version(version) >= (1, 9, 0): - if comment.get("thinking") != "Synthetic private compatibility reasoning.": - _fail("candidate did not preserve additive comment thinking") - if "Synthetic private compatibility reasoning." in rendered: - _fail("toolkit posting consumer exposed private comment thinking") - thinking_probe = { - "additive_field_preserved": True, - "posting_exposes_thinking": False, - "result": "passed", - } + if comment.get("thinking") != "Synthetic private compatibility reasoning.": + _fail("candidate did not preserve additive comment thinking") + if "Synthetic private compatibility reasoning." in rendered: + _fail("toolkit posting consumer exposed private comment thinking") + thinking_probe = { + "additive_field_preserved": True, + "posting_exposes_thinking": False, + "result": "passed", + } contracts: dict[str, Any] = { "numeric_cli_probe": _numeric_cli_probe(binary, repo, base, head, directory), "optional_capabilities": optional_capabilities, - "review_budget_probe": _budget_result_probe(binary, version, directory), - "target_rule_selection_probe": _target_rule_selection_probe(binary, version, directory), + "review_budget_probe": _budget_result_probe(binary, directory), + "target_rule_selection_probe": _target_rule_selection_probe(binary, directory), "version_probe": "passed", "required_review_flags": sorted(required_review_flags), "preview_probe": { - "format": "json" if json_preview else "text", + "format": "json", "path": "example.py", "result": "passed", "session_store_created": False, @@ -2272,17 +2344,13 @@ def run_contracts(binary: Path, version: str, directory: Path) -> dict[str, Any] "result": "passed", }, } - if _version(version) >= (1, 10, 0): - contracts["semantic_grouping_probe"] = _semantic_grouping_probe(binary, version, directory) - if _version(version) >= (1, 11, 1): - contracts["small_change_grouping_probe"] = _small_change_grouping_probe( - binary, version, directory - ) - contracts["language_rule_probe"] = _language_rule_probe(binary, version, directory) - if _version(version) >= (1, 9, 10): - contracts["completion_cap_probe"] = _completion_cap_probe(binary, version, directory) - if thinking_probe is not None: - contracts["comment_thinking_probe"] = thinking_probe + contracts["semantic_grouping_probe"] = _semantic_grouping_probe(binary, directory) + contracts["small_change_grouping_probe"] = _small_change_grouping_probe(binary, directory) + contracts["language_rule_probe"] = _language_rule_probe(binary, directory) + contracts["completion_cap_probe"] = _completion_cap_probe(binary, directory) + contracts["comment_arguments_probe"] = _comment_arguments_probe(binary, directory) + contracts["comment_thinking_probe"] = thinking_probe + _validate_current_contracts(contracts) return contracts @@ -2488,7 +2556,7 @@ def artifacts() -> dict[str, Path]: contracts_passed=True, ) evidence = { - "schema_version": 3 if _version(version) >= (1, 11, 1) else 2, + "schema_version": 3, "upstream_repository": UPSTREAM_REPOSITORY, "version": version, "tag": tag, @@ -2631,6 +2699,12 @@ def prepare_update( if unknown_conclusions: _fail("human conclusions may reference only evidence versions in this promotion") + for item in evidences: + if _version(str(item["version"])) >= history.HISTORICAL_CUTOFF: + if item.get("schema_version") != 3: + _fail("current candidate requires evidence schema 3") + _validate_current_contracts(item.get("contracts")) + version = versions[-1] releases = manifest.get("releases") assert isinstance(releases, list) diff --git a/scripts/ocr_compat_history.py b/scripts/ocr_compat_history.py new file mode 100644 index 0000000..44ebee7 --- /dev/null +++ b/scripts/ocr_compat_history.py @@ -0,0 +1,147 @@ +"""Frozen readback checks for already-qualified OCR evidence; never execute binaries.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any, NoReturn + +HISTORICAL_CUTOFF = (1, 11, 4) + +ARCHIVED_NUMERIC_CLI_CONTRACT: dict[str, object] = { + "max_tokens_budget": { + "cases": { + "invalid_below": {"effective": None, "input": -1, "outcome": "rejected"}, + "minimum": {"effective": 1, "input": 1, "outcome": "accepted"}, + "omitted": {"effective": "unlimited", "input": None, "outcome": "accepted"}, + "representative": {"effective": 30_000, "input": 30_000, "outcome": "accepted"}, + "sentinel": {"effective": "unlimited", "input": 0, "outcome": "accepted"}, + }, + "maximum": None, + "owner": "ocr-cli", + }, + "max_tools": { + "cases": { + "invalid_below": {"effective": None, "input": -1, "outcome": "rejected"}, + "minimum": {"effective": 100, "input": 50, "outcome": "accepted"}, + "minimum_minus_one": { + "effective": 100, + "input": 49, + "outcome": "normalized", + "reported_normalization": 50, + }, + "omitted": {"effective": 100, "input": None, "outcome": "accepted"}, + "representative": {"effective": 101, "input": 101, "outcome": "accepted"}, + "sentinel": {"effective": 100, "input": 0, "outcome": "accepted"}, + }, + "maximum": None, + "owner": "ocr-template-or-higher-cli", + "reported_minimum": 50, + }, + "result": "passed", +} + + +def language_extensions(version_tuple: tuple[int, int, int]) -> list[str]: + """Return the language inventory recorded in the historical evidence epochs.""" + + extensions = {".pug", ".sv", ".v", ".vh", ".vhd", ".vhdl"} + if version_tuple >= (1, 11, 2): + extensions.update({".cjs", ".cxx", ".hxx", ".mjs"}) + return sorted(extensions) + + +def validate_contracts( + version: str, + version_tuple: tuple[int, int, int], + evidence: dict[str, Any], + fail: Callable[[str], NoReturn], +) -> None: + """Validate old evidence using its original semantics, independent of the live suite.""" + + if version_tuple >= HISTORICAL_CUTOFF: + fail("current evidence cannot use historical validation") + if version_tuple >= (1, 9, 5): + contracts = evidence.get("contracts") + required_flags = ( + contracts.get("required_review_flags") if isinstance(contracts, dict) else None + ) + budget_probe = contracts.get("review_budget_probe") if isinstance(contracts, dict) else None + if not isinstance(required_flags, list) or "--max-tokens-budget" not in required_flags: + fail(f"evidence does not qualify the review budget flag for {version}") + expected_budget_probe: dict[str, object] = { + "budget": 30_000, + "completed": 2, + "failed_budget": 1, + "partial_findings_preserved": True, + "result": "passed", + "selected": 3, + } + if version_tuple >= (1, 11, 1): + expected_budget_probe.update({"grouping_requests": 0, "grouping_strategy": "per_file"}) + if budget_probe != expected_budget_probe: + fail(f"evidence does not qualify partial review budget behavior for {version}") + if version_tuple >= (1, 10, 0): + contracts = evidence.get("contracts") + required_flags = ( + contracts.get("required_review_flags") if isinstance(contracts, dict) else None + ) + capabilities = ( + contracts.get("optional_capabilities") if isinstance(contracts, dict) else None + ) + if not isinstance(required_flags, list) or "--effort" not in required_flags: + fail(f"evidence does not qualify the review effort flag for {version}") + if not isinstance(capabilities, list) or not { + "review_effort", + "semantic_grouping", + }.issubset(capabilities): + fail(f"evidence does not qualify effort and grouping for {version}") + expected_grouping_probe = { + "default_effort": "medium", + "filter_requests": 1, + "grouping_requests": 1, + "main_requests": 3, + "result": "passed", + "review_rounds": 2, + } + if version_tuple >= (1, 10, 2): + expected_grouping_probe["grouping_completion_cap"] = 16_384 + if version_tuple >= (1, 11, 1): + expected_grouping_probe.update( + { + "files": 4, + "prior_finding_semantics": "filter_survivors_as_confirmed", + "recheck_instruction_requests": 3, + } + ) + if contracts.get("semantic_grouping_probe") != expected_grouping_probe: + fail(f"evidence does not qualify semantic grouping behavior for {version}") + if version_tuple >= (1, 11, 1) and contracts.get("small_change_grouping_probe") != { + "grouping_requests": 0, + "high_churn": "per_file", + "low_churn": "bundle_all", + "result": "passed", + "single_file": "per_file", + "threshold_files": 4, + }: + fail(f"evidence does not qualify small-change grouping behavior for {version}") + expected_language_probe = { + "excluded_extensions": [".svh"], + "extensions": language_extensions(version_tuple), + "result": "passed", + "rule_source": "system_builtin", + "selected": len(language_extensions(version_tuple)), + } + if ( + version_tuple >= (1, 11, 1) + and contracts.get("language_rule_probe") != expected_language_probe + ): + fail(f"evidence does not qualify built-in language rules for {version}") + if contracts.get("completion_cap_probe") != { + "explicit": 4_096, + "inherited": 16_384, + "result": "passed", + "wire_field": "max_completion_tokens", + }: + fail(f"evidence does not qualify the completion cap for {version}") + if contracts.get("numeric_cli_probe") != ARCHIVED_NUMERIC_CLI_CONTRACT: + fail(f"evidence does not qualify numeric CLI boundaries for {version}") diff --git a/src/ocr_toolkit/preflight.py b/src/ocr_toolkit/preflight.py index f255576..5d732b8 100644 --- a/src/ocr_toolkit/preflight.py +++ b/src/ocr_toolkit/preflight.py @@ -28,7 +28,7 @@ "Accept": "application/json", "User-Agent": "open-code-review-ci-preflight/1.0", } -EXPECTED_OCR_VERSION = "1.11.3" +EXPECTED_OCR_VERSION = "1.11.5" class PreflightError(Exception): diff --git a/tests/test_environment_contract.py b/tests/test_environment_contract.py index 8fef1a3..2791ad8 100644 --- a/tests/test_environment_contract.py +++ b/tests/test_environment_contract.py @@ -70,8 +70,8 @@ } EXAMPLE_DEFAULTS = { - "OCR_VERSION": "v1.11.3", - "OCR_SHA256": "9726204ac81baee153fd65b1ff357c380f73e9d8091c4a73c3c9fb541b5164cb", + "OCR_VERSION": "v1.11.5", + "OCR_SHA256": "53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95", "OCR_TOOLKIT_VERSION": STABLE_TOOLKIT_VERSION, "OCR_TOOLKIT_CHECKSUMS_URL": "Release URL derived from `OCR_TOOLKIT_VERSION`", "OCR_TOOLKIT_WHEEL": "open_code_review_toolkit-${OCR_TOOLKIT_VERSION}-py3-none-any.whl", @@ -306,7 +306,7 @@ def test_example_local_defaults_match_the_pipeline() -> None: encoding="utf-8" ) for name, value in { - "OCR_VERSION": "v1.11.3", + "OCR_VERSION": "v1.11.5", "OCR_SHA256": EXAMPLE_DEFAULTS["OCR_SHA256"], "OCR_TOOLKIT_VERSION": STABLE_TOOLKIT_VERSION, "OCR_MAX_TOOLS": "0", diff --git a/tests/test_ocr_compat.py b/tests/test_ocr_compat.py index 671a156..724f426 100644 --- a/tests/test_ocr_compat.py +++ b/tests/test_ocr_compat.py @@ -23,7 +23,8 @@ def load_script() -> ModuleType: assert spec is not None and spec.loader is not None module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = module - spec.loader.exec_module(module) + with patched_attr(sys, "path", [str(SCRIPT.parent), *sys.path]): + spec.loader.exec_module(module) return module @@ -43,7 +44,9 @@ def manifest_before_1_11_3(module: ModuleType) -> dict[str, Any]: manifest = module.load_json(MANIFEST) manifest["recommended_version"] = "1.11.2" manifest["monitoring_floor"] = "1.11.2" - manifest["releases"] = [item for item in manifest["releases"] if item["version"] != "1.11.3"] + manifest["releases"] = [ + item for item in manifest["releases"] if module._version(item["version"]) < (1, 11, 3) + ] return manifest @@ -53,8 +56,8 @@ def test_committed_manifest_is_valid_and_has_recommended_tested_baseline() -> No module.validate_manifest(manifest, PROJECT_ROOT) - assert manifest["recommended_version"] == "1.11.3" - assert manifest["monitoring_floor"] == "1.11.3" + assert manifest["recommended_version"] == "1.11.5" + assert manifest["monitoring_floor"] == "1.11.5" assert [(item["version"], item["status"]) for item in manifest["releases"]] == [ ("1.7.17", "tested"), ("1.8.0", "tested"), @@ -86,6 +89,8 @@ def test_committed_manifest_is_valid_and_has_recommended_tested_baseline() -> No ("1.11.1", "tested"), ("1.11.2", "tested"), ("1.11.3", "tested"), + ("1.11.4", "tested"), + ("1.11.5", "tested"), ] @@ -93,13 +98,13 @@ def test_language_probe_generation_and_validation_share_canonical_order() -> Non """Keep regenerated evidence byte-compatible with the manifest validator.""" module = load_script() - for version in ("1.11.1", "1.11.2", "1.11.3"): + for version in ("1.11.4", "1.11.5"): evidence = module.load_json( PROJECT_ROOT / "compatibility" / "evidence" / f"ocr-{version}.json" ) extensions = evidence["contracts"]["language_rule_probe"]["extensions"] - assert extensions == module._expected_language_rule_extensions(version) + assert extensions == sorted(Path(path).suffix for path in module.CURRENT_LANGUAGE_RULES) assert extensions == sorted(extensions) @@ -164,6 +169,65 @@ def test_manifest_rejects_assets_that_differ_from_evidence() -> None: module.validate_manifest(manifest, PROJECT_ROOT) +@pytest.mark.parametrize( + "corruption", + ["lost-comment", "changed-suggestion", "wrong-anchor", "missing-warning", "missing-arguments"], +) +def test_comment_probe_rejects_broken_boundary_observations( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, corruption: str +) -> None: + """Ensure the qualification verifier cannot certify lost output or diagnostic data.""" + + import contextlib + + module = load_script() + monkeypatch.setattr(module, "_synthetic_repo", lambda *_: (tmp_path, "a" * 40, "b" * 40)) + mode = "array" + + def gateway(**kwargs: object) -> object: + nonlocal mode + mode = str(kwargs["comment_mode"]) + return contextlib.nullcontext("http://127.0.0.1:1/v1") + + def observation(*_args: object, **_kwargs: object) -> str: + comments = [ + {**item, "start_line": 2} for item in module._comment_probe_records("example.py") + ] + warnings = [{"type": "comment_args_repaired"}] if mode == "repaired" else [] + calls = {} + if mode == "rejected": + comments = [] + detail = { + "tool_call_number": 1, + "tool_name": "code_comment", + "error": "invalid character", + "arguments": module._comment_probe_arguments(mode, "example.py"), + } + if corruption == "missing-arguments": + detail.pop("arguments") + calls = { + "total": 1, + "by_tool": {"code_comment": 1}, + "failure": 1, + "failure_by_tool": {"code_comment": 1}, + "failure_details": [detail], + } + elif corruption == "lost-comment": + comments.pop() + elif corruption == "changed-suggestion": + comments[0]["suggestion_code"] = "wrong" + elif corruption == "wrong-anchor": + comments[0]["start_line"] = 99 + elif corruption == "missing-warning": + warnings = [] + return json.dumps({"comments": comments, "warnings": warnings, "tool_calls": calls}) + + monkeypatch.setattr(module, "_stub_gateway", gateway) + monkeypatch.setattr(module, "_run", observation) + with pytest.raises(module.CompatibilityError, match="comment arguments"): + module._comment_arguments_probe(tmp_path / "ocr", tmp_path) + + def test_discovery_filters_known_prerelease_and_old_versions() -> None: module = load_script() manifest = module.load_json(MANIFEST) @@ -1117,7 +1181,7 @@ def test_compatibility_gateway_distinguishes_tool_free_plan_requests() -> None: ] } - with module._stub_gateway(grouping_inventory_version="1.11.1") as gateway: + with module._stub_gateway() as gateway: request = module.urllib.request.Request( f"{gateway}/chat/completions", data=json.dumps(payload).encode(), @@ -1135,8 +1199,8 @@ def test_compatibility_gateway_distinguishes_tool_free_plan_requests() -> None: assert content == "Summary: Review the changed code.\n\nIssues\n(none)" -def test_grouping_inventory_strictly_parses_old_and_new_release_shapes() -> None: - """Qualification pins the 1.10 line and 1.11 releases to exact wire shapes.""" +def test_grouping_inventory_strictly_parses_current_shape() -> None: + """Live qualification accepts status-first data and rejects legacy wire grammar.""" module = load_script() @@ -1160,40 +1224,20 @@ def messages(inventory: str) -> list[dict[str, str]]: "win\\deleted.hbs (DELETED, +0/-5)\n" "renamed.mustache (RENAMED, +0/-0)" ) - old_results = [ - module.parse_grouping_inventory(old_inventory, version) - for version in ("1.10.0", "1.10.1", "1.10.2") - ] - new_results = [ - module.parse_grouping_inventory( - messages( - "ADDED src/space (unicode) λ.py (+10/-0)\n" - "DELETED win\\deleted.hbs (+0/-5)\n" - "RENAMED renamed.mustache (+0/-0)" - ), - version, + result = module.parse_grouping_inventory( + messages( + "ADDED src/space (unicode) λ.py (+10/-0)\n" + "DELETED win\\deleted.hbs (+0/-5)\n" + "RENAMED renamed.mustache (+0/-0)" ) - for version in ("1.11.0", "1.11.1", "1.11.2") - ] - - assert ( - old_results[0] - == old_results[1] - == old_results[2] - == new_results[0] - == new_results[1] - == [ - module.GroupingInventoryEntry("ADDED", "src/space (unicode) λ.py", 10, 0), - module.GroupingInventoryEntry("DELETED", "win\\deleted.hbs", 0, 5), - module.GroupingInventoryEntry("RENAMED", "renamed.mustache", 0, 0), - ] ) + assert result == [ + module.GroupingInventoryEntry("ADDED", "src/space (unicode) λ.py", 10, 0), + module.GroupingInventoryEntry("DELETED", "win\\deleted.hbs", 0, 5), + module.GroupingInventoryEntry("RENAMED", "renamed.mustache", 0, 0), + ] with pytest.raises(module.CompatibilityError, match="invalid grouping inventory entry"): - module.parse_grouping_inventory(messages("ADDED path.py (+1/-0)"), "1.10.2") - with pytest.raises(module.CompatibilityError, match="invalid grouping inventory entry"): - module.parse_grouping_inventory(messages("path.py (ADDED, +1/-0)"), "1.11.0") - with pytest.raises(module.CompatibilityError, match="not qualified"): - module.parse_grouping_inventory(messages("path.py (ADDED, +1/-0)"), "1.9.10") + module.parse_grouping_inventory(old_inventory) def test_schema_three_candidate_remains_chain_aware() -> None: @@ -1251,7 +1295,7 @@ def test_grouping_inventory_rejects_mixed_duplicate_and_malformed_values( ] with pytest.raises(module.CompatibilityError, match="invalid grouping inventory"): - module.parse_grouping_inventory(messages, "1.11.0") + module.parse_grouping_inventory(messages) def test_grouping_inventory_rejects_missing_duplicate_and_oversized_blocks() -> None: @@ -1267,9 +1311,9 @@ def test_grouping_inventory_rejects_missing_duplicate_and_oversized_blocks() -> } with pytest.raises(module.CompatibilityError, match="exactly one"): - module.parse_grouping_inventory([], "1.11.0") + module.parse_grouping_inventory([]) with pytest.raises(module.CompatibilityError, match="exactly one"): - module.parse_grouping_inventory([valid, valid], "1.11.0") + module.parse_grouping_inventory([valid, valid]) oversized = "\n".join( f"MODIFIED file-{index}.py (+1/-1)" for index in range(module.MAX_GROUPING_INVENTORY_ENTRIES + 1) @@ -1285,7 +1329,6 @@ def test_grouping_inventory_rejects_missing_duplicate_and_oversized_blocks() -> ), } ], - "1.11.0", ) @@ -1679,7 +1722,22 @@ def test_prepare_update_promotes_one_reviewed_release_chain(tmp_path: Path) -> N assert "1.8.7 through 1.8.8" in fragment -def test_prepare_update_rejects_human_review_candidate(tmp_path: Path) -> None: +@pytest.fixture +def promotion_manifest(tmp_path: Path) -> Path: + """Freeze the promotion-policy baseline independently of the current runtime pin.""" + + module = load_script() + manifest = module.load_json(MANIFEST) + manifest["recommended_version"] = manifest["monitoring_floor"] = "1.11.3" + manifest["releases"] = [ + item for item in manifest["releases"] if module._version(item["version"]) <= (1, 11, 3) + ] + target = tmp_path / "manifest.json" + target.write_bytes(module.canonical_json(manifest)) + return target + + +def test_prepare_update_rejects_human_review_candidate(promotion_manifest: Path) -> None: module = load_script() evidence = { "schema_version": 2, @@ -1692,14 +1750,47 @@ def test_prepare_update_rejects_human_review_candidate(tmp_path: Path) -> None: with pytest.raises(module.CompatibilityError, match="bounded conclusion"): module.prepare_update( - manifest_path=MANIFEST, + manifest_path=promotion_manifest, evidence=evidence, fragment_number=42, root=PROJECT_ROOT, ) -def test_prepare_update_requires_human_review_for_minor_transition() -> None: +def test_historical_evidence_is_independent_of_live_contract_defaults( + monkeypatch: pytest.MonkeyPatch, promotion_manifest: Path +) -> None: + """Current suite changes cannot reinterpret already-qualified historical artifacts.""" + + module = load_script() + monkeypatch.setattr(module, "CURRENT_NUMERIC_CLI_CONTRACT", {}) + monkeypatch.setattr(module, "CURRENT_LANGUAGE_RULES", {}) + module.validate_manifest(module.load_json(promotion_manifest), PROJECT_ROOT) + + +def test_current_promotion_rejects_missing_contract_before_writing( + promotion_manifest: Path, +) -> None: + """A compatible label alone cannot promote a candidate missing consumed proof.""" + + module = load_script() + evidence = module.load_json(PROJECT_ROOT / "compatibility/evidence/ocr-1.11.4.json") + evidence["contracts"].pop("comment_arguments_probe") + before = promotion_manifest.read_bytes() + with pytest.raises(module.CompatibilityError, match="comment_arguments_probe"): + module.prepare_update( + manifest_path=promotion_manifest, + evidence=evidence, + fragment_number=176, + human_conclusions={"1.11.4": "Reviewed candidate."}, + root=PROJECT_ROOT, + ) + assert promotion_manifest.read_bytes() == before + + +def test_prepare_update_requires_human_review_for_minor_transition( + promotion_manifest: Path, +) -> None: module = load_script() evidence = { "schema_version": 2, @@ -1712,7 +1803,7 @@ def test_prepare_update_requires_human_review_for_minor_transition() -> None: with pytest.raises(module.CompatibilityError, match="explicit human review"): module.prepare_update( - manifest_path=MANIFEST, + manifest_path=promotion_manifest, evidence=evidence, fragment_number=73, root=PROJECT_ROOT, @@ -1762,7 +1853,7 @@ def test_prepare_update_rejects_nonadjacent_minor_transition() -> None: ) -def test_prepare_update_rejects_conclusion_outside_evidence_chain() -> None: +def test_prepare_update_rejects_conclusion_outside_evidence_chain(promotion_manifest: Path) -> None: module = load_script() evidence = { "schema_version": 2, @@ -1775,7 +1866,7 @@ def test_prepare_update_rejects_conclusion_outside_evidence_chain() -> None: with pytest.raises(module.CompatibilityError, match="only evidence versions"): module.prepare_update( - manifest_path=MANIFEST, + manifest_path=promotion_manifest, evidence=evidence, fragment_number=72, human_conclusions={"1.11.5": "Synthetic unrelated conclusion."}, @@ -1786,6 +1877,7 @@ def test_prepare_update_rejects_conclusion_outside_evidence_chain() -> None: @pytest.mark.parametrize("conclusion", ["", "x" * 2_001, "unsafe\x00text"]) def test_prepare_update_rejects_invalid_optional_reviewed_conclusion( conclusion: str, + promotion_manifest: Path, ) -> None: module = load_script() evidence = { @@ -1799,7 +1891,7 @@ def test_prepare_update_rejects_invalid_optional_reviewed_conclusion( with pytest.raises(module.CompatibilityError, match="bounded plain text"): module.prepare_update( - manifest_path=MANIFEST, + manifest_path=promotion_manifest, evidence=evidence, fragment_number=72, human_conclusions={"1.11.4": conclusion}, @@ -1841,9 +1933,13 @@ def test_prepare_update_rejects_invalid_optional_reviewed_conclusion( ), ], ) -def test_preview_file_selection_supports_json_and_legacy_text( +def test_preview_file_selection_accepts_json_and_rejects_legacy_text( payload: dict[str, Any] | str, expected: tuple[bool, object] ) -> None: module = load_script() - assert module._preview_file_selection(payload, "fixture.unknown") == expected + if isinstance(payload, str): + with pytest.raises(module.CompatibilityError, match="JSON object"): + module._preview_file_selection(payload, "fixture.unknown") + else: + assert module._preview_file_selection(payload, "fixture.unknown") == expected diff --git a/tests/test_review_runner.py b/tests/test_review_runner.py index a4d3e1a..9e94b21 100644 --- a/tests/test_review_runner.py +++ b/tests/test_review_runner.py @@ -1157,7 +1157,7 @@ def test_private_failure_arguments_preserve_finalized_signal_and_gitlab_notes( payload["comments"][0]["content"] = "Contact person@example.invalid" elif public_case == "warning": payload["warnings"] = [ - {"code": "comment_args_repaired", "message": "Repaired serialized comments"} + {"type": "comment_args_repaired", "message": "Repaired serialized comments"} ] elif public_case == "partial": payload["manifest"]["terminal_state"] = "partial" diff --git a/tests/test_runtime_helpers.py b/tests/test_runtime_helpers.py index af2e237..5a26cbf 100644 --- a/tests/test_runtime_helpers.py +++ b/tests/test_runtime_helpers.py @@ -1569,7 +1569,7 @@ def test_invalid_json_error_does_not_echo_secret_payload(self) -> None: class PreflightTests(unittest.TestCase): def test_validate_ocr_binary_accepts_supported_version(self) -> None: completed = subprocess.CompletedProcess( - args=["ocr", "--version"], returncode=0, stdout="ocr 1.11.3\n", stderr="" + args=["ocr", "--version"], returncode=0, stdout="ocr 1.11.5\n", stderr="" ) with ( patched_attr(preflight.shutil, "which", lambda _name: "/usr/bin/ocr"), From 3f56a7689a7adf1710272e451178389a49a71809 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 11:01:48 +0200 Subject: [PATCH 4/9] Document private diagnostics and deferred OCR qualification --- .github/workflows/ocr-compatibility.yml | 2 +- PLANS.md | 9 +++-- changelog.d/177.doc.md | 1 + docs/codex/TASKS_BACKLOG.md | 5 +++ docs/compatibility.md | 51 ++++++++++++++++++++++++ docs/engineering/test_evidence_matrix.md | 8 ++++ docs/operations.md | 7 ++++ docs/review-decision-flow.md | 14 ++++++- docs/security.md | 8 ++++ 9 files changed, 99 insertions(+), 6 deletions(-) create mode 100644 changelog.d/177.doc.md diff --git a/.github/workflows/ocr-compatibility.yml b/.github/workflows/ocr-compatibility.yml index 3828152..dc6e8bd 100644 --- a/.github/workflows/ocr-compatibility.yml +++ b/.github/workflows/ocr-compatibility.yml @@ -6,7 +6,7 @@ on: workflow_dispatch: inputs: tag: - description: Optional exact stable upstream tag, for example v1.7.18 + description: Optional exact stable upstream tag to qualify against the current contract required: false type: string diff --git a/PLANS.md b/PLANS.md index 531be61..130e324 100644 --- a/PLANS.md +++ b/PLANS.md @@ -89,8 +89,8 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-03 | done | Bounded failure arguments and DLP/summary regressions | | WQ-04 | done | Expanded real OCR qualification, adjacent evidence and final pins | | WQ-04R | done | Forward-only live qualification, frozen historical readback and maintenance instructions | -| WQ-05 | in_progress | Public docs, decision flow, changelog and backlog reconciliation | -| WQ-06 | pending | Final local gate, push, hosted checks and external Draft handoff | +| WQ-05 | done | Public docs, decision flow, changelog and backlog reconciliation | +| WQ-06 | in_progress | Final local gate, push, hosted checks and external Draft handoff | #### Locked Decisions @@ -158,7 +158,10 @@ outputs until verification, then remove only task-owned files. #### Resume Point -Finish WQ-05 public contract review, then perform WQ-06 final local/hosted gates. +Complete WQ-06 local final gates and aggregate self-review, then push the completed +history to Draft #178 and reconcile hosted checks. External model qualification +will start from the final pushed commit/tree with a built toolkit artifact; the +published example's toolkit package pin advances only in the later release PR. After the initial Draft push, keep later implementation commits local until all slices and final local validation complete. diff --git a/changelog.d/177.doc.md b/changelog.d/177.doc.md new file mode 100644 index 0000000..d375234 --- /dev/null +++ b/changelog.d/177.doc.md @@ -0,0 +1 @@ +Document group-scoped concurrency and prompt ceilings, the separate per-file large-diff filter, and the diagnostic-argument discard step in the canonical review decision flow. Distinguish private OCR artifacts from published GitLab summaries and identify the built toolkit artifact required for pre-release qualification. diff --git a/docs/codex/TASKS_BACKLOG.md b/docs/codex/TASKS_BACKLOG.md index 4175f6f..9ddf8f4 100644 --- a/docs/codex/TASKS_BACKLOG.md +++ b/docs/codex/TASKS_BACKLOG.md @@ -6,6 +6,11 @@ Statuses are `ready`, `planned`, `parked`, `conditional`, or `owner action`. Rel ## Existing backlog reconciliation +OCR 1.11.4/1.11.5 integration does not complete any remaining conditional item: +Objective-C++ routing is an OCR Rules change rather than a framework evidence pack; +viewer fixed/ignored marks are browser state rather than forge lifecycle support; +private failure arguments add no telemetry exporter or model-profile capability. + | Previous item | Disposition | Result | | --- | --- | --- | | M2 evidence gaps (BL-008) | Completed and removed | Demonstrated framework resolution, component ownership, scoped completeness, and first-class MCP delta projection ship through shared evidence contracts. | diff --git a/docs/compatibility.md b/docs/compatibility.md index bee6b0a..307b0fa 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -4,6 +4,13 @@ The versioned support contract lives in [`compatibility/ocr-support.json`](../co ## Qualification lanes +Live qualification runs one forward-only suite for the current consumed review +contract. It does not select parsers, fixtures, or required probes by candidate +version. Historical evidence before the suite boundary is checked separately by +`scripts/ocr_compat_history.py` using frozen expectations; that reader never +executes old binaries. Generic promotion tests use a frozen baseline. See +[qualification maintenance](development.md#maintaining-ocr-qualification). + The **OCR compatibility** workflow discovers stable upstream releases newer than the manifest monitoring floor. Its daily trigger is scheduled for `07:15 UTC`, after the observed upstream release window; GitHub may delay or omit scheduled delivery, so exact-tag manual dispatch remains the recovery path. Drafts, prereleases, non-semantic tags, unexpected asset sets, oversized metadata or downloads, redirects outside the reviewed GitHub origins, and checksum disagreement fail closed. Every binary digest must agree with both GitHub release metadata and the upstream `sha256sum.txt`. Candidate execution uses the verified Linux amd64 binary on an Ubuntu runner. The harness checks the reported version, the CLI flags consumed by the GitLab integration, range preview behavior, an actual JSON review through a deterministic local gateway, an aggregate-budget review that must preserve completed findings and emit budget-attributed partial coverage, and the additive JSON fields consumed by posting. Toolkit-managed numeric OCR options are also exercised at omitted/default, sentinel, invalid-below-boundary, minimum minus one, minimum, representative, and maximum edges when bounded. The evidence records closed outcomes, recognized diagnostics, normalization, ownership, and effective values observed through real loop behavior; CLI help text is not accepted as runtime evidence. For OCR 1.9.0 and later the harness also requires JSON preview without a session-store side effect and proves that additive comment `thinking` is accepted but not published to GitLab. Upstream source review separately verifies how OCR derives that field; the toolkit probe does not claim to reproduce a provider's private reasoning channel. Evidence permits unknown new fields but requires the fields the toolkit reads. Legacy result statuses and the versioned `ocr.run-manifest/v1` outcome are normalized through one shared toolkit contract; manifest coverage sets, failure classifications, terminal state, and budget attribution must agree before a result can be published. @@ -123,6 +130,50 @@ Upstream SIGTERM handling is compatible with the toolkit's termination masking a Deploy toolkit 0.9.0 directly with OCR 1.11.3. The public Linux amd64 example verifies SHA-256 `9726204ac81baee153fd65b1ff357c380f73e9d8091c4a73c3c9fb541b5164cb`; Darwin arm64 verifies `515cd92ce300b62dea2bcaf53e910cdf57b681e1d58bbd0a9e8e49617bca52fa`; and upstream `sha256sum.txt` verifies `472a3ce64834f4909bc9fac50e6de08a3c8ef5c0489b755dd1d2b19c14c8792e`. +### OCR 1.11.4 — toolkit 0.9.1 qualification predecessor + +OCR 1.11.4 adds raw `arguments` to failed-tool details. The toolkit accepts that +optional field only as an opaque string up to 32768 characters and 131072 UTF-8 +bytes, then omits it from normalized diagnostics before publication DLP. It cannot +enter toolkit console output, finalized results, receipts, GitLab notes, or toolkit +telemetry. Raw OCR stderr remains a private artifact. Invalid diagnostic shape or +bounds preserve a valid review publication and only affect the independent +later-action decision under existing receipt v8 rules. + +The native `code_comment` parser can recover serialized batches with damaged +escaping. Controlled real-binary probes verify native arrays, serialized arrays, +intact repaired comments/anchors/suggestions, rejection of suspect truncation, +and the ordinary `comment_args_repaired` warning. The toolkit does not implement +another repair parser or treat a repair as independent finding validation. +Objective-C++ `.mm` files use Objective-C rules; `.m` retains content-dependent +MATLAB/Objective-C routing. Documentation-site headings and plugin distribution +guardrails are upstream-only changes. + +### OCR 1.11.5 — toolkit 0.9.1 target + +Deploy toolkit 0.9.1 directly with exact OCR 1.11.5; an intermediate 1.11.4 +installation is unnecessary. The two releases retain separate adjacent source +audits and evidence. Assets were verified by hosted run `33962853525`; expanded +contract evidence explicitly identifies checksum-verified Darwin arm64 probes. +Final hosted qualification and configured external model qualification remain +distinct proofs. + +The grouped-review refactor preserves runtime behavior: concurrency counts groups, +the conversation prompt ceiling applies per group, and large-diff preselection +still checks each file separately. A diagnostic `file_path` may be a group key; +it is not a validated GitLab file location. The inherited completion cap remains +`16384`, an explicit positive override is still supported, default effort stays +`medium`, and max-tools retains template-default semantics. No public environment, +receipt v8, action receipt v3, summary, or approval contract is added. + +The binary updates grpc to 1.83.1; fast-uri and browserslist changes belong to the +upstream VS Code extension. Viewer fixed/ignored marks are browser-local state and +do not resolve or suppress GitLab findings. No viewer state is consumed by toolkit. + +Linux amd64 SHA-256: `53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95`. +Darwin arm64: `c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9`. +Upstream checksum file: `0519c13b03d69dd6c4aa5470a8eb52727f5e3fd2ce43e5887d8a67cdf107aa1a`. + ## Promotion and rollback Promotion changes `recommended_version`, advances `monitoring_floor`, adds the tested release and evidence, and updates every machine-readable or executable version/checksum pin. Never edit only one copy. Human-qualified candidates must record the compatibility conclusion and release-note impact; an automatic-safe candidate may also record a reviewed conclusion when it is delivered with a human-reviewed chain instead of using the generic machine conclusion. Conclusions may name only versions present in that promotion. A reviewed promotion may cross only one adjacent semantic-version boundary at a time: the next patch, the next minor at `.0`, or the next major at `.0.0`; minor and major transitions always require an explicit human conclusion. Automatic-safe preparation remains limited to adjacent patches in the already-tested major/minor line and retains the same protected review boundary even though the patch itself is mechanical. diff --git a/docs/engineering/test_evidence_matrix.md b/docs/engineering/test_evidence_matrix.md index a6c0a7d..d4d6feb 100644 --- a/docs/engineering/test_evidence_matrix.md +++ b/docs/engineering/test_evidence_matrix.md @@ -189,6 +189,14 @@ Every top-level test module is classified below. A module can contain more than | `test_runtime_helpers.py` | config filesystem boundaries and real local preflight HTTP transport; MCP/config parsing | mocked binary and `URL_OPENER` cases prove version/request/error policy only, not executable/network integration | | `test_testpypi_preview.py` | registry-manifest parser and static workflow contract | fixture index payloads do not prove publication; live TestPyPI/PyPI verification is a release gate | +## v0.9.1 private arguments and qualification evidence + +| Boundary | Evidence | Claim limit | +| --- | --- | --- | +| Opaque failure arguments → finalization → DLP → GitLab posting | Paired clean/finding/warning/partial/filtered results in `test_review_runner.py` preserve byte-equivalent finalized artifacts and note bodies; malformed/oversized arguments degrade diagnostics only | Real toolkit parser and posting workflow; GitLab transport is replaced with a capture, not a live service | +| Serialized comment repair and failure arguments | Checksum-verified OCR 1.11.4 and 1.11.5 use a local deterministic gateway; native, serialized and repaired batches preserve fields and anchors, suspect batches fail with original arguments in private failure data | Binary behavior and field preservation, not model judgment | +| Forward-only suite and historical evidence | Complete real no-LLM runs before/after refactoring yield identical JSON contracts; frozen promotion-baseline and historical-independence tests reject missing proof before writes | Historical readback does not imply supported execution of old binaries | + ## Unsafe or nondeterministic external boundaries The suite intentionally does not perform live GitLab comment, discussion, cleanup, or approval writes; live GitHub issue/release mutations; or PyPI publication. Their tests prove closed payloads, ordering, fail-closed decisions, transport serialization, and receipt parsing. Release completion requires independent live readback as defined in `docs/release.md`. diff --git a/docs/operations.md b/docs/operations.md index 694dddd..7cf374e 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -130,6 +130,13 @@ Suppression checks both the recorded inline position and compatible fingerprints ## OCR diagnostics +OCR's optional failed-tool `arguments` are validated only for type and bounds and +discarded before normalized diagnostics and publication DLP. Private arguments +alone do not change the GitLab summary or DLP counters. Malformed diagnostics +preserve valid review signal; genuine DLP failures in public fields still filter +those fields and leave a safe summary. Raw OCR stderr remains private. See the +[diagnostic decision flow](review-decision-flow.md#additive-failed-tool-diagnostic-states). + Receipt v8 is emitted only for a validated GitLab merge-request profile with a real `protected|unprotected` target observation. An ordinary local review still receives the same result parsing, evidence-action reconciliation, and publication DLP, but remains receipt-less because local Git state cannot establish GitLab protection. The compatible direct-posting path accepts an absent receipt, remains approval-ineligible, and cannot render receipt-derived MCP or target-protection claims; any present malformed or incomplete receipt still fails closed before prior review state is read. Run OCR through `ocr-ci review --result PATH --stderr PATH -- ...`. This wrapper does not post to GitLab: it creates private artifacts, acquires enriched context when selected, asks the exact resolved and preflight-qualified OCR executable to preview the production refs/rules/selection/background without an LLM, then runs the model review only if OCR accepts that background. OCR owns the current recommendation and rejection thresholds; the toolkit has no threshold setting. A recognized soft diagnostic is reduced to a toolkit-authored `ocr.toolkit-advisory/v1` enum and two positive character counts. It is attached after publication DLP, rendered only with an exact receipt v8 in Technical details, and does not change warnings, coverage, DLP counts, telemetry, or automatic approval. A recognized hard character/file-size rejection stops before the model and lets `ocr-ci post` publish only a static numeric failure summary; the OCR path and raw diagnostic remain private. Unknown preview failures use the generic fail-closed diagnostic path. The ordinary review still validates the same background, the wrapper validates the complete output, and context/session/configuration data is removed. On an unclassified ordinary failure it prints only a bounded redacted stderr excerpt to the runner log; a classified provider failure keeps that excerpt private. Pass the paths and captured exit code to `ocr-ci post` afterward. Set `OCR_POST_ERROR_DETAILS=1` only when the generic path's safe excerpt should also appear in the merge-request failure note. Cleanup uncertainty blocks result publication. DLP atomically converts unsafe publication output into a safe `completed_with_errors` subset, but sanitizes unsafe private-only result fields without discarding an otherwise valid manifest or finding set. Safe findings are posted, unsafe finding content/warnings and unsafe optional fields are omitted, previous OCR comments remain, and matching prior findings are consumed one-for-one rather than duplicated. Receipt v8 and the `ocr.publication-dlp-signal/v2` marker distinguish `private-sanitized`, where the canonical published and approval-relevant projection is unchanged, from approval-ineligible `publication-filtered`, where the public projection is incomplete even when the receipt proves OCR coverage was complete. Posting uses only the validated original coverage kind and counts for status and Technical details; it never reconstructs or republishes rejected values. Horizontal tabs survive only in `existing_code` and `suggestion_code`; the unchanged values pass the same secret, PII, forbidden-value, laundering, and budget checks, while tabs elsewhere and all other control/format characters remain blocking. The same count-only JSON is logged as `OCR toolkit telemetry event` for optional CI collection/alerting. It is not an OTLP/network exporter and contains no rejected value or location. Never interpret a filtered subset as a full publishable review or as proof that OCR coverage was partial. diff --git a/docs/review-decision-flow.md b/docs/review-decision-flow.md index 2f61a48..04a3e6c 100644 --- a/docs/review-decision-flow.md +++ b/docs/review-decision-flow.md @@ -89,7 +89,8 @@ flowchart TD present -- Yes --> absent[State: absent
no diagnostic log] present -- No --> shape{failure, failure_by_tool, and
failure_details form one bounded envelope?} shape -- No --> invalid[State: invalid
static malformed notice in console] - shape -- Yes --> failed{Failed count is zero?} + shape -- Yes --> omit_args[Discard optional opaque arguments
before normalized diagnostics and publication DLP] + omit_args --> failed{Failed count is zero?} failed -- Yes --> verified_zero[State: verified, failed: 0
no detail lines] failed -- No --> reconcile{Contradicts toolkit-owned
completed evidence actions?} reconcile -- Yes --> conflicting[State: conflicting
toolkit completion remains authoritative] @@ -112,7 +113,7 @@ flowchart TD class retain,later_ok success; class invalid,conflicting,verified,later_warn warning; - class raw,absent,verified_zero auxiliary; + class raw,absent,verified_zero,omit_args auxiliary; class present,shape,failed,reconcile,later decision; ``` @@ -123,6 +124,15 @@ inputs. Receipt v8 stores only `absent|verified|invalid|conflicting` and a bound `failed` integer for `verified`; the toolkit action receipt v3 remains authoritative for evidence attempts and completions. +The optional raw `arguments` string is bounded to 32768 characters and 131072 +UTF-8 bytes, then discarded rather than rendered. Its private content alone cannot +change publication-DLP counters, review status, findings, or summary text. Invalid +arguments make the additive diagnostic envelope invalid while retaining the review +signal. Public findings, warnings and suggestions still pass the full DLP checks. +Raw OCR stderr can contain arguments and stays private. A recovered comment batch +keeps its ordinary repair warning; neither repair nor diagnostic failure controls +summary publication through the later-action branch. + This separation applies in every execution profile. It is not a local-mode exception and it is not an automatic-approval feature: publication owns review-signal delivery, while any later action evaluates the already-published result independently. diff --git a/docs/security.md b/docs/security.md index b7cd765..becde31 100644 --- a/docs/security.md +++ b/docs/security.md @@ -63,6 +63,14 @@ Receipt v8 records the bounded configured capability inventory and positive call ## Preserved current safety properties +Failed-tool `arguments` are opaque private diagnostic input. The toolkit validates +their string type and bounded size, omits them from normalized diagnostics, and +removes the diagnostic envelope before publication DLP. This prevents private-only +arguments from creating false publication degradation without weakening DLP on +public findings, warnings, or suggestions. The original OCR stderr may contain +raw arguments and must remain a private artifact; `OCR_RAW_LOGGING` remains removed +from toolkit-owned OCR child environments. + - Repository reads are bounded, rooted, symlink-aware, immutable-object reads that exclude common dependency/build trees and never execute repository content. - Review context uses a closed `off|metadata|enriched` selector. `off` retains only validated source/protected-target/author identities; `metadata` admits bounded MR fields; `enriched` requires the immutable protected policy and admits only stable bounded discussion, CI-outcome, and adapter projections. Source policy, unknown fields/classes, raw display identities, arbitrary URLs/IDs, tokens, and ambient environment values cannot expand it. - Target protection uses a separate closed selector. Unset means exact `required`; explicit empty or unknown values fail closed. Exact `unprotected` changes behavior only when GitLab reports the actual target unprotected. That run permits `off|metadata`, built-in immutable evidence, and required exact-target Rules as untrusted guidance, while rejecting enriched acquisition, any adapter setting, direct or inherited external MCP, accepted decisions, and structured target guidance before OCR. From d570195a595a8cf33bdde2d7a7d909e7d7e4eabe Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 11:05:45 +0200 Subject: [PATCH 5/9] Record validation and route qualification maintenance --- AGENTS.md | 2 +- PLANS.md | 21 +++++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 69d8124..ef38f35 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ Use this file as the short repository map and startup workflow for Open Code Rev - `docs/engineering/toolkit_strategy.md` and `ROADMAP.md` - durable direction and outcome state. - `docs/engineering/project_principles.md` - cross-cutting engineering invariants and ownership boundaries; `docs/engineering/m5_context_contracts.md` owns the current context and evidence contracts. - `docs/review-decision-flow.md` - canonical detailed Mermaid map for review, diagnostics, receipts, DLP, publication, and later-action decisions; keep it synchronized with runtime and public contracts. -- `docs/development.md` - implementation workflow, boundary checklists, and local validation. +- `docs/development.md` - implementation workflow, boundary checklists, local validation, and [OCR qualification maintenance](docs/development.md#maintaining-ocr-qualification). - `docs/release.md` - release classification, authorization, publication, and archival lifecycle. - `docs/codex/TASKS_BACKLOG.md` - inactive work with activation conditions. - `docs/codex/AGENT_EXECUTION_PITFALLS.md` - incident catalogue for diagnosis, not an instruction source. diff --git a/PLANS.md b/PLANS.md index 130e324..10789c1 100644 --- a/PLANS.md +++ b/PLANS.md @@ -147,6 +147,14 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. baselines. Complete real-binary JSON proof is identical before/after refactoring for both releases. Focused suite passed 389 tests and 109 subtests; documentation and metadata suite passed 163 tests. Canonical maintenance instructions updated. +- 2026-09-06: final quality gate passed 1524 tests and 408 subtests at 86.39% + combined coverage; risk groups passed at 85/82/86/88%. Ruff format/lint, MyPy, + Bandit, lock, manifest/evidence, Towncrier and plan lifecycle checks passed. + Pinned Gitleaks 8.24.3 passed full feature history and current tree. Value-free + privacy comparison against main found only two added email matches in the + intentional private/public PII regression fixtures; no new hard-category finding. + Aggregate self-review confirmed unchanged publication/summary/DLP authority, + isolated historical readback and forward-only current qualification. #### Risks And Recovery @@ -158,12 +166,13 @@ outputs until verification, then remove only task-owned files. #### Resume Point -Complete WQ-06 local final gates and aggregate self-review, then push the completed -history to Draft #178 and reconcile hosted checks. External model qualification -will start from the final pushed commit/tree with a built toolkit artifact; the -published example's toolkit package pin advances only in the later release PR. -After the initial Draft push, keep later implementation commits local until all -slices and final local validation complete. +Local implementation and validation are complete. WQ-06's external receipt is +Draft PR #178: verify its current remote head and hosted checks before continuing. +If the head has green checks and the recorded runtime tree is unchanged, the next +action is configured external model qualification with OCR 1.11.5 and the toolkit +artifact built from that exact head. Otherwise inspect the failed hosted gate and +apply only an evidence-driven correction. Keep the PR Draft and both issues open; +the example's published toolkit package pin advances only in a later release PR. #### Plan Fidelity Check From 029cdf09cc01d47035c0cf37772c922d9027d2ce Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Sun, 6 Sep 2026 11:34:32 +0200 Subject: [PATCH 6/9] docs: centralize OCR version references in compatibility manifest --- PLANS.md | 18 +++++++++ README.md | 17 ++++---- changelog.d/177.doc.md | 2 + docs/codex/TASKS_BACKLOG.md | 10 ++--- docs/compatibility.md | 2 +- docs/configuration.md | 25 +++++++----- docs/development.md | 8 ++++ docs/engineering/review_signal_ownership.md | 2 +- docs/engineering/toolkit_strategy.md | 2 +- docs/gitlab.md | 4 +- docs/operations.md | 11 +++-- docs/security.md | 2 +- examples/gitlab/README.md | 2 +- tests/test_environment_contract.py | 9 +++-- tests/test_operations_docs.py | 45 +++++++++++++++------ tests/test_runtime_helpers.py | 7 +++- 16 files changed, 112 insertions(+), 54 deletions(-) diff --git a/PLANS.md b/PLANS.md index 10789c1..210845f 100644 --- a/PLANS.md +++ b/PLANS.md @@ -40,6 +40,7 @@ summary publication. External configured qualification owns the later release de | REQ-006 | Docs, decision flow, changelog and backlog reflect current behavior | WQ-05 | documentation checks and rendered Towncrier | | REQ-007 | Green pushed Draft with truthful external qualification | WQ-06 | local/hosted gates and remote readback | | REQ-008 | Forward-only live qualification, isolated historical readback and stable tests | WQ-04R | no live version branches, frozen-history validation, CLI and probe regressions | +| REQ-009 | Version-neutral current guidance with exact compatibility identities preserved | WQ-07 | documentation contracts, pin validation, self-review and green Draft push | #### Explicit Non-Goals @@ -91,6 +92,7 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-04R | done | Forward-only live qualification, frozen historical readback and maintenance instructions | | WQ-05 | done | Public docs, decision flow, changelog and backlog reconciliation | | WQ-06 | in_progress | Final local gate, push, hosted checks and external Draft handoff | +| WQ-07 | done | Version-neutral guidance and generic fixtures validated and self-reviewed; Draft push/readback remains owned by WQ-06 | #### Locked Decisions @@ -110,6 +112,11 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index validation semantics in a separate owner. Promotion-policy tests use frozen baselines; current pin tests alone assert the current version. Update canonical development/compatibility instructions rather than adding duplicate agent rules. +- Follow-up documentation maintenance is `no-release` within this deferred release + branch. Current guidance links to the compatibility manifest instead of repeating + OCR release numbers or asset hashes. Preserve executable pins, exact identity + checks, historical evidence, release notes and this release-specific plan. + No runtime behavior, supported-version policy or release authorization changes. #### Verification @@ -155,6 +162,12 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. intentional private/public PII regression fixtures; no new hard-category finding. Aggregate self-review confirmed unchanged publication/summary/DLP authority, isolated historical readback and forward-only current qualification. +- 2026-09-06: version-neutral follow-up passed 1525 tests and 408 subtests at + 86.39% combined coverage, all scoped floors, Ruff, MyPy and Bandit. Focused + documentation/environment/integration tests passed 48 tests; manifest, lock, + Towncrier and diff checks passed. Self-review removed the remaining duplicated + documentation checksum and separated manifest-reference assertions from exact + executable-pin assertions. Runtime, pins and historical evidence are unchanged. #### Risks And Recovery @@ -166,6 +179,11 @@ outputs until verification, then remove only task-owned files. #### Resume Point +WQ-07's version-neutral follow-up is locally validated and self-reviewed. WQ-06 +owns the signed commit's pinned Gitleaks check, push to Draft #178 and current +remote-head/hosted-check readback. The PR receipt records that external state; +do not infer the latest head's checks from a predecessor's green result. + Local implementation and validation are complete. WQ-06's external receipt is Draft PR #178: verify its current remote head and hosted checks before continuing. If the head has green checks and the recorded runtime tree is unchanged, the next diff --git a/README.md b/README.md index 8c7f3cd..cc85790 100644 --- a/README.md +++ b/README.md @@ -30,13 +30,12 @@ python -m venv .venv python -m pip install open-code-review-toolkit ``` -The toolkit does not bundle OCR. Download the platform asset for exact -[Open Code Review 1.11.5](https://github.com/alibaba/open-code-review/releases/tag/v1.11.5), -verify it before installation against the [compatibility manifest](compatibility/ocr-support.json), -and place the verified binary on `PATH`. The current Linux amd64 digest is -`53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95`; Darwin arm64 is -`c041b03cc840957b52df28514e8dbb51f798e6cb1259d97555a41a2e3e3ccaf9`. The manifest owns -the other platform hashes. +The toolkit does not bundle OCR. Use `recommended_version` in the +[compatibility manifest](compatibility/ocr-support.json) for this toolkit revision +to select the exact [upstream release](https://github.com/alibaba/open-code-review/releases). +Download its platform asset, verify it against that release entry's asset SHA-256, +and place the verified binary on `PATH`. Do not substitute the latest upstream +release or use another toolkit revision's manifest. Verify the installation without contacting an LLM provider: @@ -45,14 +44,14 @@ ocr --version ocr-ci --help ``` -`ocr --version` must report `open-code-review v1.11.5`; `ocr-ci --help` must exit +`ocr --version` must report the manifest's exact `recommended_version`; `ocr-ci --help` must exit successfully. `ocr-ci preflight` is an operational configuration check, not the installation smoke test. The exact recommended OCR release and its verified asset checksums live in the [versioned compatibility manifest](compatibility/ocr-support.json). CI should pin that release and checksum before execution. The [versioned compatibility policy](docs/compatibility.md) records tested assets and evidence and describes the conservative Dependabot-like qualification workflow for later upstream releases. Review output defaults to English. `OCR_REVIEW_LANGUAGE` accepts another explicit language name when a project needs localized review output; for example, `OCR_REVIEW_LANGUAGE=Russian`. -The current OCR 1.11.5 integration defaults `OCR_REVIEW_EFFORT` to `medium` for two review rounds. `low` and `high` are explicit one- and three-round alternatives; see the [configuration reference](docs/configuration.md#review-effort) for cost, budget, and precedence boundaries. +The toolkit defaults `OCR_REVIEW_EFFORT` to `medium` for two review rounds. `low` and `high` are explicit one- and three-round alternatives; see the [configuration reference](docs/configuration.md#review-effort) for cost, budget, and precedence boundaries. Stable distributions are published to [PyPI](https://pypi.org/project/open-code-review-toolkit/) and mirrored as checksum-listed, provenance-attested assets in the corresponding [GitHub Release](https://github.com/xeonvs/open-code-review-toolkit/releases). Development snapshots are published only to TestPyPI. diff --git a/changelog.d/177.doc.md b/changelog.d/177.doc.md index d375234..cb37ffe 100644 --- a/changelog.d/177.doc.md +++ b/changelog.d/177.doc.md @@ -1 +1,3 @@ Document group-scoped concurrency and prompt ceilings, the separate per-file large-diff filter, and the diagnostic-argument discard step in the canonical review decision flow. Distinguish private OCR artifacts from published GitLab summaries and identify the built toolkit artifact required for pre-release qualification. + +Keep current operator guidance version-neutral and point installation and pin lookup to the same toolkit revision's compatibility manifest; preserve exact executable pins and historical qualification evidence. diff --git a/docs/codex/TASKS_BACKLOG.md b/docs/codex/TASKS_BACKLOG.md index 9ddf8f4..36c7cb5 100644 --- a/docs/codex/TASKS_BACKLOG.md +++ b/docs/codex/TASKS_BACKLOG.md @@ -6,7 +6,7 @@ Statuses are `ready`, `planned`, `parked`, `conditional`, or `owner action`. Rel ## Existing backlog reconciliation -OCR 1.11.4/1.11.5 integration does not complete any remaining conditional item: +The qualified OCR capabilities do not complete any remaining conditional item: Objective-C++ routing is an OCR Rules change rather than a framework evidence pack; viewer fixed/ignored marks are browser state rather than forge lifecycle support; private failure arguments add no telemetry exporter or model-profile capability. @@ -56,9 +56,9 @@ Provider/network telemetry remains outside M1 and M5. OCR owns token, cost, budg - **Status:** parked - **Priority:** medium - **Roadmap theme:** M6 Profiles and quality measurement -- **Dependencies:** Established built-in MCP lifecycle and OCR per-run model/provider overrides; OCR 1.8.7 satisfies the capability dependency. +- **Dependencies:** Established built-in MCP lifecycle and qualified OCR per-run model/provider overrides satisfy the capability dependency. - **Activation trigger:** Not met. Activate only after repeated operations show direct settings are insufficient and the owner approves a closed model/provider matrix plus precedence contract. -- **Upstream overlap:** OCR 1.8.7 supplies direct run-level selection; OCR 1.9.0 per-file and 1.9.5 aggregate budgets remain explicit completeness controls, not profile defaults. +- **Upstream overlap:** OCR supplies direct run-level selection; per-file and aggregate budgets remain explicit completeness controls, not profile defaults. - **Goal:** If need appears, offer `economy`, `standard`, and `strong` aliases for one OCR run without hiding aggregate, per-file, or tool controls. - **Scoped deliverables:** Define an owner-approved closed matrix and precedence contract; map an alias to one OCR run; publish effective non-secret identity; validate compatibility and environment precedence. - **Acceptance criteria:** One model remains active per run, `standard` preserves current behavior, explicit provider/model settings override profile aliases, aggregate/per-file/tool limits remain independent explicit operator inputs, secrets remain environment-only, and unavailable combinations fail before OCR execution. @@ -94,7 +94,7 @@ Provider/network telemetry remains outside M1 and M5. OCR owns token, cost, budg - **Acceptance criteria:** The use case and completion signal are documented before implementation; false-positive behavior and unsupported versions are explicit through the shared scoped coverage contract. - **Exclusions:** Checkbox coverage, network resolution, runtime execution, or unrelated bundles. - **Validation:** Pack fixtures plus common evidence/bootstrap/MCP contracts. -- **Upstream overlap:** OCR language allowlists and review rules are review-engine capabilities; OCR 1.11.0 Handlebars/Mustache selection improves review coverage but does not supply a framework evidence contract or activate an evidence pack. +- **Upstream overlap:** OCR language allowlists and review rules are review-engine capabilities; Handlebars/Mustache selection improves review coverage but does not supply a framework evidence contract or activate an evidence pack. - **Release classification expectation:** `release-required`. ### BL-019: Run a native fuzzing campaign @@ -132,7 +132,7 @@ Provider/network telemetry remains outside M1 and M5. OCR owns token, cost, budg - **Roadmap theme:** M7 Later and conditional work - **Dependencies:** Stable provider-neutral core contracts and a funded non-GitLab use case. GitLab-first M5 does not depend on it. - **Activation trigger:** A named forge has an owner, synthetic fixtures, and explicit parity requirements for CI orchestration, positioning, deduplication, discussion ownership, and safe publication. -- **Upstream overlap:** OCR 1.10.2 reusable GitHub Action checkpoint ranges and OCR 1.11.0 Action/plugin changes cover only upstream execution surfaces. They do not provide toolkit forge acquisition, discussion, publication, or lifecycle parity, so the trigger and acceptance criteria remain unmet. +- **Upstream overlap:** OCR reusable GitHub Action checkpoint ranges and Action/plugin capabilities cover only upstream execution surfaces. They do not provide toolkit forge acquisition, discussion, publication, or lifecycle parity, so the trigger and acceptance criteria remain unmet. - **Goal:** Add one coherent host adapter without leaking forge semantics into evidence or core result handling. - **Scoped deliverables:** The capability matrix covers authentication, diff positions, drafts, discussion acquisition, protected same-revision CI outcomes, provider-declared account classification, thread/reply structure, edit/version identity, anchors, resolved/stale state, pagination/snapshot mutation, ambiguous writes, permissions, and idempotency. - **Acceptance criteria:** Core remains provider-neutral, GitLab behavior does not regress, unsupported host capabilities fail or degrade explicitly rather than emulate unsafe parity, and the new host meets the approved lifecycle and security matrix. diff --git a/docs/compatibility.md b/docs/compatibility.md index 307b0fa..ef97b2d 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -13,7 +13,7 @@ executes old binaries. Generic promotion tests use a frozen baseline. See The **OCR compatibility** workflow discovers stable upstream releases newer than the manifest monitoring floor. Its daily trigger is scheduled for `07:15 UTC`, after the observed upstream release window; GitHub may delay or omit scheduled delivery, so exact-tag manual dispatch remains the recovery path. Drafts, prereleases, non-semantic tags, unexpected asset sets, oversized metadata or downloads, redirects outside the reviewed GitHub origins, and checksum disagreement fail closed. Every binary digest must agree with both GitHub release metadata and the upstream `sha256sum.txt`. -Candidate execution uses the verified Linux amd64 binary on an Ubuntu runner. The harness checks the reported version, the CLI flags consumed by the GitLab integration, range preview behavior, an actual JSON review through a deterministic local gateway, an aggregate-budget review that must preserve completed findings and emit budget-attributed partial coverage, and the additive JSON fields consumed by posting. Toolkit-managed numeric OCR options are also exercised at omitted/default, sentinel, invalid-below-boundary, minimum minus one, minimum, representative, and maximum edges when bounded. The evidence records closed outcomes, recognized diagnostics, normalization, ownership, and effective values observed through real loop behavior; CLI help text is not accepted as runtime evidence. For OCR 1.9.0 and later the harness also requires JSON preview without a session-store side effect and proves that additive comment `thinking` is accepted but not published to GitLab. Upstream source review separately verifies how OCR derives that field; the toolkit probe does not claim to reproduce a provider's private reasoning channel. Evidence permits unknown new fields but requires the fields the toolkit reads. Legacy result statuses and the versioned `ocr.run-manifest/v1` outcome are normalized through one shared toolkit contract; manifest coverage sets, failure classifications, terminal state, and budget attribution must agree before a result can be published. +Candidate execution uses the verified Linux amd64 binary on an Ubuntu runner. The harness checks the reported version, the CLI flags consumed by the GitLab integration, range preview behavior, an actual JSON review through a deterministic local gateway, an aggregate-budget review that must preserve completed findings and emit budget-attributed partial coverage, and the additive JSON fields consumed by posting. Toolkit-managed numeric OCR options are also exercised at omitted/default, sentinel, invalid-below-boundary, minimum minus one, minimum, representative, and maximum edges when bounded. The evidence records closed outcomes, recognized diagnostics, normalization, ownership, and effective values observed through real loop behavior; CLI help text is not accepted as runtime evidence. The harness also requires JSON preview without a session-store side effect and proves that additive comment `thinking` is accepted but not published to GitLab. Upstream source review separately verifies how OCR derives that field; the toolkit probe does not claim to reproduce a provider's private reasoning channel. Evidence permits unknown new fields but requires the fields the toolkit reads. Legacy result statuses and the versioned `ocr.run-manifest/v1` outcome are normalized through one shared toolkit contract; manifest coverage sets, failure classifications, terminal state, and budget attribution must agree before a result can be published. Built-in MCP qualification follows the protocol revisions supported by the recommended OCR release's exact MCP SDK. The current recommended release uses Go MCP SDK v1.6.1 and initiates revision `2025-11-25`; the evidence server also retains `2025-06-18`, `2025-03-26`, and `2024-11-05` for qualified older clients. For an unknown client revision the server follows MCP negotiation semantics by returning its current supported revision, leaving acceptance or termination to the client. Qualification exercises initialize, the initialized notification, ping, tool discovery, and bounded summary/list/get calls through the exact SDK rather than relying only on handcrafted JSON-RPC fixtures. diff --git a/docs/configuration.md b/docs/configuration.md index cac230e..1033b21 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -2,6 +2,10 @@ Open Code Review Toolkit uses environment variables for CI/runtime configuration. Empty values are generally treated as absent. Exact defaults and safety caps are enforced by the runtime modules. **Bold variable names are required in the command, mode, example, or declaration scope stated in their `Required` cell.** +OCR behavior below refers to the exact `recommended_version` in the +[compatibility manifest](../compatibility/ocr-support.json) for this toolkit +revision, not arbitrary upstream releases. + ## Toolkit runtime variables These are the complete supported toolkit-owned runtime inputs. `Required` is scoped to the command or mode named in the behavior column; an unrelated command does not require the variable. @@ -22,7 +26,7 @@ These are the complete supported toolkit-owned runtime inputs. `Required` is sco | `OCR_LLM_VALIDATE_MODEL` | Operator / `ocr-ci preflight` | No | `false` | `true` validates through `/models`; `auto` may use the offline allowlist; false values skip validation. | | `OCR_LLM_MODELS_URL` | Operator / `ocr-ci preflight` | No | Derived from `OCR_LLM_URL` | Explicit absolute credential-free HTTPS metadata URL when validation is enabled or inference query parameters make derivation ambiguous. | | `OCR_LLM_ALLOWED_MODELS` | Operator / `ocr-ci preflight` | No | Empty list | Comma-separated exact model identifiers for offline or `auto` validation. | -| `OCR_TELEMETRY_ENABLED` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR telemetry configuration; OCR 1.11.5 spans may include path-derived group keys, model-produced labels, and local grouping decisions. | +| `OCR_TELEMETRY_ENABLED` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR telemetry configuration; OCR spans may include path-derived group keys, model-produced labels, and local grouping decisions. | | `OCR_TELEMETRY_CONTENT_LOGGING` | Operator / `ocr-ci configure` | No | `false` | Exact `true` enables OCR content logging; keep disabled for private review data. | | `OCR_TELEMETRY_EXPORTER` | Operator / `ocr-ci configure` | No | Empty string | Exporter name written only when telemetry is enabled. | | `OCR_TELEMETRY_OTLP_ENDPOINT` | Operator / `ocr-ci configure` | No | Unset | OTLP endpoint written only when telemetry is enabled and the value is non-empty. | @@ -60,13 +64,13 @@ If `OCR_LLM_EXTRA_BODY` already owns that field, an exactly equal JSON integer i The toolkit does not derive this value from `/models.max_completion_tokens`. That metadata is a model capability boundary, not an account spending limit or proof of how a gateway reserves request cost. -The inherited value is version-owned and therefore changes with a qualified OCR upgrade. The toolkit observed `max_completion_tokens=58888` from OCR 1.9.10 and `16384` from OCR 1.10.0 through 1.11.5 when the variable was unset. Grouping requests use the same template-owned cap; an explicit toolkit override still applies to every protocol request. Deployments that require an invariant gateway-specific cap must set `OCR_LLM_MAX_COMPLETION_TOKENS` explicitly rather than depending on an OCR default. +The inherited value is version-owned and may change with a qualified OCR upgrade. The qualified OpenAI request uses `max_completion_tokens=16384` when the variable is unset; [compatibility history](compatibility.md) records earlier values and their exact versions. Grouping requests use the same template-owned cap; an explicit toolkit override still applies to every protocol request. Deployments that require an invariant gateway-specific cap must set `OCR_LLM_MAX_COMPLETION_TOKENS` explicitly rather than depending on an OCR default. ### Review effort -`OCR_REVIEW_EFFORT` defaults to `medium` and is written to OCR's root `effort` configuration key. OCR 1.11.5 maps `low`, `medium`, and `high` to one, two, and three review rounds and scales its 15-minute subtask base to 15, 30, or 45 minutes. The environment is operator-owned; merge-request text cannot change it. An explicit caller `--effort` passed after `ocr-ci review --` has normal OCR CLI precedence over the generated config, while an unknown environment value fails configuration before preview or model execution. +`OCR_REVIEW_EFFORT` defaults to `medium` and is written to OCR's root `effort` configuration key. OCR maps `low`, `medium`, and `high` to one, two, and three review rounds and scales its 15-minute subtask base to 15, 30, or 45 minutes. The environment is operator-owned; merge-request text cannot change it. An explicit caller `--effort` passed after `ocr-ci review --` has normal OCR CLI precedence over the generated config, while an unknown environment value fails configuration before preview or model execution. -OCR 1.11.5 may present filter-surviving comments to a later round as previously confirmed, but the toolkit does not accept that wording as validation. Its mandatory background prefix travels with every main request and requires prior/filter-surviving findings to remain unverified until current code, tests, or trusted evidence support them. Survival cannot change severity, suppress or resolve a finding, authorize approval, or enter a receipt as independent validation. +OCR may present filter-surviving comments to a later round as previously confirmed, but the toolkit does not accept that wording as validation. Its mandatory background prefix travels with every main request and requires prior/filter-surviving findings to remain unverified until current code, tests, or trusted evidence support them. Survival cannot change severity, suppress or resolve a finding, authorize approval, or enter a receipt as independent validation. Effort controls review depth, not the prompt/context ceiling, per-call completion cap, aggregate token budget, or per-round tool limit. Semantic grouping and filtering can add requests even at `low`; higher effort can add further rounds until OCR stops early, reaches a coverage/budget boundary, or completes the configured depth. @@ -95,13 +99,13 @@ These names belong to `examples/gitlab/ocr-review.gitlab-ci.yml`; they are shell | Variable | Source / owner | Required | Exact default | Behavior | | --- | --- | --- | --- | --- | -| **`OCR_VERSION`** | Example pipeline | Yes | `v1.11.5` | Checksum-pinned recommended OCR binary release for toolkit 0.9.1. | -| **`OCR_SHA256`** | Example pipeline | Yes | `53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95` | Expected Linux AMD64 OCR binary digest. | +| **`OCR_VERSION`** | Example pipeline | Yes | `v` + manifest `recommended_version` | Exact binary release pinned with its asset checksum in the [pipeline](../examples/gitlab/ocr-review.gitlab-ci.yml); not resolved dynamically. | +| **`OCR_SHA256`** | Example pipeline | Yes | Manifest SHA-256 for `opencodereview-linux-amd64` | Exact digest pinned alongside `OCR_VERSION` in the pipeline; must belong to the same release entry. | | **`OCR_TOOLKIT_VERSION`** | Example pipeline | Yes | `0.9.0` | Exact toolkit wheel release installed by the current published example. | | **`OCR_TOOLKIT_CHECKSUMS_URL`** | Example pipeline | Yes | Release URL derived from `OCR_TOOLKIT_VERSION` | Toolkit `SHA256SUMS` URL. | | `OCR_TOOLKIT_WHEEL` | Example shell | Computed | `open_code_review_toolkit-${OCR_TOOLKIT_VERSION}-py3-none-any.whl` | Exact wheel filename selected from the release. | | `OCR_TOOLKIT_WHEEL_SHA256` | Example shell | Computed | Matching value from `SHA256SUMS` | Digest checked before installing the toolkit wheel. | -| `OCR_MAX_TOOLS` | Example pipeline / OCR CLI | No | `0` | OCR 1.11.5 uses template default `100`; `1-49` reports normalization to `50` but remains effectively `100`, and only a value above `100` raises the cap. | +| `OCR_MAX_TOOLS` | Example pipeline / OCR CLI | No | `0` | OCR uses template default `100`; `1-49` reports normalization to `50` but remains effectively `100`, and only a value above `100` raises the cap. | | `OCR_MAX_TOKENS_BUDGET` | Example pipeline / OCR CLI | No | `0` | Non-negative aggregate OCR token ceiling; `0` is unlimited. | During Draft qualification, install the toolkit artifact built from the reviewed @@ -181,11 +185,10 @@ finish and OCR accounts the provider-reported input plus output tokens. `OCR_MAX_TOOLS` is independent of all three token controls. Leave the example default at `0` so OCR uses its embedded template limit of `100` per subtask. -OCR 1.11.5 reports values `1-49` as normalized to the minimum `50`, but both +OCR reports values `1-49` as normalized to the minimum `50`, but both that target and explicit `50` remain below the template default and therefore remain effectively `100`. Use `101` or greater only when deliberately raising -the loop cap. OCR 1.11.0 corrected stale help text for this already-qualified -behavior; it did not change the runtime semantics. A recognized normalization +the loop cap. A recognized normalization is emitted only as a toolkit-authored CI notice; its raw stderr is not added to findings, result warnings, receipts, DLP inputs, telemetry, or automatic-approval signals. @@ -229,7 +232,7 @@ environment variables for policy thresholds or category lists in this release. ## Repository evidence -`ocr-ci review` owns this lifecycle. Before OCR starts it collects the exact immutable `--from`/`--to` refs (or the parent/commit pair selected by `--commit`), writes bounded redacted schema-versioned evidence, builds OCR's MCP registry with the mandatory evidence entry plus each independently configured optional server, reads the registry back, self-queries the evidence summary/list/get contract, and supplies the matching compact bootstrap to OCR. Those parent-process preflight calls are not counted as model use. The same preflight-qualified OCR executable first receives the exact production refs, rules, selection options, and background under `review --preview`; there is no toolkit threshold setting or duplicated OCR threshold constant. An exact recognized OCR soft background diagnostic becomes a toolkit-authored numeric `ocr.toolkit-advisory/v1` value only after publication DLP and appears in the bounded CI log plus GitLab Technical details. It is not an OCR warning, receipt or DLP input, coverage signal, telemetry field, or approval blocker. An exact recognized hard character/file-size rejection stops before model execution and leaves only an identity-bound numeric `ocr.pre-execution-status/v2` outcome for static GitLab reporting; the private path and raw OCR diagnostic are not published. Unknown preview failures fail closed through the generic diagnostic path, and the actual review independently revalidates the background. During OCR, the built-in MCP records count-only attempted and completed `summary`, `list`, `get`, `search`, and `coverage` actions without arguments, queries, IDs, paths, results, or content. Unknown or malformed MCP-dispatched primary-tool actions increment only a closed `unattributed` attempt counter. OCR also counts a dynamic tool request before parsing its JSON arguments; a parse failure never reaches the MCP owner, so receipt finalization adds only that count-only by-tool residual to `unattributed`. The parent reads and removes private action receipt v3 before cleanup. Receipt v8 is approval-valid only when MCP-received attempts do not exceed OCR's authoritative `tool_calls.by_tool` entries and every residual OCR attempt is accounted as unattributed; failed or malformed attempts cannot satisfy the mandatory completed `summary`, become successful evidence use, or authorize approval. Missing, malformed, raced, or mismatched attribution fails review finalization before a normal publishable result exists. Since OCR 1.9.9, `--background-file` takes precedence over inline `--background`, so `ocr-ci review` rejects caller forms of both options, including split and `--option=value` syntax, and remains the sole owner of the bootstrap input; caller `--preview` is likewise rejected because the toolkit owns this gate. OCR 1.10.0 adds `--output`, but `ocr-ci review` rejects its long, equals, short, and attached forms because the toolkit must remain the sole owner of the private result descriptor, atomic parsing, cleanup, and posting handoff. The 1.9.10 stage-grouped terminal retry report remains private and does not enter toolkit telemetry, receipts, DLP, findings, severity, outcomes, or approval. A completed OCR review is accepted only when structured `tool_calls.by_tool` proves at least one `ocr_toolkit_evidence` attempt and action receipt v3 proves a completed summary; a legitimately skipped no-supported-files review remains exempt. +`ocr-ci review` owns this lifecycle. Before OCR starts it collects the exact immutable `--from`/`--to` refs (or the parent/commit pair selected by `--commit`), writes bounded redacted schema-versioned evidence, builds OCR's MCP registry with the mandatory evidence entry plus each independently configured optional server, reads the registry back, self-queries the evidence summary/list/get contract, and supplies the matching compact bootstrap to OCR. Those parent-process preflight calls are not counted as model use. The same preflight-qualified OCR executable first receives the exact production refs, rules, selection options, and background under `review --preview`; there is no toolkit threshold setting or duplicated OCR threshold constant. An exact recognized OCR soft background diagnostic becomes a toolkit-authored numeric `ocr.toolkit-advisory/v1` value only after publication DLP and appears in the bounded CI log plus GitLab Technical details. It is not an OCR warning, receipt or DLP input, coverage signal, telemetry field, or approval blocker. An exact recognized hard character/file-size rejection stops before model execution and leaves only an identity-bound numeric `ocr.pre-execution-status/v2` outcome for static GitLab reporting; the private path and raw OCR diagnostic are not published. Unknown preview failures fail closed through the generic diagnostic path, and the actual review independently revalidates the background. During OCR, the built-in MCP records count-only attempted and completed `summary`, `list`, `get`, `search`, and `coverage` actions without arguments, queries, IDs, paths, results, or content. Unknown or malformed MCP-dispatched primary-tool actions increment only a closed `unattributed` attempt counter. OCR also counts a dynamic tool request before parsing its JSON arguments; a parse failure never reaches the MCP owner, so receipt finalization adds only that count-only by-tool residual to `unattributed`. The parent reads and removes private action receipt v3 before cleanup. Receipt v8 is approval-valid only when MCP-received attempts do not exceed OCR's authoritative `tool_calls.by_tool` entries and every residual OCR attempt is accounted as unattributed; failed or malformed attempts cannot satisfy the mandatory completed `summary`, become successful evidence use, or authorize approval. Missing, malformed, raced, or mismatched attribution fails review finalization before a normal publishable result exists. OCR `--background-file` takes precedence over inline `--background`, so `ocr-ci review` rejects caller forms of both options, including split and `--option=value` syntax, and remains the sole owner of the bootstrap input; caller `--preview` is likewise rejected because the toolkit owns this gate. OCR provides `--output`, but `ocr-ci review` rejects its long, equals, short, and attached forms because the toolkit must remain the sole owner of the private result descriptor, atomic parsing, cleanup, and posting handoff. The stage-grouped OCR terminal retry report remains private and does not enter toolkit telemetry, receipts, DLP, findings, severity, outcomes, or approval. A completed OCR review is accepted only when structured `tool_calls.by_tool` proves at least one `ocr_toolkit_evidence` attempt and action receipt v3 proves a completed summary; a legitimately skipped no-supported-files review remains exempt. The private `.review-context/evidence.json`, `.review-context/bootstrap.md`, repository-policy `.review-context/policy-rules.json`, and count-only evidence-action receipt/lock are internal implementation details, not public path configuration. Keep `.review-context/` ignored. The directory is mode `0700`, regular files are mode `0600`, and symlink, hard-link, non-regular, or unexpectedly permissive receipt targets are rejected. In GitLab MR pipelines, the provider adapter captures the current protected target SHA, fetches that exact immutable object when needed, and materializes only an in-repository `--rule` blob from it; explicit absolute rules outside the repository remain operator-owned. OCR still reviews the original forge diff-base-to-source-head range. The collector reads Git objects without checkout, does not follow repository symlinks or submodules, never executes repository content, and treats source-ref policy changes as untrusted. diff --git a/docs/development.md b/docs/development.md index 9dfe987..30b7b68 100644 --- a/docs/development.md +++ b/docs/development.md @@ -27,6 +27,14 @@ GitHub Actions storage is repository-owned infrastructure. Pull-request CI resto ## Maintaining OCR qualification +Current operator guidance describes the qualified OCR contract without repeating +release numbers or asset hashes; link to `compatibility/ocr-support.json` for the +exact recommendation of the same toolkit revision. Keep exact versions in +executable pins, binary-identity checks, release-specific plans/notes and historical +evidence. Generic fixtures derive the current identity from its runtime owner; +frozen historical fixtures must not follow the current pin. A version-neutral +description still needs semantic review when the consumed contract changes. + The live suite in `scripts/ocr_compat.py` qualifies the current toolkit-consumed OCR contract for every candidate. Do not add release-number branches, old parser fallbacks, or patch-specific fixture copies to live probes. Version checks belong diff --git a/docs/engineering/review_signal_ownership.md b/docs/engineering/review_signal_ownership.md index 955937c..3e57eb5 100644 --- a/docs/engineering/review_signal_ownership.md +++ b/docs/engineering/review_signal_ownership.md @@ -26,7 +26,7 @@ OCR telemetry remains opt-in. The toolkit defaults both `OCR_TELEMETRY_ENABLED` and `OCR_TELEMETRY_CONTENT_LOGGING` to `false` and adds no exporter of its own. -OCR 1.10.0 constructs group span names from sorted changed paths and attaches +OCR constructs group span names from sorted changed paths and attaches group path keys, model-produced labels, file counts, round numbers, churn, and filter counters to upstream spans and events. Disabling content logging must not be treated as removing those identifiers: operators who enable OCR telemetry diff --git a/docs/engineering/toolkit_strategy.md b/docs/engineering/toolkit_strategy.md index 9f673a2..6732a2c 100644 --- a/docs/engineering/toolkit_strategy.md +++ b/docs/engineering/toolkit_strategy.md @@ -154,7 +154,7 @@ Qualified OCR releases expose explicit per-run provider/model overrides, additiv The completed [BL-017 signal-ownership audit](review_signal_ownership.md) reaches `no-new-layer`. OCR owns provider, request, latency, cost, grouping, round, and tool telemetry. The toolkit owns only deterministic closed result, receipt, -context, publication-DLP, posting, and approval projections. OCR 1.11.0 group +context, publication-DLP, posting, and approval projections. OCR group keys contain changed paths, its labels are model-produced, and its private session can retain provider-native reasoning/tool state, so none becomes a toolkit metric, routing input, public summary, or approval authority. A single bounded diff --git a/docs/gitlab.md b/docs/gitlab.md index 2e0f9e5..c444656 100644 --- a/docs/gitlab.md +++ b/docs/gitlab.md @@ -20,9 +20,9 @@ The public pipeline stores the OCR binary checksum as the non-secret `OCR_SHA256 The pipeline explicitly sets `OCR_GITLAB_TARGET_PROTECTION_MODE: "required"`. This is the secure default: an unprotected target is rejected before OCR or model execution, and an explicit empty, malformed, mixed-case, or unknown value also fails closed. Set exact `unprotected` only when the project deliberately permits unprotected targets. That setting does not constrain a target GitLab reports as protected; normal protected behavior remains available. When the actual target is unprotected, receipt v8 enforces limited, comment-only review regardless of `OCR_AUTO_APPROVE`. -`OCR_REVIEW_LANGUAGE` defaults to `English`; `Russian` is one example of an explicit review language. The example pins its qualified OCR release and sets `OCR_REVIEW_EFFORT=medium`, allowing two review rounds; `low` and `high` explicitly select one or three. OCR may stop early when a round adds no finding. OCR 1.11.0 scales its 15-minute per-subtask base to 15/30/45 minutes for low/medium/high, so the example allows 45 minutes at the GitLab job boundary. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; explicit `50` also cannot lower the template, and only a value above `100` raises the cap. `OCR_MAX_TOKENS_BUDGET` defaults to `0`, meaning unlimited; a positive budget may stop dispatch and produce an explicitly partial, automatic-approval-ineligible review. `OCR_LLM_MAX_COMPLETION_TOKENS` defaults to unset and separately controls only the provider request's completion/output cap. Select an explicit value only from the deployment's provider/model contract; the toolkit does not recommend or hardcode a provider-specific cap. +`OCR_REVIEW_LANGUAGE` defaults to `English`; `Russian` is one example of an explicit review language. The example pins its qualified OCR release and sets `OCR_REVIEW_EFFORT=medium`, allowing two review rounds; `low` and `high` explicitly select one or three. OCR may stop early when a round adds no finding. OCR scales its 15-minute per-subtask base to 15/30/45 minutes for low/medium/high, so the example allows 45 minutes at the GitLab job boundary. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; explicit `50` also cannot lower the template, and only a value above `100` raises the cap. `OCR_MAX_TOKENS_BUDGET` defaults to `0`, meaning unlimited; a positive budget may stop dispatch and produce an explicitly partial, automatic-approval-ineligible review. `OCR_LLM_MAX_COMPLETION_TOKENS` defaults to unset and separately controls only the provider request's completion/output cap. Select an explicit value only from the deployment's provider/model contract; the toolkit does not recommend or hardcode a provider-specific cap. -OCR 1.11.0 semantically groups related changed files before review and filters candidates per group. Its private grouping and other-files prompts use status-first `STATUS path (+N/-M)` entries; the toolkit does not parse that inventory in production. Grouping, filtering, and multiple rounds can increase provider requests, latency, and token cost, while the manifest and aggregate budget continue to report completeness. Grouping, grace-round, provider-native reasoning/thinking, and tool-choice request state are retained only inside OCR's isolated private session. Group labels, paths, reasoning, signed/encrypted native payloads, and request controls remain untrusted: the toolkit does not publish or use them for severity, fingerprints, lifecycle commands, receipts, DLP counts, telemetry, tool/token summaries, or approval. OCR's separately configured telemetry may export high-cardinality repository-derived values, so keep `OCR_TELEMETRY_ENABLED=false` unless the exporter and retention policy are approved. +OCR semantically groups related changed files before review and filters candidates per group. Its private grouping and other-files prompts use status-first `STATUS path (+N/-M)` entries; the toolkit does not parse that inventory in production. Grouping, filtering, and multiple rounds can increase provider requests, latency, and token cost, while the manifest and aggregate budget continue to report completeness. Grouping, grace-round, provider-native reasoning/thinking, and tool-choice request state are retained only inside OCR's isolated private session. Group labels, paths, reasoning, signed/encrypted native payloads, and request controls remain untrusted: the toolkit does not publish or use them for severity, fingerprints, lifecycle commands, receipts, DLP counts, telemetry, tool/token summaries, or approval. OCR's separately configured telemetry may export high-cardinality repository-derived values, so keep `OCR_TELEMETRY_ENABLED=false` unless the exporter and retention policy are approved. The summary keeps four inputs separate: OCR manifest coverage, publication integrity, published findings, and an optional OCR core advisory. Complete manifest coverage that later loses public fields is labelled `Review complete with publication filtering`; it is warning-bearing and approval-ineligible, but it is not called incomplete OCR coverage and does not invent failed-file diagnostics. Actual partial, failed, waived, or budget-stopped manifest coverage keeps its stronger status. Ordinary OCR warnings retain their own complete-with-warnings state. The reviewer guide ranks bounded focus areas only when at least two findings are published; one finding remains in its inline or fallback discussion without a duplicate snippet. Security focus uses explicit published metadata, strong vulnerability terms, and closed injection classes rather than the standalone word `injection`, so neutral knowledge/dependency terminology cannot increase its count or effort estimate. diff --git a/docs/operations.md b/docs/operations.md index 7cf374e..d1795da 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -2,6 +2,9 @@ This guide is for developers and CI operators who connect Open Code Review Toolkit to a GitLab merge-request pipeline and need to understand what happens after the first review. Installation and the [production bot recipes](gitlab.md#production-bot-configuration) remain in `gitlab.md`; the complete environment contract is in [configuration.md](configuration.md), and enriched acquisition is in [bounded review context](review-context.md). +OCR behavior below refers to the exact recommended release in the +[compatibility manifest](../compatibility/ocr-support.json) for this toolkit revision. + ## What one review run publishes The toolkit reads the previous OCR-owned notes and discussions before it writes anything. It fingerprints new findings, removes findings already owned or suppressed by reviewers, and publishes: @@ -34,11 +37,11 @@ suppressed. `OCR_MAX_TOKENS_BUDGET` can set an aggregate input-plus-output token ceiling for the OCR diff review. The default `0` is unlimited. A positive ceiling is approximate rather than a hard billing cutoff because already-running work may complete; when it stops further dispatch, completed findings remain publishable and unreviewed files stay explicit as budget-attributed failed coverage. Such a run is partial and cannot automatically approve. -This aggregate budget is separate from both OCR's prompt/context `max_tokens` ceiling and the provider request's completion/output cap. The toolkit does not add an environment alias for OCR's prompt/context control. `OCR_LLM_MAX_COMPLETION_TOKENS` defaults to unset and, when set, overlays only the protocol-specific output field. The inherited OpenAI value was 58,888 in OCR 1.9.10 and is 16,384 in OCR 1.10.0 through 1.11.3, including grouping requests; an explicit toolkit override applies to all requests. A gateway may reserve cost against that requested maximum before generation even when the eventual response would be short. The `/models` capability value does not reveal an account spending limit or reservation policy, so the toolkit never selects the cap from it automatically. Set an explicit cap only from the deployment's provider/model contract when that deployment must not change with the qualified OCR version; the toolkit does not recommend or hardcode a provider-specific value. +This aggregate budget is separate from both OCR's prompt/context `max_tokens` ceiling and the provider request's completion/output cap. The toolkit does not add an environment alias for OCR's prompt/context control. `OCR_LLM_MAX_COMPLETION_TOKENS` defaults to unset and, when set, overlays only the protocol-specific output field. The qualified inherited OpenAI value is 16,384, including grouping requests; [compatibility history](compatibility.md) records earlier values and their exact versions; an explicit toolkit override applies to all requests. A gateway may reserve cost against that requested maximum before generation even when the eventual response would be short. The `/models` capability value does not reveal an account spending limit or reservation policy, so the toolkit never selects the cap from it automatically. Set an explicit cap only from the deployment's provider/model contract when that deployment must not change with the qualified OCR version; the toolkit does not recommend or hardcode a provider-specific value. -`OCR_REVIEW_EFFORT=medium` is the toolkit default for the qualified OCR release and permits two review rounds; `low` permits one and `high` permits three. OCR 1.11.3 multiplies its 15-minute subtask base by those rounds, yielding 15/30/45-minute limits; the GitLab example's 45-minute job timeout accommodates the largest preset. This is a review-depth choice, not one of the three token controls. OCR first groups related changed files and may make group-filter requests; additional rounds can add requests, latency, and cost, but may stop early when they add no finding. Below four changed files, OCR 1.11.3 skips the grouping LLM: low total churn is bundled and high churn stays per-file. The published GitLab example separately passes `OCR_MAX_TOOLS=0`, selecting the embedded template default `100`. The CLI reports `1-49` as normalized to `50`, but both that target and explicit `50` remain effectively `100`; only values above `100` raise the cap. OCR 1.11.0 corrects stale help text for the already-qualified behavior; OCR 1.11.3 leaves that runtime loop contract unchanged. Exhausting the effective limit, an aggregate budget stop, or incomplete manifest coverage remains explicit and approval-ineligible; increasing either effort or tools is never a way to hide partial coverage. +`OCR_REVIEW_EFFORT=medium` is the toolkit default for the qualified OCR release and permits two review rounds; `low` permits one and `high` permits three. OCR multiplies its 15-minute subtask base by those rounds, yielding 15/30/45-minute limits; the GitLab example's 45-minute job timeout accommodates the largest preset. This is a review-depth choice, not one of the three token controls. OCR first groups related changed files and may make group-filter requests; additional rounds can add requests, latency, and cost, but may stop early when they add no finding. Below four changed files, OCR skips the grouping LLM: low total churn is bundled and high churn stays per-file. The published GitLab example separately passes `OCR_MAX_TOOLS=0`, selecting the embedded template default `100`. The CLI reports `1-49` as normalized to `50`, but both that target and explicit `50` remain effectively `100`; only values above `100` raise the cap. Exhausting the effective limit, an aggregate budget stop, or incomplete manifest coverage remains explicit and approval-ineligible; increasing either effort or tools is never a way to hide partial coverage. -The outcome wording distinguishes skipped, complete, complete-with-warnings, incomplete, token-budget, and failed reviews while preserving the finding state in that same line. A complete clean review is visibly positive; a complete review with findings or only reviewer-suppressed findings is neutral; warning, partial, budget, and failed states never look clean. Findings withheld by the posting limit remain counted even when the limit allows no individual finding note. Recommended focus areas ranks only its copy of already-published findings by the closed severity, category, safe repository location, and stable-identity order before its existing display cap; inline and fallback discussion order, suppression, counts, security focus, and approval policy remain unchanged. OCR 1.8.5 and later manifest failures provide the canonical failed-file receipt; legacy warnings are a bounded fallback, and `summary.files_reviewed` is never treated as proof of successful coverage. Technical details label the aggregate as all OCR tool calls and retain the existing inline format while listing every admitted non-zero count for the closed native/context/evidence review set. An empty admitted list produces no tool-call line. The counts describe review activity, not per-tool token consumption: one read or search can return a different amount of context from another. Dynamic external tool names remain private; reconciled MCP-server attempt totals stay in their separate aggregate, and completed built-in evidence `summary`/`list`/`get` plus dedicated `search`/`coverage` counts appear only after action-receipt v3 accounts for every MCP-received or OCR-preparse attempted call. Failed and malformed attempts remain private, cannot appear as completed evidence, and cannot satisfy the mandatory summary. Zero completed action counters, unattributed attempts, queries, scopes, IDs, token counters, and configured-but-unused MCP servers are omitted. A present but incomplete or invalid receipt is rejected before prior comments are inspected or normal findings are published; absence alone selects the compatible direct-posting path. Token usage renders only validated input/output/cached/reasoning/total/derived-other buckets; malformed or contradictory counters are unavailable and unknown provider keys are not published. Status and aggregate semantic-category emoji are enabled by default and can be disabled together with `OCR_POST_EMOJI=false`; finding labels remain text unless their separate badge mode is enabled. +The outcome wording distinguishes skipped, complete, complete-with-warnings, incomplete, token-budget, and failed reviews while preserving the finding state in that same line. A complete clean review is visibly positive; a complete review with findings or only reviewer-suppressed findings is neutral; warning, partial, budget, and failed states never look clean. Findings withheld by the posting limit remain counted even when the limit allows no individual finding note. Recommended focus areas ranks only its copy of already-published findings by the closed severity, category, safe repository location, and stable-identity order before its existing display cap; inline and fallback discussion order, suppression, counts, security focus, and approval policy remain unchanged. OCR manifest failures provide the canonical failed-file receipt; legacy warnings are a bounded fallback, and `summary.files_reviewed` is never treated as proof of successful coverage. Technical details label the aggregate as all OCR tool calls and retain the existing inline format while listing every admitted non-zero count for the closed native/context/evidence review set. An empty admitted list produces no tool-call line. The counts describe review activity, not per-tool token consumption: one read or search can return a different amount of context from another. Dynamic external tool names remain private; reconciled MCP-server attempt totals stay in their separate aggregate, and completed built-in evidence `summary`/`list`/`get` plus dedicated `search`/`coverage` counts appear only after action-receipt v3 accounts for every MCP-received or OCR-preparse attempted call. Failed and malformed attempts remain private, cannot appear as completed evidence, and cannot satisfy the mandatory summary. Zero completed action counters, unattributed attempts, queries, scopes, IDs, token counters, and configured-but-unused MCP servers are omitted. A present but incomplete or invalid receipt is rejected before prior comments are inspected or normal findings are published; absence alone selects the compatible direct-posting path. Token usage renders only validated input/output/cached/reasoning/total/derived-other buckets; malformed or contradictory counters are unavailable and unknown provider keys are not published. Status and aggregate semantic-category emoji are enabled by default and can be disabled together with `OCR_POST_EMOJI=false`; finding labels remain text unless their separate badge mode is enabled. When a fully validated receipt v8 records an actually unprotected target, the summary adds exactly *The target branch was not protected in GitLab. This review ran in limited, comment-only mode.* immediately after the primary status line. It appears for clean, findings, warnings, partial, budget-stopped, failed-result, and publication-filtered summaries without replacing or reclassifying the canonical outcome. The limitation is a trust and approval boundary, not a coverage result. A protected target does not show it merely because the operator allowed unprotected targets; legacy, malformed, or contradictory receipt state cannot show it as trusted status. @@ -141,7 +144,7 @@ Receipt v8 is emitted only for a validated GitLab merge-request profile with a r Run OCR through `ocr-ci review --result PATH --stderr PATH -- ...`. This wrapper does not post to GitLab: it creates private artifacts, acquires enriched context when selected, asks the exact resolved and preflight-qualified OCR executable to preview the production refs/rules/selection/background without an LLM, then runs the model review only if OCR accepts that background. OCR owns the current recommendation and rejection thresholds; the toolkit has no threshold setting. A recognized soft diagnostic is reduced to a toolkit-authored `ocr.toolkit-advisory/v1` enum and two positive character counts. It is attached after publication DLP, rendered only with an exact receipt v8 in Technical details, and does not change warnings, coverage, DLP counts, telemetry, or automatic approval. A recognized hard character/file-size rejection stops before the model and lets `ocr-ci post` publish only a static numeric failure summary; the OCR path and raw diagnostic remain private. Unknown preview failures use the generic fail-closed diagnostic path. The ordinary review still validates the same background, the wrapper validates the complete output, and context/session/configuration data is removed. On an unclassified ordinary failure it prints only a bounded redacted stderr excerpt to the runner log; a classified provider failure keeps that excerpt private. Pass the paths and captured exit code to `ocr-ci post` afterward. Set `OCR_POST_ERROR_DETAILS=1` only when the generic path's safe excerpt should also appear in the merge-request failure note. Cleanup uncertainty blocks result publication. DLP atomically converts unsafe publication output into a safe `completed_with_errors` subset, but sanitizes unsafe private-only result fields without discarding an otherwise valid manifest or finding set. Safe findings are posted, unsafe finding content/warnings and unsafe optional fields are omitted, previous OCR comments remain, and matching prior findings are consumed one-for-one rather than duplicated. Receipt v8 and the `ocr.publication-dlp-signal/v2` marker distinguish `private-sanitized`, where the canonical published and approval-relevant projection is unchanged, from approval-ineligible `publication-filtered`, where the public projection is incomplete even when the receipt proves OCR coverage was complete. Posting uses only the validated original coverage kind and counts for status and Technical details; it never reconstructs or republishes rejected values. Horizontal tabs survive only in `existing_code` and `suggestion_code`; the unchanged values pass the same secret, PII, forbidden-value, laundering, and budget checks, while tabs elsewhere and all other control/format characters remain blocking. The same count-only JSON is logged as `OCR toolkit telemetry event` for optional CI collection/alerting. It is not an OTLP/network exporter and contains no rejected value or location. Never interpret a filtered subset as a full publishable review or as proof that OCR coverage was partial. -OCR 1.11.3 may add group labels, file membership, grouping-task records, round diagnostics, readable reasoning, and provider-native signed or encrypted replay payloads to its private session data. Safe values remain private; the isolated OCR home is removed before publication. Unknown private-only result fields are sanitized through the existing DLP owner, while any value reaching a canonical finding or warning makes publication incomplete and approval-ineligible. Reasoning, native payloads, and `tool_choice` are deliberately absent from the canonical finding/posting projection, receipt v8, DLP counts, toolkit telemetry, tool/token summary, and approval inputs. If they appear inside receipt v8, the receipt is invalid and approval fails closed. Caller `--output`/`-o` is rejected before preview: only `ocr-ci review --result` owns the result descriptor and posting handoff. +OCR may add group labels, file membership, grouping-task records, round diagnostics, readable reasoning, and provider-native signed or encrypted replay payloads to its private session data. Safe values remain private; the isolated OCR home is removed before publication. Unknown private-only result fields are sanitized through the existing DLP owner, while any value reaching a canonical finding or warning makes publication incomplete and approval-ineligible. Reasoning, native payloads, and `tool_choice` are deliberately absent from the canonical finding/posting projection, receipt v8, DLP counts, toolkit telemetry, tool/token summary, and approval inputs. If they appear inside receipt v8, the receipt is invalid and approval fails closed. Caller `--output`/`-o` is rejected before preview: only `ocr-ci review --result` owns the result descriptor and posting handoff. When OCR exits nonzero with a valid bounded `ocr.llm-retry-report/v1`, the toolkit reads only its closed error class, failure phase, terminal outcome, HTTP status, and schema counters. It maps those facts to `authentication`, `authorization`, `rate-or-spending-limit`, `overloaded`, `timeout`, `network`, `endpoint-or-model-not-found`, `request-rejected`, `provider-unavailable`, `invalid-response`, `cancelled`, `mixed`, or `unknown`, then writes a completely toolkit-authored note. A runtime `404` remains `endpoint-or-model-not-found` because safely distinguishing the endpoint from the model would require trusting the raw response body. diff --git a/docs/security.md b/docs/security.md index becde31..27b6563 100644 --- a/docs/security.md +++ b/docs/security.md @@ -126,7 +126,7 @@ Policy v2 adds optional toolkit-owned remediation threads without changing these - **Identity spoofing and privacy:** represent authors by provider-declared class plus run-local pseudonym; omit name, email, avatar, and profile URL. Unknown classification fails closed. - **TOCTOU/replay/cache poisoning:** bind snapshots and handles to version/ETag or digest, policy version, run, and expiry; use atomic store commits and uniform unavailable outcomes. - **PII/secret bypass and output laundering:** make retrieval, model egress, publication sinks, and private retention separate decisions. Minimize before model egress; publication DLP cannot undo prior disclosure. Scan decoded source and rendered forms so HTML comments/entities/tags, Markdown inline/reference/autolink destinations, controls, and Unicode formatting cannot hide exact protected values. Sink uncertainty yields a safe partial result; private-only uncertainty yields an atomically sanitized valid result or falls back to that partial form. Expose only low-cardinality receipt/summary/log signals in ordinary runs. The explicit local-only diagnostic switch retains owner-only raw private state, emits no receipt, and is rejected by the validated GitLab MR profile. -- **Upstream session and telemetry privacy:** OCR 1.11.0 can retain readable reasoning, signed/encrypted provider-native replay payloads, tool arguments/results, sorted changed paths, model-produced group labels, and round counters in its owner-only session. Telemetry may also carry repository-derived high-cardinality grouping values. OCR telemetry is disabled by default, and the toolkit removes the isolated session home before publication. Enabling an exporter authorizes it to receive the fields OCR emits; content-logging disablement is not a promise that every identifier disappears. The toolkit neither ingests nor duplicates these values. +- **Upstream session and telemetry privacy:** OCR can retain readable reasoning, signed/encrypted provider-native replay payloads, tool arguments/results, sorted changed paths, model-produced group labels, and round counters in its owner-only session. Telemetry may also carry repository-derived high-cardinality grouping values. OCR telemetry is disabled by default, and the toolkit removes the isolated session home before publication. Enabling an exporter authorizes it to receive the fields OCR emits; content-logging disablement is not a promise that every identifier disappears. The toolkit neither ingests nor duplicates these values. - **OCR advisory spoofing:** raw OCR output cannot supply `_ocr_toolkit_advisory`. The toolkit adds only the exact numeric `ocr.toolkit-advisory/v1` shape after publication DLP; posting requires a valid receipt v8 and rejects duplicate, extended, unknown, or malformed forms. The advisory cannot change warnings, coverage, DLP, telemetry, findings, or approval. - **Local executable and subprocess confused deputy:** accept the OCR executable only through absolute search-path entries and reject resolution inside the reviewed repository. Stdio adapter deadlines cover request delivery as well as response/exit, use a clean environment and isolated directory, and terminate the process group on uncertainty. - **Marker and retry spoofing:** recognize ownership, run-summary, and setup identities only in the exact toolkit preamble. Repository/provider text containing marker lookalikes has no authority. Duplicate findings consume visible prior occurrences one-for-one; successful setup retries replace only previous toolkit setup-pending notes. diff --git a/examples/gitlab/README.md b/examples/gitlab/README.md index fb272db..4b379f9 100644 --- a/examples/gitlab/README.md +++ b/examples/gitlab/README.md @@ -2,7 +2,7 @@ [`ocr-review.gitlab-ci.yml`](ocr-review.gitlab-ci.yml) is the complete checksum-pinned pipeline. It defaults to identity-only review, explicitly keeps `OCR_GITLAB_TARGET_PROTECTION_MODE: "required"`, and contains the install, preflight, configure, review, and post lifecycle. Copy one mode file below into the pipeline's top-level `variables` mapping, or include exactly one file after reviewing its trust boundary. -The pipeline pins OCR 1.11.5 and explicitly sets `OCR_REVIEW_EFFORT=medium`, which permits two review rounds. Use `low` for one round when latency/cost is the priority, or `high` for three only after accepting the additional provider work. OCR uses a 15-minute subtask base and scales it to 15/30/45 minutes for low/medium/high; the example's 45-minute job timeout accommodates every preset without adding another environment control. Semantic grouping and filtering are OCR behavior shared by every context mode; they do not change which merge-request text a mode admits. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; only values above `100` raise the per-subtask cap. `OCR_MAX_TOKENS_BUDGET` and the optional completion cap remain independent controls. +The pipeline pins the exact qualified OCR release from the [compatibility manifest](../../compatibility/ocr-support.json) and explicitly sets `OCR_REVIEW_EFFORT=medium`, which permits two review rounds. Use `low` for one round when latency/cost is the priority, or `high` for three only after accepting the additional provider work. OCR uses a 15-minute subtask base and scales it to 15/30/45 minutes for low/medium/high; the example's 45-minute job timeout accommodates every preset without adding another environment control. Semantic grouping and filtering are OCR behavior shared by every context mode; they do not change which merge-request text a mode admits. `OCR_MAX_TOOLS=0` selects the embedded template default `100`. Values `1-49` report normalization to `50` but remain effectively `100`; only values above `100` raise the per-subtask cap. `OCR_MAX_TOKENS_BUDGET` and the optional completion cap remain independent controls. | Mode | Recipe | MR text admitted | External access | Automatic approval | | --- | --- | --- | --- | --- | diff --git a/tests/test_environment_contract.py b/tests/test_environment_contract.py index 2791ad8..6d41d02 100644 --- a/tests/test_environment_contract.py +++ b/tests/test_environment_contract.py @@ -70,8 +70,9 @@ } EXAMPLE_DEFAULTS = { - "OCR_VERSION": "v1.11.5", - "OCR_SHA256": "53a4ab7c8ce6dc07d5362c7c4984bf8d98b55e4e8d4c01b9399d488a2a983d95", + # Executable pin equality is checked against the manifest in integration contracts. + "OCR_VERSION": "`v` + manifest `recommended_version`", + "OCR_SHA256": "Manifest SHA-256 for `opencodereview-linux-amd64`", "OCR_TOOLKIT_VERSION": STABLE_TOOLKIT_VERSION, "OCR_TOOLKIT_CHECKSUMS_URL": "Release URL derived from `OCR_TOOLKIT_VERSION`", "OCR_TOOLKIT_WHEEL": "open_code_review_toolkit-${OCR_TOOLKIT_VERSION}-py3-none-any.whl", @@ -302,12 +303,12 @@ def test_removed_and_redaction_only_names_do_not_reenter_public_configuration() def test_example_local_defaults_match_the_pipeline() -> None: + """Check example controls; integration contracts own exact OCR pin equality.""" + workflow = (PROJECT_ROOT / "examples" / "gitlab" / "ocr-review.gitlab-ci.yml").read_text( encoding="utf-8" ) for name, value in { - "OCR_VERSION": "v1.11.5", - "OCR_SHA256": EXAMPLE_DEFAULTS["OCR_SHA256"], "OCR_TOOLKIT_VERSION": STABLE_TOOLKIT_VERSION, "OCR_MAX_TOOLS": "0", "OCR_MAX_TOKENS_BUDGET": "0", diff --git a/tests/test_operations_docs.py b/tests/test_operations_docs.py index 79fca8b..0d5a6a1 100644 --- a/tests/test_operations_docs.py +++ b/tests/test_operations_docs.py @@ -1,6 +1,5 @@ """Contracts for the public GitLab operations documentation.""" -import json import re from pathlib import Path @@ -63,27 +62,45 @@ def test_readme_install_is_isolated_checksum_pinned_and_no_llm() -> None: readme = README.read_text(encoding="utf-8") install = readme.split("## Install", 1)[1].split("## How reviews evolve", 1)[0] - manifest = json.loads( - (PROJECT_ROOT / "compatibility" / "ocr-support.json").read_text(encoding="utf-8") - ) - recommended = manifest["recommended_version"] - release = next(item for item in manifest["releases"] if item["version"] == recommended) - digests = {asset["name"]: asset["sha256"] for asset in release["assets"]} - assert "Python 3.12 through 3.14" in install assert "uv tool install open-code-review-toolkit" in install assert install.index(". .venv/bin/activate") < install.index( "python -m pip install open-code-review-toolkit" ) - assert f"Open Code Review {recommended}" in install - assert f"open-code-review v{recommended}" in install - assert digests["opencodereview-linux-amd64"] in install - assert digests["opencodereview-darwin-arm64"] in install + assert "compatibility/ocr-support.json" in install + assert "`recommended_version`" in install + assert "for this toolkit revision" in install + assert "https://github.com/alibaba/open-code-review/releases)" in install + assert "release entry's asset SHA-256" in install + assert "Do not substitute the latest upstream" in install + assert "manifest's exact `recommended_version`" in install + assert not re.search(r"\b[a-f0-9]{64}\b", install) assert "ocr --version" in install assert "ocr-ci --help" in install assert "not the installation\nsmoke test" in install +def test_current_operator_guidance_uses_manifest_without_ocr_release_literals() -> None: + """Keep living guidance tied to qualified pins without per-release text churn.""" + + for document in ( + README, + CONFIGURATION, + OPERATIONS, + GITLAB_GUIDE, + GITLAB_EXAMPLES / "README.md", + ): + content = document.read_text(encoding="utf-8") + assert "compatibility/ocr-support.json" in content + assert not re.search(r"(?:OCR|Open Code Review) v?\d+\.\d+\.\d+", content) + assert not re.search(r"open-code-review/releases/tag/v\d+\.\d+\.\d+", content) + configuration = CONFIGURATION.read_text(encoding="utf-8") + assert "`v` + manifest `recommended_version`" in configuration + assert "not resolved dynamically" in configuration + assert "Manifest SHA-256 for `opencodereview-linux-amd64`" in configuration + assert not re.search(r"\b[a-f0-9]{64}\b", configuration) + + def test_documentation_indexes_route_to_canonical_owners() -> None: docs_index = (PROJECT_ROOT / "docs" / "README.md").read_text(encoding="utf-8") codex_index = (PROJECT_ROOT / "docs" / "codex" / "README.md").read_text(encoding="utf-8") @@ -785,7 +802,9 @@ def test_numeric_ocr_controls_use_behavioral_qualification_and_template_delegati assert "raw stderr is not added to\nfindings, result warnings" in configuration assert "receipts, DLP inputs, telemetry" in configuration assert "`OCR_MAX_TOOLS=0`" in operations - assert "corrects stale help text for the already-qualified behavior" in operations + assert "normalized to `50`" in operations + assert "explicit `50` remain effectively `100`" in operations + assert "only values above `100` raise the cap" in operations assert "effective `100` for omitted, sentinel `0`, `49`, and `50`" in compatibility assert "help text\n alone is not compatibility evidence" in development diff --git a/tests/test_runtime_helpers.py b/tests/test_runtime_helpers.py index 5a26cbf..ca2b56a 100644 --- a/tests/test_runtime_helpers.py +++ b/tests/test_runtime_helpers.py @@ -1568,8 +1568,13 @@ def test_invalid_json_error_does_not_echo_secret_payload(self) -> None: class PreflightTests(unittest.TestCase): def test_validate_ocr_binary_accepts_supported_version(self) -> None: + """Exercise the accepted identity using the production pin owner.""" + completed = subprocess.CompletedProcess( - args=["ocr", "--version"], returncode=0, stdout="ocr 1.11.5\n", stderr="" + args=["ocr", "--version"], + returncode=0, + stdout=f"ocr {preflight.EXPECTED_OCR_VERSION}\n", + stderr="", ) with ( patched_attr(preflight.shutil, "which", lambda _name: "/usr/bin/ocr"), From 57f523685aa9e40f2832e62046744547fc1ee776 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Mon, 7 Sep 2026 09:44:40 +0200 Subject: [PATCH 7/9] fix: close malformed OCR qualification probes --- PLANS.md | 89 +++++++++++++++++----------- changelog.d/176.bugfix.md | 2 + scripts/ocr_compat.py | 19 +++++- tests/test_ocr_compat.py | 120 +++++++++++++++++++++++++++++++++++++- 4 files changed, 191 insertions(+), 39 deletions(-) diff --git a/PLANS.md b/PLANS.md index 210845f..6bb5502 100644 --- a/PLANS.md +++ b/PLANS.md @@ -8,17 +8,19 @@ before handoff or commit. Completed stable plans are indexed in ### Toolkit 0.9.1 — OCR 1.11.4 and 1.11.5 -- **Status:** active +- **Status:** active; external model qualification complete, remediation in progress - **Plan Origin:** plan_mode_approved -- **Release classification:** release-required; stable delivery release-deferred +- **Release classification:** release-required; stable delivery authorized - **Target stable version:** 0.9.1 - **Branch:** `codex/v0.9.1-ocr-1.11.5` #### Goal -Integrate OCR 1.11.4 and 1.11.5 through a green published Draft PR, with exact -1.11.5 runtime support and precise diagnostic/DLP boundaries that preserve GitLab -summary publication. External configured qualification owns the later release decision. +Integrate OCR 1.11.4 and 1.11.5 through a reviewed protected PR and complete the +0.9.1 stable release, with exact 1.11.5 runtime support and precise diagnostic/DLP +boundaries that preserve GitLab summary publication. Remediate the configured model +qualification findings before feature merge and independently verify every published +artifact, receipt and tracked issue before closure. #### Requested Scope @@ -26,7 +28,13 @@ summary publication. External configured qualification owns the later release de - Accept bounded private failure arguments without exposing them or degrading publication. - Qualify serialized comment repair, Objective-C++ routing, grouped review and defaults. - Update local OCR to checksum-verified 1.11.5, current docs, examples and changelog. -- Publish a green Draft with external qualification instructions; keep issues/milestone open. +- Remediate the completed model-backed OCR findings without discarding its owner-only + evidence, then run the holistic quality and Codex Security gates. +- Publish only the completed feature head, finish the protected feature and release PRs, + verify stable delivery, close #176/#177 and milestone `v0.9.1`, clean branches and + synchronize `main`. +- Complete the required no-release external-reconciliation PR without producing another + stable release. #### Requirement Traceability @@ -38,16 +46,20 @@ summary publication. External configured qualification owns the later release de | REQ-004 | Comment repair and Objective-C++ routing are qualified | WQ-04 | deterministic real OCR gateway and rules previews | | REQ-005 | Grouping, defaults, MCP and receipt boundaries remain correct | WQ-04 | compatibility and runtime tests | | REQ-006 | Docs, decision flow, changelog and backlog reflect current behavior | WQ-05 | documentation checks and rendered Towncrier | -| REQ-007 | Green pushed Draft with truthful external qualification | WQ-06 | local/hosted gates and remote readback | +| REQ-007 | Green protected feature PR with truthful model qualification | WQ-06, WQ-08 | local/hosted gates, OCR receipt and remote readback | | REQ-008 | Forward-only live qualification, isolated historical readback and stable tests | WQ-04R | no live version branches, frozen-history validation, CLI and probe regressions | | REQ-009 | Version-neutral current guidance with exact compatibility identities preserved | WQ-07 | documentation contracts, pin validation, self-review and green Draft push | +| REQ-010 | Malformed probe output always fails through the closed qualification status path | WQ-08 | focused shape matrix and CLI status-output regression | +| REQ-011 | Stable 0.9.1 is published and independently reconciled | WQ-09, WQ-10 | release workflow, registry/provenance/install readback and protected closure PR | #### Explicit Non-Goals -No real LLM calls, merge, release PR, tags, package publication, or issue/milestone -closure. No new environment variables, MCP tools, receipt schemas, upstream repair -implementation, or runtime compatibility fallback. No OCR config/credentials or -user HOME changes. No separate Codex Security scan. +No GitLab posting, GitLab MR, new environment variables, MCP tools, receipt schemas, +upstream repair implementation, runtime compatibility fallback, or unrelated B2B, +`core/common` or shared-template work. Do not expose OCR configuration, credentials, +raw private output or owner-only evidence. Do not repeat the completed model-backed +OCR run without separate authorization; remediation is verified with deterministic +tests and the required Codex Security diff scan. #### Constraints @@ -68,9 +80,9 @@ New tests receive purpose-focused docstrings and stay with existing owners. #### User Decisions And Answers -Delivery stops at a published green Draft. External configured environment owns -real model testing and confirmation. Local OCR may immediately advance to 1.11.5 -after plan materialization; all local binary checks use isolated HOME and no LLM. +Stable delivery is authorized through final reconciliation. The configured model-backed +review has completed without GitLab posting and owns the two parser-hardening findings +below. Local OCR is 1.11.5; all deterministic binary checks use isolated HOME. Private failure arguments alone must not alter DLP counts, review status, findings or GitLab summary. Public secrets remain subject to normal DLP filtering. @@ -93,6 +105,9 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-05 | done | Public docs, decision flow, changelog and backlog reconciliation | | WQ-06 | in_progress | Final local gate, push, hosted checks and external Draft handoff | | WQ-07 | done | Version-neutral guidance and generic fixtures validated and self-reviewed; Draft push/readback remains owned by WQ-06 | +| WQ-08 | remediation_done | Malformed-output remediation and focused fuzz-style tests complete; holistic review and Codex Security diff scan remain | +| WQ-09 | pending | Push completed feature head, make PR ready, verify checks/threads, squash-merge and verify development publication | +| WQ-10 | pending | Prepare/merge `Release v0.9.1`, verify stable delivery and close issues/milestone; then no-release external reconciliation and cleanup | #### Locked Decisions @@ -168,6 +183,19 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. Towncrier and diff checks passed. Self-review removed the remaining duplicated documentation checksum and separated manifest-reference assertions from exact executable-pin assertions. Runtime, pins and historical evidence are unchanged. +- 2026-09-07: configured model-backed OCR 1.11.5 completed on immutable range + `ae0a9ac..029cdf0`: 9/9 selected items completed, 0 failed/reused/waived, + 77 tool calls and no GitLab posting. Owner-only result SHA-256 is + `3a5374a846c3033734da0c66d7a9032e54043e96b8a8322fd9724f37691c2873`. + The retained evidence is under `/tmp/ocr-v091-plan.1B3JOS` with mode 0700/0600. + Two medium findings identify uncontrolled `JSONDecodeError` and non-object + payload/detail shapes in `_comment_arguments_probe`; both must become controlled + contract-probe failures and preserve a safe `--status-output` result. +- 2026-09-07: `_comment_arguments_probe` now rejects malformed JSON, non-object + top-level values, non-object comments, invalid `tool_calls`/`failure_details` and + non-object detail entries through `CompatibilityError`. Sixteen focused cases pass, + including the complete qualification CLI path to a private stderr diagnostic and + closed `contracts/contract-probe-failed` status without traceback or evidence output. #### Risks And Recovery @@ -179,18 +207,9 @@ outputs until verification, then remove only task-owned files. #### Resume Point -WQ-07's version-neutral follow-up is locally validated and self-reviewed. WQ-06 -owns the signed commit's pinned Gitleaks check, push to Draft #178 and current -remote-head/hosted-check readback. The PR receipt records that external state; -do not infer the latest head's checks from a predecessor's green result. - -Local implementation and validation are complete. WQ-06's external receipt is -Draft PR #178: verify its current remote head and hosted checks before continuing. -If the head has green checks and the recorded runtime tree is unchanged, the next -action is configured external model qualification with OCR 1.11.5 and the toolkit -artifact built from that exact head. Otherwise inspect the failed hosted gate and -apply only an evidence-driven correction. Keep the PR Draft and both issues open; -the example's published toolkit package pin advances only in a later release PR. +WQ-08's remediation is focused-tested and ready for its signed logical commit. +Preserve the existing OCR evidence. Continue with one holistic branch review, full +quality, pinned Gitleaks and Codex Security before pushing the finished head. #### Plan Fidelity Check @@ -206,18 +225,18 @@ the example's published toolkit package pin advances only in a later release PR. #### Closure Gate -- [ ] Implementation and required local/hosted checks complete. -- [ ] Draft open and mergeable; unresolved review threads absent. -- [ ] Issues and milestone open; external qualification checklist recorded. -- [ ] Worktree clean and resume point reflects external qualification. +- [ ] Implementation, OCR remediation, Codex Security and local/hosted checks complete. +- [ ] Feature and release PR exact heads merged through the protected process. +- [ ] TestPyPI/PyPI bytes, provenance, attestations, tag, immutable Release and installs verified. +- [ ] #176/#177 and milestone closed from exact receipts; branches cleaned and `main` synchronized. +- [ ] No-release reconciliation PR merged without changing stable artifacts. #### Post-Close Delivery -External agent starts with exact Draft head/tree and checksum-verified OCR 1.11.5 -in isolated HOME, then runs production ocr review using configured LLM. Verify -MCP use, diagnostics/repair, grouping, findings/suggestions, privacy and cleanup. -Publish only bounded structural evidence, hashes and pass/fail. Owner confirmation -precedes the ordinary protected merge/release process. +The configured review is complete. Publish only bounded structural evidence, hashes +and pass/fail state in PR/issue receipts. After remediation, follow `docs/release.md` +for feature merge, development artifact verification, release PR, stable publication, +independent readback, issue/milestone closure and the separate no-release reconciliation. #### Handoff Notes diff --git a/changelog.d/176.bugfix.md b/changelog.d/176.bugfix.md index 98a9611..f68370d 100644 --- a/changelog.d/176.bugfix.md +++ b/changelog.d/176.bugfix.md @@ -1 +1,3 @@ Accept bounded private `failure_details.arguments` from OCR without treating otherwise valid failed-tool diagnostics as malformed. Arguments are omitted before normalized console output and publication DLP; they never enter finalized results, receipts, GitLab notes, or toolkit telemetry. Invalid diagnostics still preserve valid review findings and summary publication. Existing DLP checks on public findings, warnings, and suggestions and independent later-action restrictions remain in effect. + +Fail OCR qualification probes through the closed compatibility-status path when a review emits malformed JSON or invalid result, tool-call, failure-detail, or comment shapes. Private diagnostics remain available on stderr for CI troubleshooting without leaking into the public status artifact. diff --git a/scripts/ocr_compat.py b/scripts/ocr_compat.py index 563ed44..0129186 100644 --- a/scripts/ocr_compat.py +++ b/scripts/ocr_compat.py @@ -2169,27 +2169,40 @@ def _comment_arguments_probe(binary: Path, directory: Path) -> dict[str, object] cwd=repo, env=env, ) - payload = json.loads(raw) + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise CompatibilityError(f"comment arguments {mode}: review did not emit JSON") from exc + if not isinstance(payload, dict): + _fail(f"comment arguments {mode}: review result must be an object") comments = payload.get("comments") warnings = payload.get("warnings", []) - if not isinstance(comments, list) or not isinstance(warnings, list): + if ( + not isinstance(comments, list) + or not all(isinstance(comment, dict) for comment in comments) + or not isinstance(warnings, list) + ): _fail(f"comment arguments {mode}: invalid result shape") if mode == "rejected": calls = payload.get("tool_calls", {}) + if not isinstance(calls, dict): + _fail("comment arguments rejection emitted invalid tool-call evidence") telemetry = _tool_failure_telemetry(calls) details = calls.get("failure_details", []) if ( comments or not telemetry.valid or telemetry.failed != 1 + or not isinstance(details, list) or len(details) != 1 + or not isinstance(details[0], dict) or details[0].get("arguments") != _comment_probe_arguments(mode, "example.py") ): _fail("comment arguments rejection lost its closed failure evidence") else: expected = _comment_probe_records("example.py") actual = [{key: comment.get(key) for key in expected[0]} for comment in comments] - if sorted(actual, key=lambda item: item["content"]) != expected: + if sorted(actual, key=lambda item: str(item["content"])) != expected: _fail(f"comment arguments {mode}: changed or lost comment fields") if any(comment.get("start_line") != 2 for comment in comments): _fail(f"comment arguments {mode}: lost deterministic anchors") diff --git a/tests/test_ocr_compat.py b/tests/test_ocr_compat.py index 724f426..4c4422a 100644 --- a/tests/test_ocr_compat.py +++ b/tests/test_ocr_compat.py @@ -171,7 +171,22 @@ def test_manifest_rejects_assets_that_differ_from_evidence() -> None: @pytest.mark.parametrize( "corruption", - ["lost-comment", "changed-suggestion", "wrong-anchor", "missing-warning", "missing-arguments"], + [ + "lost-comment", + "changed-suggestion", + "wrong-anchor", + "missing-warning", + "missing-arguments", + "malformed-json", + "top-level-list", + "top-level-string", + "top-level-null", + "tool-calls-list", + "tool-calls-string", + "tool-calls-null", + "failure-details-object", + "detail-string", + ], ) def test_comment_probe_rejects_broken_boundary_observations( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, corruption: str @@ -190,6 +205,14 @@ def gateway(**kwargs: object) -> object: return contextlib.nullcontext("http://127.0.0.1:1/v1") def observation(*_args: object, **_kwargs: object) -> str: + invalid_top_levels = { + "malformed-json": "{", + "top-level-list": "[]", + "top-level-string": '"invalid"', + "top-level-null": "null", + } + if corruption in invalid_top_levels: + return invalid_top_levels[corruption] comments = [ {**item, "start_line": 2} for item in module._comment_probe_records("example.py") ] @@ -212,6 +235,17 @@ def observation(*_args: object, **_kwargs: object) -> str: "failure_by_tool": {"code_comment": 1}, "failure_details": [detail], } + invalid_calls: dict[str, object] = { + "tool-calls-list": [], + "tool-calls-string": "invalid", + "tool-calls-null": None, + } + if corruption in invalid_calls: + calls = invalid_calls[corruption] + elif corruption == "failure-details-object": + calls["failure_details"] = {"detail": detail} + elif corruption == "detail-string": + calls["failure_details"] = ["invalid"] elif corruption == "lost-comment": comments.pop() elif corruption == "changed-suggestion": @@ -228,6 +262,90 @@ def observation(*_args: object, **_kwargs: object) -> str: module._comment_arguments_probe(tmp_path / "ocr", tmp_path) +def test_qualify_cli_closes_malformed_comment_probe_as_contract_failure( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """Malformed OCR JSON reaches a safe failed status without an uncaught traceback.""" + + import contextlib + + module = load_script() + output = tmp_path / "evidence.json" + status_output = tmp_path / "status.json" + binary_digest = "0" * 64 + assets = [ + module.Asset( + name="opencodereview-linux-amd64", + size=1, + sha256=binary_digest, + url="https://github.com/alibaba/open-code-review/releases/download/v1.11.6/bin", + ), + module.Asset( + name="sha256sum.txt", + size=1, + sha256="1" * 64, + url="https://github.com/alibaba/open-code-review/releases/download/v1.11.6/sums", + ), + ] + + def download(asset: Any, directory: Path) -> Path: + path = directory / asset.name + path.write_bytes(b"x") + return path + + monkeypatch.setattr(module, "release_assets", lambda _release: assets) + monkeypatch.setattr(module, "_download", download) + monkeypatch.setattr( + module, + "parse_checksum_file", + lambda _path: {"opencodereview-linux-amd64": binary_digest}, + ) + monkeypatch.setattr(module, "_synthetic_repo", lambda root, _env: (root, "a" * 40, "b" * 40)) + monkeypatch.setattr( + module, + "_stub_gateway", + lambda **_kwargs: contextlib.nullcontext("http://127.0.0.1:1/v1"), + ) + monkeypatch.setattr(module, "_run", lambda *_args, **_kwargs: "{") + monkeypatch.setattr( + module, + "run_contracts", + lambda binary, _version, directory: module._comment_arguments_probe(binary, directory), + ) + + with ( + patched_env(RUNNER_OS="Linux"), + patched_attr(module, "_request_json", lambda _url: release("1.11.6")), + ): + result = module.main( + [ + "--manifest", + str(MANIFEST), + "qualify", + "--tag", + "v1.11.6", + "--comparison-version", + "1.11.5", + "--tested-baseline-version", + "1.11.5", + "--output", + str(output), + "--status-output", + str(status_output), + ] + ) + + captured = capsys.readouterr() + status = json.loads(status_output.read_text(encoding="utf-8")) + assert result == 1 + assert status["phase"] == "contracts" + assert status["reason"] == "contract-probe-failed" + assert status["result"] == "failed" + assert "comment arguments array: review did not emit JSON" in captured.err + assert "Traceback" not in captured.err + assert not output.exists() + + def test_discovery_filters_known_prerelease_and_old_versions() -> None: module = load_script() manifest = module.load_json(MANIFEST) From ecd7ecbf7f69bea32873a087218aad306368d1d3 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Mon, 7 Sep 2026 09:52:51 +0200 Subject: [PATCH 8/9] fix: reject malformed language preview paths --- PLANS.md | 8 +++++++- scripts/ocr_compat.py | 28 ++++++++++++++++++++-------- tests/test_ocr_compat.py | 16 ++++++++++++++++ 3 files changed, 43 insertions(+), 9 deletions(-) diff --git a/PLANS.md b/PLANS.md index 6bb5502..c982300 100644 --- a/PLANS.md +++ b/PLANS.md @@ -105,7 +105,7 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-05 | done | Public docs, decision flow, changelog and backlog reconciliation | | WQ-06 | in_progress | Final local gate, push, hosted checks and external Draft handoff | | WQ-07 | done | Version-neutral guidance and generic fixtures validated and self-reviewed; Draft push/readback remains owned by WQ-06 | -| WQ-08 | remediation_done | Malformed-output remediation and focused fuzz-style tests complete; holistic review and Codex Security diff scan remain | +| WQ-08 | remediation_done | Malformed-output and hostile language-preview remediation complete; final Codex Security remains in WQ-06 | | WQ-09 | pending | Push completed feature head, make PR ready, verify checks/threads, squash-merge and verify development publication | | WQ-10 | pending | Prepare/merge `Release v0.9.1`, verify stable delivery and close issues/milestone; then no-release external reconciliation and cleanup | @@ -196,6 +196,12 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. non-object detail entries through `CompatibilityError`. Sixteen focused cases pass, including the complete qualification CLI path to a private stderr diagnostic and closed `contracts/contract-probe-failed` status without traceback or evidence output. +- 2026-09-07: holistic repository-script MyPy and trust-flow review found that a + selected language-preview entry with a non-string `path` could reach mixed-type set + sorting and raise `TypeError`. Explicit selected-path validation now rejects hostile + `null`/numeric values through `CompatibilityError`. Seven focused preview tests, + repository-script MyPy, Ruff and diff checks pass; self-review found no remaining + uncontrolled path in this slice. #### Risks And Recovery diff --git a/scripts/ocr_compat.py b/scripts/ocr_compat.py index 0129186..00504f7 100644 --- a/scripts/ocr_compat.py +++ b/scripts/ocr_compat.py @@ -1914,6 +1914,25 @@ def _preview_file_selection(payload: object, path: str) -> tuple[bool, object]: return records[0].get("will_review") is True, records[0].get("exclude_reason") +def _selected_preview_paths(payload: object) -> set[str]: + """Return selected paths from a language preview after validating their shape.""" + + if not isinstance(payload, dict): + _fail("language preview must be a JSON object") + files = payload.get("files") + if not isinstance(files, list): + _fail("language preview emitted an invalid file manifest") + selected: set[str] = set() + for item in files: + if not isinstance(item, dict) or item.get("will_review") is not True: + continue + path = item.get("path") + if not isinstance(path, str): + _fail("language preview selected a file with an invalid path") + selected.add(path) + return selected + + def _target_rule_selection_probe(binary: Path, directory: Path) -> dict[str, object]: """Prove the real OCR selector consumes target rules without changing its range.""" @@ -2040,14 +2059,7 @@ def _language_rule_probe(binary: Path, directory: Path) -> dict[str, object]: payload = json.loads(preview) except json.JSONDecodeError as exc: raise CompatibilityError("language preview did not emit JSON") from exc - files = payload.get("files") if isinstance(payload, dict) else None - if not isinstance(files, list): - _fail("language preview emitted an invalid file manifest") - selected = { - item.get("path") - for item in files - if isinstance(item, dict) and item.get("will_review") is True - } + selected = _selected_preview_paths(payload) if selected != set(supported_paths): missing = sorted(set(supported_paths) - selected) unexpected = sorted(selected - set(supported_paths)) diff --git a/tests/test_ocr_compat.py b/tests/test_ocr_compat.py index 4c4422a..b17ed02 100644 --- a/tests/test_ocr_compat.py +++ b/tests/test_ocr_compat.py @@ -2061,3 +2061,19 @@ def test_preview_file_selection_accepts_json_and_rejects_legacy_text( module._preview_file_selection(payload, "fixture.unknown") else: assert module._preview_file_selection(payload, "fixture.unknown") == expected + + +@pytest.mark.parametrize("invalid_path", [None, 7]) +def test_selected_preview_paths_rejects_non_string_selected_path(invalid_path: object) -> None: + """Hostile preview paths fail closed instead of reaching mixed-type sorting.""" + + module = load_script() + payload = { + "files": [ + {"path": "valid.py", "will_review": True}, + {"path": invalid_path, "will_review": True}, + ] + } + + with pytest.raises(module.CompatibilityError, match="selected a file with an invalid path"): + module._selected_preview_paths(payload) From b47fa551b5de42dcfe9cd5b316a32f4184525be6 Mon Sep 17 00:00:00 2001 From: xeonvs <11463419+xeonvs@users.noreply.github.com> Date: Mon, 7 Sep 2026 09:59:34 +0200 Subject: [PATCH 9/9] docs: reconcile completed qualification gates --- PLANS.md | 24 ++++++++++++++++++------ compatibility/ocr-support.json | 2 +- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/PLANS.md b/PLANS.md index c982300..0b1f053 100644 --- a/PLANS.md +++ b/PLANS.md @@ -8,7 +8,7 @@ before handoff or commit. Completed stable plans are indexed in ### Toolkit 0.9.1 — OCR 1.11.4 and 1.11.5 -- **Status:** active; external model qualification complete, remediation in progress +- **Status:** active; qualification, remediation and security review complete; final push pending - **Plan Origin:** plan_mode_approved - **Release classification:** release-required; stable delivery authorized - **Target stable version:** 0.9.1 @@ -103,9 +103,9 @@ an extra diagnostic field. Workflow audit found canonical owners and valid index | WQ-04 | done | Expanded real OCR qualification, adjacent evidence and final pins | | WQ-04R | done | Forward-only live qualification, frozen historical readback and maintenance instructions | | WQ-05 | done | Public docs, decision flow, changelog and backlog reconciliation | -| WQ-06 | in_progress | Final local gate, push, hosted checks and external Draft handoff | +| WQ-06 | in_progress | Final reconciliation commit, exact-head push and hosted checks | | WQ-07 | done | Version-neutral guidance and generic fixtures validated and self-reviewed; Draft push/readback remains owned by WQ-06 | -| WQ-08 | remediation_done | Malformed-output and hostile language-preview remediation complete; final Codex Security remains in WQ-06 | +| WQ-08 | done | Malformed-output and hostile language-preview remediation, regressions and Codex Security complete | | WQ-09 | pending | Push completed feature head, make PR ready, verify checks/threads, squash-merge and verify development publication | | WQ-10 | pending | Prepare/merge `Release v0.9.1`, verify stable delivery and close issues/milestone; then no-release external reconciliation and cleanup | @@ -202,6 +202,17 @@ owns OS/Python matrix, package checks, dependencies, Security and CodeQL. `null`/numeric values through `CompatibilityError`. Seven focused preview tests, repository-script MyPy, Ruff and diff checks pass; self-review found no remaining uncontrolled path in this slice. +- 2026-09-07: Codex Security scan `5db2da0c-3f84-46a4-b4dd-cf3521fd1402` + completed the security-relevant feature range `ae0a9ac..57f5236` with 7/7 + surfaces closed and no reportable findings. Incremental scan + `90628f3c-10aa-4ce9-a3fa-b8c497e504d4` then covered the final + `57f5236..ecd7ecb` remediation commit, closed its sole runtime surface and found + no reportable issue. Both scans produced sealed reports and SARIF; the TAC status + was unavailable because its connector was not configured and did not gate review. +- 2026-09-07: final completed-head gate passed 1537 tests and 408 subtests at + 86.40% combined coverage; risk groups passed at 85/82/86/88%. Ruff, runtime + MyPy, Bandit, lock, manifest/evidence validation, rendered 0.9.1 Towncrier, + diff checks and pinned Gitleaks 8.24.3 passed. #### Risks And Recovery @@ -213,9 +224,10 @@ outputs until verification, then remove only task-owned files. #### Resume Point -WQ-08's remediation is focused-tested and ready for its signed logical commit. -Preserve the existing OCR evidence. Continue with one holistic branch review, full -quality, pinned Gitleaks and Codex Security before pushing the finished head. +Preserve the existing OCR evidence. Reconcile the current manifest and plan with the +completed model qualification and two complementary Codex Security scans, run the +final scoped/full local gates, commit the truthful state, then push only the finished +head and complete WQ-09/WQ-10 through protected PR and release workflows. #### Plan Fidelity Check diff --git a/compatibility/ocr-support.json b/compatibility/ocr-support.json index 5386221..b1a2abe 100644 --- a/compatibility/ocr-support.json +++ b/compatibility/ocr-support.json @@ -1606,7 +1606,7 @@ ], "evidence": "compatibility/evidence/ocr-1.11.5.json", "evidence_sha256": "18b56303554d3d8eee89f3f8db17c3d491f5f2bb990f5ffa412d64db685d1ada", - "human_conclusion": "Reviewed adjacent source changes and verified hosted assets plus expanded Darwin no-LLM contracts. Grouped-review renaming preserves consumed runtime behavior; grpc updates the binary dependency, while VS Code packages and browser-only viewer marks do not add toolkit surfaces. Exact 1.11.5 is the deployment target; configured LLM qualification remains pending.", + "human_conclusion": "Reviewed adjacent source changes and verified hosted assets plus expanded Darwin contracts. Grouped-review renaming preserves consumed runtime behavior; grpc updates the binary dependency, while VS Code packages and browser-only viewer marks do not add toolkit surfaces. A configured model-backed review completed all 9 selected items without GitLab posting; its malformed-output findings were remediated with fail-closed shape validation and focused hostile-input regressions. Exact 1.11.5 is the deployment target.", "published_at": "2026-09-05T04:23:21Z", "release_url": "https://github.com/alibaba/open-code-review/releases/tag/v1.11.5", "status": "tested",