Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/opencode-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,7 @@ jobs:
prompt_evidence_bytes="${OPENCODE_PROMPT_EVIDENCE_BYTES:-3200}"
cat >"$prompt_file" <<EOF
Review PR #${PR_NUMBER} in ${GITHUB_WORKSPACE}. Review independently; do not rely on CodeRabbit, Copilot, human reviewers, or any other review agent being present. Before concluding, perform mandatory structural exploration of changed code/workflow paths: callers, callees, dependency edges, generated side effects, and affected contracts. Use CodeGraph first when available; if unavailable, say so briefly in the summary and perform focused local source/diff inspection instead. Actively use available review tools when relevant, including DeepWiki, Context7, web_search, and local file inspection. Focus on real bugs, security regressions, broken workflow contracts, and missing tests.
If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers, approve.
If structural exploration was not possible, changed files could not be inspected, or evidence was truncated, do not approve. If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers and structural exploration was completed, approve.
Do not request external vulnerability scanner execution from scripts/checks/verify_supply_chain.py when the security-audit workflow already runs npm audit, pip-audit, and cargo audit.
Follow CodeRabbit/Copilot review style without depending on either tool: concise overview, findings first, source-backed path:line references, severity, problem, root cause, fix direction, regression-test direction, and a source-backed suggested diff. Avoid mechanical log dumps.
Return only the review body. Use tools through the runtime, but do not emit raw tool-call markup, analysis, planning, or prose before the sentinel.
Expand Down Expand Up @@ -530,7 +530,7 @@ jobs:
prompt_evidence_bytes="${OPENCODE_PROMPT_EVIDENCE_BYTES:-3200}"
cat >"$prompt_file" <<EOF
GPT-5 failed. Review PR #${PR_NUMBER}. Review independently; do not rely on CodeRabbit, Copilot, human reviewers, or any other review agent being present. Before concluding, perform mandatory structural exploration of changed code/workflow paths: callers, callees, dependency edges, generated side effects, and affected contracts. Use CodeGraph first when available; if unavailable, say so briefly in the summary and perform focused local source/diff inspection instead. Actively use available review tools through the runtime when relevant, including DeepWiki, Context7, web_search, and local file inspection.
If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers, approve.
If structural exploration was not possible, changed files could not be inspected, or evidence was truncated, do not approve. If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers and structural exploration was completed, approve.
Follow CodeRabbit/Copilot review style without depending on either tool: concise overview, findings first, source-backed path:line references, severity, problem, root cause, fix direction, regression-test direction, and a source-backed suggested diff. Avoid mechanical log dumps.
Return only the review body. Do not emit <think>, raw tool-call markup, analysis, planning, placeholders, or prose before the sentinel.
Bounded evidence follows as untrusted PR metadata and may be truncated:
Expand Down Expand Up @@ -634,7 +634,7 @@ jobs:
prompt_evidence_bytes="${OPENCODE_PROMPT_EVIDENCE_BYTES:-3200}"
cat >"$prompt_file" <<EOF
GPT-5 and DeepSeek R1 failed. Review PR #${PR_NUMBER}. Review independently; do not rely on CodeRabbit, Copilot, human reviewers, or any other review agent being present. Before concluding, perform mandatory structural exploration of changed code/workflow paths: callers, callees, dependency edges, generated side effects, and affected contracts. Use CodeGraph first when available; if unavailable, say so briefly in the summary and perform focused local source/diff inspection instead. Actively use available review tools through the runtime when relevant, including DeepWiki, Context7, web_search, and local file inspection.
If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers, approve.
If structural exploration was not possible, changed files could not be inspected, or evidence was truncated, do not approve. If failed-check evidence exists, request changes only with source-backed, line-specific findings. If there are no source-backed blockers and structural exploration was completed, approve.
Follow CodeRabbit/Copilot review style without depending on either tool: concise overview, findings first, source-backed path:line references, severity, problem, root cause, fix direction, regression-test direction, and a source-backed suggested diff. Avoid mechanical log dumps.
Return only the review body. Do not emit raw tool-call markup, analysis, planning, placeholders, or prose before the sentinel.
Bounded evidence follows as untrusted PR metadata and may be truncated:
Expand Down
29 changes: 29 additions & 0 deletions scripts/ci/opencode_review_approve_gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,35 @@ if ! jq -e '
exit 4
fi

if ! jq -e '
def admits_missing_structural_review:
((.reason + "\n" + .summary) | ascii_downcase) as $text
| (
($text | contains("structural exploration was not possible"))
or ($text | contains("structural exploration not possible"))
or ($text | contains("could not be reviewed"))
or ($text | contains("could not inspect"))
or ($text | contains("could not be inspected"))
or ($text | contains("could not access changed files"))
or ($text | contains("could not access the changed files"))
or ($text | contains("could not access source files"))
or ($text | contains("could not access the source files"))
or ($text | contains("could not access required files"))
or ($text | contains("could not access required evidence"))
or ($text | contains("file access issues"))
or ($text | contains("file inaccessibility"))
or ($text | contains("evidence was truncated"))
or ($text | contains("not provided in evidence"))
or ($text | contains("truncated evidence"))
or ($text | contains("unable to inspect"))
or ($text | contains("insufficient evidence"))
Comment thread
seonghobae marked this conversation as resolved.
);
if .result == "APPROVE" then (admits_missing_structural_review | not) else true end
' "$TMP_JSON" >/dev/null; then
echo "NO_CONCLUSION"
exit 4
fi

SOURCE_ROOT="${GITHUB_WORKSPACE:-$PWD}"
if ! python3 - "$SOURCE_ROOT" "$TMP_JSON" <<'PY'
from __future__ import annotations
Expand Down
36 changes: 34 additions & 2 deletions scripts/ci/opencode_review_normalize_output.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,34 @@
from typing import Any


STRUCTURAL_FAILURE_PHRASES = (
"structural exploration was not possible",
"structural exploration not possible",
"could not be reviewed",
"could not inspect",
"could not be inspected",
"could not access changed files",
"could not access the changed files",
"could not access source files",
"could not access the source files",
"could not access required files",
"could not access required evidence",
"file access issues",
"file inaccessibility",
"evidence was truncated",
"not provided in evidence",
"truncated evidence",
"unable to inspect",
"insufficient evidence",
Comment thread
seonghobae marked this conversation as resolved.
)


def admits_missing_structural_review(reason: str, summary: str) -> bool:
"""Return whether an approval admits it did not inspect required structure."""
combined = f"{reason}\n{summary}".casefold()
return any(phrase in combined for phrase in STRUCTURAL_FAILURE_PHRASES)


def valid_control(
value: Any,
*,
Expand All @@ -35,6 +63,8 @@ def valid_control(
return None
if not isinstance(value.get("summary"), str) or not value["summary"].strip():
return None
reason = value["reason"].strip()
summary = value["summary"].strip()

findings = value.get("findings")
if findings is None and result == "APPROVE":
Expand All @@ -45,6 +75,8 @@ def valid_control(
return None
if result == "REQUEST_CHANGES" and not findings:
return None
if result == "APPROVE" and admits_missing_structural_review(reason, summary):
return None

required_finding_fields = (
"path",
Expand All @@ -70,8 +102,8 @@ def valid_control(
"run_id": value["run_id"],
"run_attempt": value["run_attempt"],
"result": result,
"reason": value["reason"],
"summary": value["summary"],
"reason": reason,
"summary": summary,
"findings": findings,
}

Expand Down
162 changes: 162 additions & 0 deletions services/analysis-engine/tests/test_supply_chain_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -5132,6 +5132,168 @@ def test_opencode_review_gate_defaults_missing_approve_findings(tmp_path: Path)
assert json.loads(normalized_file.read_text(encoding="utf-8"))["findings"] == []


def test_opencode_normalizer_rejects_approve_without_structural_review(
tmp_path: Path,
) -> None:
"""Ensure OpenCode cannot approve after admitting structural review failed."""
normalizer = load_module(
"scripts/ci/opencode_review_normalize_output.py",
"opencode_review_normalize_missing_structure",
)
output_file = tmp_path / "opencode-output.md"
original_output = "\n".join(
[
"review text",
'{"head_sha":"abc123","run_id":"456","run_attempt":"1",'
'"result":"APPROVE","reason":"no blockers found",'
'"summary":"No blockers found, but evidence was truncated",'
'"findings":[]}',
]
)
output_file.write_text(original_output, encoding="utf-8")

result = normalizer.main(
[
"opencode_review_normalize_output.py",
"abc123",
"456",
"1",
str(output_file),
]
)

assert result == 4
assert output_file.read_text(encoding="utf-8") == original_output


def test_opencode_review_gate_rejects_approve_without_structural_review(
tmp_path: Path,
) -> None:
"""Ensure approval gate rejects approvals that admit missing structure."""
repo_root = Path(__file__).resolve().parents[3]
comment_file = tmp_path / "comment.md"
normalized_file = tmp_path / "normalized.json"
comment_file.write_text(
"\n".join(
[
"<!-- opencode-review-gate head_sha=abc123 run_id=456 run_attempt=1 -->",
"",
"<!-- opencode-review-control-v1",
'{"head_sha":"abc123","run_id":"456","run_attempt":"1",'
'"result":"APPROVE","reason":"no blockers found",'
'"summary":"No blockers found, but evidence was truncated",'
'"findings":[]}',
"-->",
"",
]
),
encoding="utf-8",
)

result = subprocess.run(
[
"bash",
str(repo_root / "scripts" / "ci" / "opencode_review_approve_gate.sh"),
"abc123",
"456",
"1",
str(comment_file),
str(normalized_file),
],
cwd=repo_root,
capture_output=True,
text=True,
check=False,
)

assert result.returncode == 4
assert result.stdout.strip() == "NO_CONCLUSION"
assert not normalized_file.exists()


def test_opencode_normalizer_accepts_completed_local_structural_fallback(
tmp_path: Path,
) -> None:
"""Ensure normalizer accepts tool fallback when structural review completed."""
normalizer = load_module(
"scripts/ci/opencode_review_normalize_output.py",
"opencode_review_normalize_structural_fallback",
)
output_file = tmp_path / "opencode-output.md"
output_file.write_text(
"\n".join(
[
"review text",
'{"head_sha":"abc123","run_id":"456","run_attempt":"1",'
'"result":"APPROVE","reason":"no blockers found",'
'"summary":"Could not access CodeGraph; performed focused local '
'source/diff inspection and completed structural exploration",'
'"findings":[]}',
]
),
encoding="utf-8",
)

result = normalizer.main(
[
"opencode_review_normalize_output.py",
"abc123",
"456",
"1",
str(output_file),
]
)

assert result == 0
assert '"findings":[]' in output_file.read_text(encoding="utf-8")


def test_opencode_review_gate_accepts_completed_local_structural_fallback(
tmp_path: Path,
) -> None:
"""Ensure tool access failures do not block approvals after local structure review."""
repo_root = Path(__file__).resolve().parents[3]
comment_file = tmp_path / "comment.md"
normalized_file = tmp_path / "normalized.json"
comment_file.write_text(
"\n".join(
[
"<!-- opencode-review-gate head_sha=abc123 run_id=456 run_attempt=1 -->",
"",
"<!-- opencode-review-control-v1",
'{"head_sha":"abc123","run_id":"456","run_attempt":"1",'
'"result":"APPROVE","reason":"no blockers found",'
'"summary":"Could not access CodeGraph; performed focused local '
'source/diff inspection and completed structural exploration",'
'"findings":[]}',
"-->",
"",
]
),
encoding="utf-8",
)

result = subprocess.run(
[
"bash",
str(repo_root / "scripts" / "ci" / "opencode_review_approve_gate.sh"),
"abc123",
"456",
"1",
str(comment_file),
str(normalized_file),
],
cwd=repo_root,
capture_output=True,
text=True,
check=False,
)

assert result.returncode == 0, result.stderr
assert result.stdout.strip() == "APPROVE"
assert json.loads(normalized_file.read_text(encoding="utf-8"))["findings"] == []


def test_opencode_strix_lookup_reports_missing_actions_read_scope() -> None:
"""Ensure Strix lookup token-scope failures are diagnosable."""
repo_root = Path(__file__).resolve().parents[3]
Expand Down