You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
generic-http AI gate: schema-invalid output goes red, prose-factcheck misses PR text, wiring tests are per-mutation pins #260
PR #259 (#252) added a generic-http runner for the AI gates. It only runs once vars.AI_GATE_HTTP_BASE_URL is registered, so everything below is dormant today. The integration review and the test-quality audits of that PR left these gaps.
1. Schema-invalid model output turns an advisory check red
response_format: json_object (scripts/ai_gate_http_runner.py:73) guarantees JSON, not schema conformance. Valid JSON that fails the verdict schema, or names the wrong gate, makes scripts/ai_gate_verdict.py exit 2 (l.66-74). The 판정 반영 step in .github/workflows/_ai-gate.yml (~l.451) turns exit 2 into a red job regardless of blocking: false. With a non-compliant model, every PR would show four red "게이트가 고장났다" checks. docs/CI-GATES.md (l.60-65) lists the generic-http failures that fall to unavailable but not this one.
Proposal: map exit 2 to unavailable for the generic-http runner (or document it explicitly), with a test.
2. prose-factcheck on generic-http never sees the PR text
build_messages sends the prompt and the diff only (scripts/ai_gate_http_runner.py:69). .github/ai-gates/prose-factcheck.md:3-4 defines the gate's subject as the PR title/body and commit messages, so prose-factcheck-generic-http (.github/workflows/ai-gates-quality.yml:88) can only judge prose inside the diff. expect_nonempty does not catch this, because examined.files comes from the diff.
Proposal: write the PR title/body and commit messages to a second $RUNNER_TEMP file and pass it to the runner; or document the limitation.
3. Workflow wiring tests are per-mutation pins
Three test-quality audits in PR #259 each found new workflow mutations that changed CI behaviour while all tests stayed green (step order, shared-step if: lines, defaults injected by the tests themselves). Each was fixed with a verbatim pin. A structural test would end this whack-a-mole: build the step graph from _ai-gate.yml and assert data flow (every file a step reads is written by an earlier step in the same runner path; every shared step runs for both runners).
Also noted (low): an extra --timeout 1 in one test, the == true skip line not counted, a duplicate gate_name key in a fixture.
Done when
Items 1 and 2 decided and implemented or documented, each with a test.
The structural workflow test replaces or backs the verbatim pins.
Activity
Merge pull request #269 from choiyounggi/orch/open-issues-1026