Skip to content

generic-http AI gate: schema-invalid output goes red, prose-factcheck misses PR text, wiring tests are per-mutation pins #260

Closed

Description

@choiyounggi

Context

PR #259 (#252) added a generic-http runner for the AI gates. It only runs once vars.AI_GATE_HTTP_BASE_URL is registered, so everything below is dormant today. The integration review and the test-quality audits of that PR left these gaps.

1. Schema-invalid model output turns an advisory check red

response_format: json_object (scripts/ai_gate_http_runner.py:73) guarantees JSON, not schema conformance. Valid JSON that fails the verdict schema, or names the wrong gate, makes scripts/ai_gate_verdict.py exit 2 (l.66-74). The 판정 반영 step in .github/workflows/_ai-gate.yml (~l.451) turns exit 2 into a red job regardless of blocking: false. With a non-compliant model, every PR would show four red "게이트가 고장났다" checks. docs/CI-GATES.md (l.60-65) lists the generic-http failures that fall to unavailable but not this one.

Proposal: map exit 2 to unavailable for the generic-http runner (or document it explicitly), with a test.

2. prose-factcheck on generic-http never sees the PR text

build_messages sends the prompt and the diff only (scripts/ai_gate_http_runner.py:69). .github/ai-gates/prose-factcheck.md:3-4 defines the gate's subject as the PR title/body and commit messages, so prose-factcheck-generic-http (.github/workflows/ai-gates-quality.yml:88) can only judge prose inside the diff. expect_nonempty does not catch this, because examined.files comes from the diff.

Proposal: write the PR title/body and commit messages to a second $RUNNER_TEMP file and pass it to the runner; or document the limitation.

3. Workflow wiring tests are per-mutation pins

Three test-quality audits in PR #259 each found new workflow mutations that changed CI behaviour while all tests stayed green (step order, shared-step if: lines, defaults injected by the tests themselves). Each was fixed with a verbatim pin. A structural test would end this whack-a-mole: build the step graph from _ai-gate.yml and assert data flow (every file a step reads is written by an earlier step in the same runner path; every shared step runs for both runners).

Also noted (low): an extra --timeout 1 in one test, the == true skip line not counted, a duplicate gate_name key in a fixture.

Done when

  • Items 1 and 2 decided and implemented or documented, each with a test.
  • The structural workflow test replaces or backs the verbatim pins.

Activity

  1. added a commit that references this issue on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-debtTest gaps, duplication, drift risk

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions