Skip to content

fix(ship-check): require exact model ids in attribution #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aliasunder merged 1 commit into from
Sep 17, 2026
Merged

fix(ship-check): require exact model ids in attribution #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions plugins/ship-check/agents/bug-checker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,14 @@ skipping decision points.

Every PR comment or review you post — inline comments, review bodies, PR-level
comments — MUST include the footer: `\n\n---\n*🔍 ship-check · bug-check · <model-id>*`
where `<model-id>` is your own model ID from your system prompt. No exceptions — a
comment without a footer misattributes automated output to the repo owner.
where `<model-id>` is your verified exact runtime model ID, including version and
variant suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess
it. In Codex, when visible system prose is generic, use a runtime-provided current
thread or session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting. No exceptions — a comment without a footer misattributes
automated output to the repo owner.
Comment on lines +145 to +146

@umm-actually umm-actually Bot Sep 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Report attribution blockers instead of stopping silently
Medium severity · correctness · medium confidence

The new stop-condition halts phase agents before posting when the exact runtime model ID cannot be verified, yet the comment-mode summary templates only support a 0-findings reason for not posting, and the four agent files and four phase skills — unlike pr-monitor and the ship-check orchestrator — omit any instruction to report the blocker. A phase that stops for attribution reasons therefore reports indistinguishably from a clean 0-findings run, letting the orchestrator advance the pipeline and potentially declare the PR merge-ready without that phase's review.

Failure scenario: A Codex run where the runtime does not expose the current session ID, or where it cannot be matched to session_meta.payload.id. The phase agent stops before posting per the new rule and returns its summary with the only not-posted state the template provides (0 findings), so the orchestrator records the phase as clean, dispatches the next phase, and may report the PR merge-ready without this phase's review having landed.

Suggested fix 
Extend the stop-condition in the four agent files and four phase skills to "stop before posting and report the attribution blocker to the orchestrator" (matching pr-monitor/SKILL.md and ship-check/SKILL.md), and add a blocked state to the comment-mode summary templates (e.g. "Review posted: no (attribution blocker)") so a stopped phase is distinguishable from one with no findings.

umm-actually · deepseek/deepseek-v4-flash-0731


### Comment mode

Expand Down
10 changes: 8 additions & 2 deletions plugins/ship-check/agents/code-quality-reviewer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,14 @@ You loaded `sequentialthinking` in orientation. Call it at these decision points

Every PR comment or review you post — inline comments, review bodies, PR-level
comments — MUST include the footer: `\n\n---\n*🔍 ship-check · code-quality · <model-id>*`
where `<model-id>` is your own model ID from your system prompt. No exceptions — a
comment without a footer misattributes automated output to the repo owner.
where `<model-id>` is your verified exact runtime model ID, including version and
variant suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess
it. In Codex, when visible system prose is generic, use a runtime-provided current
thread or session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting. No exceptions — a comment without a footer misattributes
automated output to the repo owner.

### Comment mode

Expand Down
10 changes: 8 additions & 2 deletions plugins/ship-check/agents/pr-reviewer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,14 @@ you're shortcutting the review.

Every PR comment or review you post — inline comments, review bodies, PR-level
comments — MUST include the footer: `\n\n---\n*🔍 ship-check · pr-review · <model-id>*`
where `<model-id>` is your own model ID from your system prompt. No exceptions — a
comment without a footer misattributes automated output to the repo owner.
where `<model-id>` is your verified exact runtime model ID, including version and
variant suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess
it. In Codex, when visible system prose is generic, use a runtime-provided current
thread or session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting. No exceptions — a comment without a footer misattributes
automated output to the repo owner.

### Comment mode

Expand Down
10 changes: 8 additions & 2 deletions plugins/ship-check/agents/test-auditor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,14 @@ You loaded `sequentialthinking` in orientation. Call it at these decision points

Every PR comment or review you post — inline comments, review bodies, PR-level
comments — MUST include the footer: `\n\n---\n*🔍 ship-check · test-audit · <model-id>*`
where `<model-id>` is your own model ID from your system prompt. No exceptions — a
comment without a footer misattributes automated output to the repo owner.
where `<model-id>` is your verified exact runtime model ID, including version and
variant suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess
it. In Codex, when visible system prose is generic, use a runtime-provided current
thread or session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting. No exceptions — a comment without a footer misattributes
automated output to the repo owner.

### Comment mode

Expand Down
8 changes: 7 additions & 1 deletion plugins/ship-check/skills/bug-check/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -538,7 +538,13 @@ REVIEW
```

Replace `OWNER_REPO` and `PR_NUMBER` with values from the dispatch prompt. Replace
`MODEL_ID` with your own model ID (from your system prompt).
`MODEL_ID` with your verified exact runtime model ID, including version and variant
suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess it. In
Codex, when visible system prose is generic, use a runtime-provided current thread or
session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting.

5. **If 0 findings and no dismissals**, skip the API call — report "0 findings"
to the orchestrator only. With 0 findings but cleared suspicions, post a
Expand Down
8 changes: 7 additions & 1 deletion plugins/ship-check/skills/code-quality/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,13 @@ REVIEW
```

Replace `OWNER_REPO` and `PR_NUMBER` with values from the dispatch prompt. Replace
`MODEL_ID` with your own model ID (from your system prompt).
`MODEL_ID` with your verified exact runtime model ID, including version and variant
suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess it. In
Codex, when visible system prose is generic, use a runtime-provided current thread or
session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting.

4. **If 0 findings and no dismissals**, skip the API call — report "0 findings"
to the orchestrator only. With 0 findings but cleared suspicions, post a
Expand Down
13 changes: 11 additions & 2 deletions plugins/ship-check/skills/pr-monitor/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,8 +235,17 @@ For each unresolved bot thread, do ALL of these in order:
4. **Reply to the comment** -- do this BEFORE resolving, for EVERY bot comment.

Since `gh` posts as the user's account, every reply MUST include an attribution
footer. Read your model ID from your system context ("You are powered by the model
named..."). The footer format is: `\n\n---\n*🔍 ship-check · pr-monitor · <model-id>*`
footer. Use the exact runtime model ID, including its version and variant suffixes
(for example, `gpt-5.6-sol`). Never shorten it to a family label such as `gpt-5`,
translate it to a display name, or guess. Read it from runtime system context. In
Codex, when the visible system prose is generic, first obtain the current thread or
session ID from runtime-provided task metadata, then match it exactly to
`session_meta.payload.id` in the rollout metadata and read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout
by recency, cwd, or display name. If the runtime does not expose the current ID or
no exact model ID can be verified, stop before posting and report the attribution
blocker. The footer format is:
`\n\n---\n*🔍 ship-check · pr-monitor · <model-id>*`

```
gh api graphql -f query='mutation {
Expand Down
8 changes: 7 additions & 1 deletion plugins/ship-check/skills/pr-review/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,13 @@ REVIEW
```

Replace `OWNER_REPO` and `PR_NUMBER` with values from the dispatch prompt. Replace
`MODEL_ID` with your own model ID (from your system prompt).
`MODEL_ID` with your verified exact runtime model ID, including version and variant
suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess it. In
Codex, when visible system prose is generic, use a runtime-provided current thread or
session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting.

5. **If 0 findings and no dismissals**, skip the API call — report "0 findings"
to the orchestrator only. With 0 findings but cleared suspicions, post a
Expand Down
29 changes: 20 additions & 9 deletions plugins/ship-check/skills/ship-check/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,20 @@ orchestrator triage, pr-monitor replies).

- `<component>` is the phase or role: `pr-review`, `code-quality`, `test-audit`,
`bug-check`, `pr-monitor`, or `triage` (for orchestrator inter-phase triage posts).
- `<model-id>` is the poster's own model ID, read from the system prompt ("You are
powered by the model named..."). Agents self-identify — the orchestrator does not
look up or pass model IDs for them.
- `<model-id>` is the poster's exact runtime model ID, including version and variant
suffixes (for example, `gpt-5.6-sol`). Never shorten it to a family label such as
`gpt-5`, translate it to a display name, or guess. Read it from runtime system
context. In Codex, when visible system prose is generic, first obtain the current
thread or session ID from runtime-provided task metadata, match it exactly to
`session_meta.payload.id` in the rollout metadata, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the runtime does not expose the current ID or no
exact model ID can be verified, stop before posting and report the attribution
blocker. Agents self-identify — the orchestrator does not look up or pass model IDs
for them.
- **The orchestrator's own PR-level comments** (non-inline findings, deferred items
posted via `gh pr comment`) use the orchestrator's own model ID with component
`triage` or `ship-check`.
posted via `gh pr comment`) use the orchestrator's own verified exact runtime model
ID with component `triage` or `ship-check`.

A comment posted without a footer is indistinguishable from the repo owner's manual
comments and misattributes automated output to a human. Every `gh api` and
Expand All @@ -70,7 +78,8 @@ commit):
When committing, add this trailer to every commit message (after the body, before
any trailers the harness adds):
Ship-Check: PHASE_NAME · YOUR_MODEL_ID
Read YOUR_MODEL_ID from your system prompt ("You are powered by the model named...").
Use your verified exact runtime model ID for YOUR_MODEL_ID. Preserve version and
variant suffixes; never abbreviate, translate, or guess it.
Comment on lines +81 to +82

@umm-actually umm-actually Bot Sep 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define unverifiable-ID behavior for commit trailers
Low severity · correctness · medium confidence

The commit-trailer instructions now require a verified exact runtime model ID but define no behavior when verification fails, while the footer rules in the same file do (stop before posting, report the blocker). A phase agent that cannot verify its ID must choose between committing with a guessed or family-level trailer — explicitly forbidden — or silently dropping the trailer, both of which violate the PR's stated requirement that every ship-check commit trailer carry the exact ID.

Failure scenario: A phase agent in default mode fixes bugs on a Codex run but cannot verify its exact runtime model ID. With no instruction covering the commit path, it either commits with a guessed family-level ID in the Ship-Check trailer (forbidden by the new rule) or omits the trailer, leaving the commit misattributed either way.

Suggested fix 
Mirror the footer rule: state that when the exact runtime model ID cannot be verified the agent must stop before committing and report the attribution blocker to the orchestrator, rather than committing with an unverified or omitted trailer.

umm-actually · deepseek/deepseek-v4-flash-0731

```

The harness appends its own `Co-Authored-By` trailer automatically; the `Ship-Check`
Expand Down Expand Up @@ -122,7 +131,8 @@ mode" section in your preloaded skill for the gh api template. Only post a revie
you have findings — skip the API call for 0 findings.
Repo: OWNER_REPO
Append a footer to the review body AND every inline comment body using your own
model ID (from your system prompt):
exact runtime model ID, including version and variant suffixes. Never abbreviate,
translate, or guess it:
\n\n---\n*🔍 ship-check · PHASE_NAME · YOUR_MODEL_ID*
```

Expand Down Expand Up @@ -170,7 +180,7 @@ gh repo view --json nameWithOwner -q .nameWithOwner
```

Pass it as `Repo: owner/repo` in the dispatch prompt. In comment mode, agents
include their own model ID in the footer (they know it from their system prompt) —
include their own verified exact runtime model ID in the footer —
the orchestrator does not need to look it up or pass it.

## Local review mode (`--local`)
Expand Down Expand Up @@ -358,7 +368,8 @@ Append it to every phase that commits (phases 1, 3-5):
```
When committing, add this trailer to every commit message:
Ship-Check: PHASE_NAME · YOUR_MODEL_ID
Read YOUR_MODEL_ID from your system prompt ("You are powered by the model named...").
Use your verified exact runtime model ID for YOUR_MODEL_ID. Preserve version and
variant suffixes; never abbreviate, translate, or guess it.
```

### Phase 1: PR Review
Expand Down
8 changes: 7 additions & 1 deletion plugins/ship-check/skills/test-audit/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,13 @@ REVIEW
```

Replace `OWNER_REPO` and `PR_NUMBER` with values from the dispatch prompt. Replace
`MODEL_ID` with your own model ID (from your system prompt).
`MODEL_ID` with your verified exact runtime model ID, including version and variant
suffixes (for example, `gpt-5.6-sol`). Never abbreviate, translate, or guess it. In
Codex, when visible system prose is generic, use a runtime-provided current thread or
session ID to match `session_meta.payload.id` exactly, then read
`session_meta.payload.base_instructions.provenance.model`. Never select a rollout by
recency, cwd, or display name. If the current ID or exact model ID cannot be verified,
stop before posting.

4. **If 0 findings and no dismissals**, skip the API call — report "0 findings"
to the orchestrator only. With 0 findings but cleared suspicions, post a
Expand Down