Review agent: support repo-level review profiles to calibrate severity for experiment repos

[fullsend-ai-retro]

What happened

PR fullsend-ai/experiments#28 added a host-side API server PoC to the experiments repo. The review bot ran 3 passes (2026-05-20, 2026-05-29, 2026-06-01), each issuing CHANGES_REQUESTED for security findings including command injection and credential handling. The author posted a detailed response table accepting 4 findings as out-of-scope for experiment code and fixing 6 others. A human reviewer approved ("LGTM"). The bot's third pass still flagged the same accepted findings. The PR was merged over the bot's objection. This is expected behavior for an experiments repo — production security standards don't apply to throwaway PoC code — but the review agent has no mechanism to know this.

What could go better

The review agent applies the same severity thresholds regardless of repo purpose. In an experiments repo, critical/high security findings in PoC code are informational, not blocking. The author had to manually override the bot's verdict, which is friction that will recur on every experiment PR with security-adjacent code.

Existing issues address related but distinct problems: #1273 (PR description context), #1551 (human-authored PR severity), #1500/#1583 (re-flagging resolved findings). None address the repo-level dimension — that some repositories are inherently experimental and should have different default severity thresholds for all PRs.

Confidence: Medium-high. This PR is a clear example, but the experiments repo may be an outlier. The proposal is most valuable if other experiment/PoC repos exist or will exist.

Proposed change

Add support for a repo-level review profile configuration (e.g., in .github/fullsend.json or equivalent) that lets repo owners declare a review mode. For example:

{
  "review": {
    "profile": "experimental",
    "severity_floor": "info"
  }
}

When profile is experimental, the review agent should:

1. Downgrade all security findings to informational severity (still reported, not blocking)
2. Never issue CHANGES_REQUESTED — use COMMENT verdict instead
3. Include a note in the review summary that experimental profile is active

This affects the review agent definition (likely agents/review.md or the review skill) and the harness config schema. The review dispatch workflow would need to read the repo-level config and pass it as context to the agent.

Validation criteria

After implementation, the next 3 PRs to fullsend-ai/experiments (or any repo with experimental profile configured) should receive COMMENT verdicts instead of CHANGES_REQUESTED for security findings in PoC code. Security findings should still appear in the review summary (for awareness) but should not block merge or trigger fix agent dispatches.

---

Generated by retro agent from fullsend-ai/experiments#28

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature

Problem

The review agent applies uniform severity thresholds across all repos. In experiment/PoC repos, this causes CHANGES_REQUESTED verdicts on security findings that are intentionally out-of-scope, forcing authors to manually override the bot on every PR with security-adjacent code.

Root Cause

No repo-level configuration surface exists to modify the review agent's verdict behavior. Severity thresholds are fixed in the agent definition with no per-repo override.

Recommended Approach

Implement a repo-level config (e.g., .github/fullsend.json) with a review.profile field. When experimental, the review agent downgrades security findings to informational and uses COMMENT instead of CHANGES_REQUESTED. The dispatch workflow reads this config and passes it as agent context.

Related issues: #580 (tunable review thresholds) and #579 (general parameterization interface) cover the broader design space. This feature could be implemented standalone or as a first use case of the general parameterization system.

Proposed Test Case

Scenario: Experimental profile downgrades review verdict
  Given a repo with .github/fullsend.json containing {"review": {"profile": "experimental"}}
  When the review agent runs on a PR with critical security findings
  Then findings appear in the review summary as informational
  And the verdict is COMMENT (not CHANGES_REQUESTED)
  And the summary includes a note that experimental profile is active

Scenario: Default behavior unchanged without profile
  Given a repo without a fullsend config (or profile: "production")
  When the review agent runs on a PR with critical security findings
  Then findings retain their original severity
  And the verdict is CHANGES_REQUESTED as before

Edge case: Unrecognized profile value should fall back to default behavior

---

Labels: Review agent feature request for repo-level configuration affecting the harness config schema, medium priority given limited current scope (experiments repo).

[fullsend-ai-prioritize]

RICE Priority Score: 0.12

Score breakdown

Dimension	Score	Reasoning
Reach	0.5	Currently affects contributors to the fullsend-ai/experiments repo, which appears to be a single PoC/experiment repository. The issue itself notes 'the experiments repo may be an outlier,' and there is no evidence of multiple experiment repos or broad user demand. Score reflects a few users in one org.
Impact	0.5	The pain is real but low — a viable workaround exists (human reviewer approves and merges over the bot's objection). The friction is recurring but not blocking; authors can override the bot verdict on each PR. This is a convenience/ergonomics improvement rather than a workflow blocker.
Confidence	0.7	The issue is exceptionally well-documented with a concrete example PR, clear proposed solution, validation criteria, and thorough triage. However, uncertainty remains about how many repos and users would benefit, and whether the standalone implementation or the broader parameterization system (#579/#580) is the right approach.
Effort	1.5	Requires changes across multiple components: a new config schema (e.g., .github/fullsend.json), modifications to the review agent definition to respect profiles, and updates to the dispatch workflow to read and pass repo-level config. Some design decisions needed around the config surface and interaction with existing parameterization proposals (#579/#580).

Formula: (0.5 x 0.5 x 0.7) / 1.5 = 0.12