Skip to content

SFF-066: Shared-model bias warning for evaluator phases #79

@jmrpineda

Description

@jmrpineda

Reported by: jmrpineda
Requested by: repository maintainer
Priority: P2
Affected surfaces: VS Code execution settings panel, phase routing guidance
Constraints: Warning only; it must not block saving when the configured phase routing is otherwise valid.

Summary

Add a non-blocking warning in execution settings when the configured evaluator phase resolves to the same underlying model as its paired generator phase, so developers are reminded about shared-bias risk before saving phase routing.

Problem / opportunity

The orchestrated workflow allows per-phase model selection, which is one of its strongest practices for separating generation from evaluation. Today a developer can configure different agent or profile names that still resolve to the same underlying model, especially across spec and refinement or implementation and review, without receiving any guidance about the risk of self-confirming output.

Requested behavior

When Refinement and Spec, or Review and Implementation, resolve to the same underlying provider/model combination, the execution settings panel should surface a visible ⚠️ affordance on the evaluator phase. Clicking it should explain the shared-bias risk and recommend using a different evaluator model or provider family when possible. The warning must be advisory only and must not disable save by itself.

Scope

  • In scope: Detect shared underlying model identity for evaluator/generator phase pairs, surface a clickable warning in the execution settings UI, and cover detection with automated tests.
  • Out of scope: Blocking configuration, redefining repository-access rules, or redesigning the entire execution settings panel.

Acceptance criteria

  1. The execution settings panel detects when refinement/spec or review/implementation resolve to the same underlying model even if they use different profile names.
  2. The evaluator phase shows a clickable ⚠️ button that reveals a recommendation about reducing shared-bias risk.
  3. The warning remains non-blocking and does not disable saving when no other validation errors exist.

Notes

  • This feature should coexist with the current explicit validation rules for invalid phase routing and permission mismatches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions