Add science guidance: interpreting safety-violation rates when adversarial probing hits SUT model alignment or endpoint safety guardrails

[changliu2]

Problem

When Adaptive Eval runs adversarial probing against a System-Under-Test (SUT) that has strong built-in safety alignment (e.g. frontier-aligned models like GPT-5.5, Claude 4.6+) and/or sits behind a provider-side content-safety layer (Azure AI Content Safety, OpenAI moderation, Anthropic constitutional layers, etc.), the results need careful interpretation. Today there is no science guidance in the docs that tells the user how to read these results, and that gap is starting to bite.

The same dynamic applies in two directions:

1. Tester / auditor model alignment dampens the attack. If the auditor model is aligned (GPT-5.5, Claude 4.6, etc.), it will partially refuse to generate strong adversarial inputs. The probe lands softer than the test case definition implies. Reported "violation rate" is depressed.
2. SUT model alignment + endpoint guardrails dampen the response. Even if the probe is strong, an aligned SUT (or a content-filtered endpoint) often refuses or sanitizes its response. The judge sees a refusal, not a violation. Reported violation rate looks great — but it's a refusal floor, not a safety guarantee.
3. Aligned judge model over-flags refusals or near-refusals. An aligned judge model (same families) tends to score conservatively on borderline transcripts. Combined with (2), the user gets a transcript pile that's mostly aligned-refusal-aligned, and overrefusal rates spike.

The combined effect: on adversarially strong configurations, the reported policy_violation rate is biased low and overrefusal rate is biased high, in a way that does not reflect a non-aligned SUT's behavior under the same attack. A user comparing models or shipping a release-gate decision off these numbers can draw the wrong conclusion in either direction.

Why this matters now

• Customers running Adaptive Eval on production agents (real-world example: hitting Azure GPT-5.4 medium-filter content-rejection on benign-looking payloads — separate issue) are already seeing the symptom.
• The MLCommons / AILuminate-style external benchmarking community has partially addressed this with rail-free auditor models + careful judge calibration. We currently document neither.
• The OSS launch (release-gate sequencing) will surface this immediately because the default judge in every example config is azure/gpt-5.4 or gpt-5.4-mini — both aligned.

What we don't have today

• A docs/science/adversarial-interpretation.md (or similar) page that names the three failure modes above.
• Best-practice configuration guidance — when to use which judge model, when to layer overrefusal as a counter-metric, when to trust low policy_violation numbers and when not to.
• A "results health check" the user can run on a completed run to detect the symptom (e.g. high overrefusal rate + high judge-failure rate + low policy_violation rate is the classic signature of aligned-auditor + aligned-SUT + aligned-judge).
• Citation / reference to the science background (alignment dampening, defect-rate inflation/deflation under aligned probes) — we have internal knowledge of this from the Bloom vs P2M comparison work but no customer-facing write-up.

Proposed scope

1. Write docs/science/adversarial-interpretation.md. Audience: a developer running their first adversarial eval. Sections:
   • The three failure modes above, named and explained in one paragraph each.
   • The "results signature" table — combinations of (auditor model, SUT model, judge model, content filter on/off) and what each cell's policy_violation / overrefusal / judge-failure pattern actually means.
   • The "grain of salt" rule: when adversarial probing is strong AND aligned models are in any of the three roles, treat single-run violation rates as a lower bound and overrefusal rates as an upper bound.
   • Pointer to the paired-statistical-comparison story for when you need a defensible delta (regression testing).
2. Update docs/writing-eval-specs.md with a one-paragraph warning + link to (1).
3. Update the README "Important: Risks and limitations" section (PR docs: add CELA-reviewed risks and limitations section to README #55) with a one-bullet note that adversarial results need interpretation and point to (1).
4. Add a CLI / viewer surface that detects the signature and surfaces a one-line warning at run completion: "this run looks like it hit alignment dampening — see ." (Lower priority — can be a follow-up issue once the docs land.)

Acceptance

• A user running an adversarial eval against GPT-5.5 (or Claude 4.6) with a GPT-5.5 / Claude 4.6 judge can read one doc page and understand whether their policy_violation: 2% number means "the agent is safe" or "the auditor / SUT / judge stack canceled out and the number means nothing."
• The README explicitly warns about this in the risks section.
• A follow-up issue exists for the auto-detect "results signature" warning.

Out of scope

• Building / shipping the rail-free auditor model. That's the premium service on the roadmap. This issue is about interpretation of results from today's aligned-model stack.
• Statistical-validity tooling (paired t-test, ground-truth pairing). Covered separately in the Tier 4 regression runner work.

[changliu2]

Concrete instance: refusal/error transcripts silently score as "passed"

Adding to this issue because it's a sharp, fixable code-level instance of the failure-mode-2 dynamic described above ("SUT model alignment + endpoint guardrails dampen the response... judge sees a refusal, not a violation").

Symptom

When the rollout truncates a conversation because of a content-filter rejection (target-side or auditor-side) or a target error, the seed is still scored as passing every safety dimension.

Root cause

The judge stage never inspects the rollout's stop_reason before scoring. Every transcript on disk gets judged, including ones the rollout intentionally truncated because the conversation never really happened. The judge sees a refusal/empty/short transcript, finds no policy_violation/pii_leak/etc. evidence, returns all dimensions = false, the row writes judge_status="ok" with verdict.dimensions.<each>=false, and the analysis aggregator counts it as a successful pass against every dimension.

Code path:

• p2m/stages/rollout.py sets transcript.stop_reason to one of: target_input_refused (line 838), auditor_input_refused (line 727), invalid_auditor_turn (line 745), target_error (line 853), or max_turns (line 942).
• p2m/stages/judge.py and p2m/core/judge.py → stop_reason is never read in either file. Verified by git grep stop_reason p2m/stages/judge.py p2m/core/judge.py returning no matches.
• p2m/analysis/suite_analysis.py:90 filters by judge_status == "ok" only; refusal-truncated rows pass that filter.
• p2m/analysis/suite_analysis.py:107-110 includes those rows' False dimension values in both numerator and denominator.

Severity by stop_reason

stop_reason	What actually happened	Currently scored as	Correct treatment
auditor_input_refused	Auditor's content filter blocked generating the adversarial prompt. Seed never reached the target.	All dims = false → "passed" against the target	Inconclusive — exclude from both numerator and denominator. Can't make claims about a target that was never asked.
invalid_auditor_turn	Auditor failed to produce a valid turn after retries.	Same	Same as above.
target_error	Target crashed mid-conversation.	Same	Failure of a different kind — surface as target_error_rate, not a pass.
target_input_refused	Target's content filter rejected the prompt. Real target behavior, but specific to the alignment/guardrails.	All dims = false → "passed"	Dimension-aware: pass for policy_violation, fail for overrefusal (which is the entire point of having an overrefusal dimension).
max_turns	Conversation ran out of turns.	All dims = false → "passed"	Inconclusive — flag, don't count toward pass rate.

The target_input_refused row in this table is exactly this issue's "refusal floor" symptom showing up at the per-seed level instead of just at the aggregate level.

Suggested fix shape

1. p2m/core/judge.py: infer_judge_status — accept and consider the transcript's stop_reason. Introduce new statuses: "inconclusive" (seed never executed against target) and "target_error" (separate from judge_failed; the target broke, not the judge).
2. p2m/stages/judge.py — for rows where stop_reason ∈ {auditor_input_refused, invalid_auditor_turn, max_turns}, short-circuit: write a synthetic score row with judge_status="inconclusive" and verdict={} and skip the judge LLM call (wastes tokens to judge an empty transcript).
3. p2m/analysis/suite_analysis.py — per-dimension policy:
   • inconclusive → exclude from numerator AND denominator. Surface as inconclusive_count in the run-level metrics summary alongside judge_failures (already there).
   • target_error → exclude from violation dimensions; surface as target_error_rate.
   • target_input_refused with judge_status="ok" → keep current behavior for policy_violation etc. (refusing IS pass for those), but for overrefusal the judge should be forced to evaluate refusal-ness rather than silently scoring False. This is more nuanced — may need a separate "refusal detector" pre-pass.
4. Viewer (viewer/src/lib/judgment.ts, ResultDrawer.svelte) — render inconclusive and target_error rows with a neutral badge (gray/yellow), not green ✓. Today they show all-green dimensions because the verdict.dimensions map has all-false values.
5. Headline metrics (CSV export, p2m results status output, viewer suite summary) — report n_executed / n_attempted so the user can immediately see "10 of 200 seeds never reached the target." This is the user-visible signal that ties back into the "results health check" idea above.

Why this fits this issue

This issue frames the problem at the science / interpretation layer ("teach the user the refusal floor exists"). The fix above sits at the measurement plumbing layer ("stop silently inflating the pass rate with refusals and errors"). Both are needed: the doc + signature-detector tells the user what the number means; this fix makes the underlying number actually correspond to executed test cases.

Filing as bug-side of this issue rather than a separate one. Happy to split if the science-doc work and the code-fix work need to ship on different release trains.

Tests to add

• tests/test_suite_analysis_refusal_handling.py — given 100 score rows where 30 have stop_reason="auditor_input_refused", assert denominator is 70 not 100.
• tests/test_judge_stage_skips_inconclusive.py — given a transcript with stop_reason="auditor_input_refused", assert score_row → judge_status="inconclusive", no judge LLM call made.
• tests/test_judge_stage.py (existing) — extend with a target_error transcript fixture.

Reproducible against any adversarial scenario config (e.g. examples/incident_triage_agent/eval_config_baseline.yaml) when the auditor or target has a provider-side content filter that rejects adversarial inputs.

[mgmarcusa]

I’d be interested in helping with the documentation work here. I agree that a low violation rate can be misleading when the auditor, target, or judge is affected by safety filters or refusal behavior.

I think the guidance should also distinguish agreement between model judges from calibration against human judgment. Multiple judges can agree while sharing the same blind spot. Before treating an automated judge as reliable, users need some way to check whether it identifies the same problems that informed human reviewers identify.

The most practical workflow seems to be:

1. Select a representative sample of completed ASSERT cases.
2. Have informed reviewers score the same dimensions as the automated judge.
3. Examine false positives and false negatives—not just an overall agreement rate.
4. Revise the rubric, examples, or judge prompt based on those disagreements.
5. Continue spot-checking as the models, prompts, and guardrails change.

Human-grounded cases could come from a purpose-built review exercise, but they could also come from expert corrections already produced during normal use. For example, if a support specialist explains why a system’s replacement-eligibility recommendation was wrong, that correction can become a test case for determining whether the automated judge recognizes the same failure.

Looking briefly at ASSERT’s architecture, this could fit without turning ASSERT into a full annotation platform. Human labels could live in a separate artifact keyed by test_case_id and dimension. ASSERT could then compare them with scores.jsonl and produce a disagreement queue for review in the existing viewer.

For a first contribution, I’d keep this documentation-focused: explain why multi-judge agreement is not human calibration, describe a lightweight calibration loop, and clarify which claims should not be made when no human review has occurred.

Would that be a useful addition to the scope of #61?