Judge stage hard-fails on the first row error for runs under 20 rows, inconsistent with the inference stage's tunable 10% ratio

[changliu2]

Summary

A small run (< 20 rows) dies on the first judge row failure, while the inference stage tolerates the same failure rate. Both stages document a "10% threshold", but only one of them implements it that way, and only one is tunable.

Reported by a user running 10 queries: "My runs with 10 queries fail immediately when I hit the first error as it breaks the 10% threshold."

The inconsistency

Inference — assert_ai/stages/inference.py:1288-1305:

error_fail_ratio = float(os.environ.get("ASSERT_INFERENCE_ERROR_FAIL_RATIO", "0.10"))
actual_ratio = len(errors) / len(pending_test_cases)
if actual_ratio > error_fail_ratio:
    raise errors[0]

Judge — assert_ai/stages/judge.py:459-475:

total_pending = len(pending)
tolerance = total_pending // 10 if total_pending >= 20 else 0
if 0 < len(errors) <= tolerance:
    ...  # tolerate
else:
    raise errors[0]

At n=10 with 1 failure:

Stage	Computation	Outcome
inference	0.10 > 0.10 → False	✅ tolerated
judge	1 <= 0 → False	❌ run dies

Same pipeline, same nominal policy, opposite results. Judge is also not tunable — it has no env var at all, so there is no escape hatch.

Why it bites specifically here

The comment above the judge gate explains the expected failure mode: adversarial-eval transcripts (XPIA payloads, PII leakage, security-attack scenarios) routinely trip the judge model's provider-side content filter — e.g. Azure's "potentially high-risk cyber activity" check — surfacing as LLMContentFilterError and counted as a soft per-row failure.

So the exact workload ASSERT exists to measure is the one that generates these failures, and on any run under 20 rows a single one is fatal. Small smoke runs are the common first-touch experience.

Three stale references in the judge comment

judge.py:446 says the gate "mirrors rollout.py's ASSERT_ROLLOUT_ERROR_FAIL_RATIO ceiling". All three parts are wrong:

1. There is no rollout.py — the file is inference.py.
2. There is no ASSERT_ROLLOUT_ERROR_FAIL_RATIO — the variable is ASSERT_INFERENCE_ERROR_FAIL_RATIO.
3. It does not mirror it — inference is ratio-based, strict->, and tunable; judge is floor-division with a hard 20-row cliff and no override.

(inference.py:1310 repeats the wrong variable name in its own comment.)

Suggested fix

1. Make the judge gate ratio-based and tunable, symmetric with inference: ASSERT_JUDGE_ERROR_FAIL_RATIO, default 0.10, strict >. This alone fixes the n=10 case.
2. If a floor is genuinely wanted for tiny runs, make it explicit and documented rather than an emergent property of //.
3. Name the cause in the error. The current message says the tolerance was exceeded but not why. When errors[0] is an LLMContentFilterError, say so and point at best practices §1 — using a less-restricted judge deployment is the actual fix, and users cannot infer that from the current text.
4. Fix the three stale references.

Workarounds today

• Point the judge at a less-restricted deployment, or set every content-filter category on the judge's deployment to the lowest blocking level. ASSERT must be able to judge sensitive content in order to measure it.
• Verify which deployment is actually judging. A related trap: examples/bank_manager_agent_control/agent.py:141 silently falls back to a hard-coded gpt-4o-mini when AGENT_MODEL is unset, so filters can be disabled on the models named in the YAML while a different, filtered deployment does the work.
• Run ≥ 20 rows so tolerance becomes non-zero.