Wait for the requested scorecard calculation before finishing an eval

[CTWalk]

Thanks for the eval runner in #333 — it was a pleasure to read, and this is a
small correctness fix in that new code rather than a complaint about it. Happy
to close it if you already have this in flight.

Summary

• Keep polling when the scorecard endpoint still reports a different calculation
  ID, even when that previous calculation is already terminal.
• Extend the existing polling test with an old completed calculation ahead of
  the requested run.

Problem

recalculate_and_wait_scorecard() starts a calculation, receives its ID, then
polls the sheet scorecard endpoint. _scorecard_is_done() currently ignores a
mismatched ID only while the response is non-terminal:

if (
    latest_id is not None
    and str(latest_id) != calculation_id
    and not _is_terminal_scorecard_response(payload)
):
    return False
return _is_terminal_scorecard_response(payload)

A previous calculation is normally terminal, so if the first read still exposes
it, the terminal exception accepts it immediately — after a single poll. The
comment above the guard describes the intent ("ignore stale payloads from a
previous calculation until the new one appears"), but the terminal clause
defeats it for the most ordinary stale payload: a completed previous run.

Both recalculate_and_wait_scorecard() and
arecalculate_and_wait_scorecard() route through this predicate.

Why it is worth fixing

The mismatched ID is not only compared, it is reused. After polling,
extract_scorecard_scores() reads latest_calculation.id back out of the
payload and sends it to the row endpoint as calculation_id
(evaluations/scorecard.py L290-291 and L322-323):

calculation_id = str(latest["id"]) if latest.get("id") is not None else None
params = {"calculation_id": calculation_id} if calculation_id else None

So a stale-accepted poll yields an aggregate score and row-level scorer outputs
that agree with each other and both belong to the previous calculation. The
result is a plausible wrong report rather than a visibly broken one: a previous
green run can stand in for a new regression, or a previous red run can fail a
new healthy run.

That reuse is also why I think the ID comparison is meant to be authoritative
rather than advisory — the SDK already treats the recalculate ID and
latest_calculation.id as the same namespace.

Behavior after this change

A response carrying another calculation ID is treated as stale regardless of
status. Polling completes only once the requested calculation is visible and
terminal.

The guard stays conditional on the API returning a latest calculation ID, so the
existing fallback for payloads without one is unchanged.

Reproduce

The updated regression test supplies three responses:

1. calc-old, completed;
2. requested calc-1, running; and
3. requested calc-1, completed.

Before the change the poller returns the first response after one read. After
it, the poller makes all three reads and returns the requested completed
calculation. Reverting only the scorecard.py change makes the test fail on
calc-old != calc-1, so it pins the behavior rather than restating it.

Verification

Run in the environment integration-tests.yml declares — Python 3.11 (3.11.14),
Poetry 2.2.1, poetry install from the committed lock:

• pytest -q — 357 passed, with the 13 pre-existing Pydantic deprecation
  warnings
• ruff check and ruff check --select I on both changed files — clean
• ruff format --check on both changed files — already formatted
  (ruff 0.9.4, the .pre-commit-config.yaml pin)
• git diff --check

No API key, model request, or token spend is involved in the test path.

One question for you

If the backend can legitimately report a different latest calculation while
the requested one is still queued, this change converts today's silent wrong
result into the existing Timed out waiting for scorecard calculation to finish. error. I think failing loudly is the right trade for evaluation
results, but you know the backend semantics and I don't — if you would rather
bound the wait (accept a mismatched terminal payload only after N polls, or key
completion off a different authoritative field), tell me which and I will
rework it.

Cross-SDK note

prompt-layer-js has the same terminal-ID exception in
src/evaluations/scorecard.ts, and reuses the polled latest.id as the row
calculation_id in the same way. I kept this PR scoped to the Python SDK and
did not run the JavaScript suite; glad to mirror the fix and the regression
there if you would prefer a paired change.

🤖 Generated with Claude Code