Nightly regression: tests/integration/test_remote_update.py::TestPinHelpersRefuseLoudly::test_real_pyproject_declarations_resolve

[valorengels]

Failing node

tests/integration/test_remote_update.py::TestPinHelpersRefuseLoudly::test_real_pyproject_declarations_resolve

Surfaced by nightly regression triage (lane nightly-triage-f7ec48e1).

Observed failure

Reproduced locally with ./scripts/pytest-clean.sh <node>:

tests/integration/test_remote_update.py:586: in test_real_pyproject_declarations_resolve
    assert get_pinned_version(PROJECT_DIR, "anthropic") == "0.125.0"
E   AssertionError: assert '1.4.0' == '0.125.0'
E
E     - 0.125.0
E     + 1.4.0

Likely cause

Stale test, not broken behavior. get_pinned_version is reading the live pyproject.toml correctly; the test hardcodes version values that have since been bumped twice.

• tests/integration/test_remote_update.py:584-589 asserts the literal current values of the live pins:
  • anthropic == "0.125.0" — actual is 1.4.0 (pyproject.toml:11)
  • pydantic-ai-slim == "2.9.0" — actual is 2.40.0 (pyproject.toml:12)
  • openai is None — still correct, pyproject.toml:29 declares openai>=3.0.0, a floor not a pin
• Pins moved in 029b69abc (anthropic 1.3.0 / pydantic-ai-slim 2.39.0, Upgrade the LLM stack: anthropic 1.3.0 + pydantic-ai-slim 2.39.0, openai 3.x floor #3140) and again in 80dfecd78 ("Bump deps: anthropic 1.3.0->1.4.0, pydantic-ai-slim 2.39.0->2.40.0"). Neither updated the test.

This test is structurally guaranteed to fail on every future dependency bump: it is asserting the content of the declarations when its stated purpose (docstring: "The live file is the shape that produced every spike-2 defect") is to pin the shape the parser must handle — an exact pin, a pin buried behind an extra plus a trailing comment, and a floor.

Suggested next steps

1. Re-express the assertions against shape rather than value, so a routine bump cannot break it. For example: parse the declaration line for each package out of pyproject.toml and assert get_pinned_version returns exactly that version; keep get_pinned_version(PROJECT_DIR, "openai") is None as a literal, since "a floor is not a pin" is the invariant under test and is value-independent.
2. If an exact-value assertion is genuinely wanted as a bump tripwire, it belongs in its own clearly-named test so the parser-shape coverage does not go red with it.
3. Note that pydantic-ai-slim is declared as pydantic-ai-slim[anthropic]==2.40.0 (extra + trailing comment on the same line) — whatever replaces the hardcoded value must keep exercising that shape, since it is the case the helper was written for.

Related

Same file, same triage batch, independent root cause (the #3001 Step 2 hold was released):
tests/integration/test_remote_update.py::TestCoupledSetDeclaration::test_llm_set_is_declared_and_held — #3321.

---

Triage 2026-09-15

Status: confirmed still failing on main 205344717. This issue now owns the full cluster (duplicates closed into it).

Single-line hotfix: Yes, two small literal updates (not one shared line, since the two nodes assert independent facts). tests/integration/test_remote_update.py:586-587 — update assert get_pinned_version(PROJECT_DIR, "anthropic") == "0.125.0" to "1.5.0" and assert get_pinned_version(PROJECT_DIR, "pydantic-ai-slim") == "2.9.0" to "2.43.0", matching the live pyproject.toml:11-12 pins. tests/integration/test_remote_update.py:718 — update assert llm_set.hold == "#3001 Step 2" to assert llm_set.hold is None, matching scripts/update/deps.py:467-479 where the anthropic/pydantic-ai-slim CoupledSet no longer declares a hold (the Step 2 bump it was parking already landed).

Reduce-complexity option: For test_real_pyproject_declarations_resolve, yes — read pyproject.toml as source of truth instead of hardcoding version literals. The test's job is to catch parser regressions against the real file, not to pin exact version numbers, so it should assert get_pinned_version() against a second, deliberately-naive extraction (e.g. a plain regex over the raw dependencies list) rather than a literal string that must be hand-updated on every version bump — that keeps it a real regression check without going stale. For test_llm_set_is_declared_and_held, no — hold records a one-off human decision about when to execute a parked bump, not derived data, so hardcoding the expected value is correct; it will need this same kind of update again whenever that decision changes, which is acceptable since it's rare and intentional.