fix: stop evicting paid verdicts on project-wide comparison_complete flips

[Barsoomx]

What

Shortlist revalidation stops comparing comparison_complete. Instead, completeness is checked
only for the outcomes that actually read it, with a short bounded wait before the transaction and
a hard re-check inside it.

Why

comparison_complete is project-wide and flips constantly: every write outcome creates a new
memory version, therefore a new RetrievalDocument with a NULL embedding_pgvector, and
corpus_fully_embedded is false if any current document in the authorized corpus is
unembedded. Because it sat in shortlist_revalidation_hash, one publish anywhere in the project
evicted every other in-flight decision — after its provider call had already been paid for.

Measured on the stand: 648 paid stale_decision from _revalidate_shortlist over 9h; in the
latest 32-minute canary window, 65 stale against 33 judge_invalid_output. Paid stale is the
largest remaining source of discarded inference.

PR #306 removed authorized_corpus_count from this same hash for the same reason and doubled the
yield. It did not touch comparison_complete. This finishes that job.

How the gating stays honest

decision_requires_comparison_complete does not hardcode a list of outcomes. It re-runs
derive_decision with replace(facts, comparison_complete=False) and reports whether the derived
decision changes. If a rung's dependence on completeness ever changes, this follows automatically.
A lattice test pins the currently-gated set to exactly {publish_new, supersede_memory, open_conflict}.

This is exact rather than approximate because the verdict itself is produced by the same pure
derive_decision over the same judge_input and the same comparisons
(parse_curation_judge_verdict, curation_judge.py:478-481) — the helper re-derives, it does not
guess.

Lowering comparison_complete True→False is monotonic: it only removes eligibility, never adds
it. So derived != downgraded catches exactly the dependence on completeness, and
reject_candidate can never require it — which is why the wait is deliberately not run on the
reject path, leaving that path's lock ordering byte-for-byte unchanged.

The bounded wait

3 attempts, 2.0s apart → at most 4.0s of sleep and 3 read-only shortlist rebuilds.

• No transaction and no lock is held: _shortlist_memory_fence is a plain read, and the
  targetless-publication advisory lock is taken inside transaction.atomic() after the wait.
• WORK_TIMEOUT_SPECS[CANDIDATE_DECISION] is soft 240 / hard 270 / lease 300; a work that
  reaches the wait has spent ≤90s, so 4s is ~2.7% of the remaining budget. The wait cannot expire
  a lease, including on the retry path.
• A genuine entry change exits immediately — the probe goes through the full
  _revalidate_shortlist, which raises stale_decision at once.
• The wait is an optimisation; the in-transaction re-check is the correctness boundary.

What this does NOT fix, stated plainly

The canary outcome mix is 105 publish / 42 merge / 0 reject. Eviction used to be outcome-blind,
so roughly 71% of the paid stale was publish_new — the one outcome that still requires
completeness. The hash relaxation fully saves only merge/revise, about 29%.

For publish the saving depends on an assumption that has not been measured: that the embedding
lands inside the 4s window. The ~0.3s figure comes from PR #305 and was not re-measured here.
And structurally, the loser of two concurrent targetless publications still always gets stale —
its re-check runs immediately after the winner releases the lock, with no time for an embedding.

After deploy, measure residual paid stale for publish_new specifically, not the total.

Known limitation: if the embedding queue stalls for longer than the wait, every publish_new
attempt burns 4s and then re-pays for the judge call on retry. That is pre-existing behaviour —
this change neither worsens nor protects against it. There is no circuit breaker.

Test changes, fully disclosed

Beyond the five re-authored tests described in the commits, this branch also sweeps
replace(orch.shortlist_entry(memory), team_id=None) across roughly 20 pre-existing tests in
curation_tests.py (merge / revise / supersede / conflict / cp4-fault / concurrency /
redundant-reject).

Reason: those fixtures encoded verdicts that are not derivable under the inverted contract.
CurationShortlistEntry.team_id comes from memory.team_id, and PromoteMemoryCandidate sets
team_id = candidate.team_id even for PROJECT visibility, while the effective scope of a
project candidate is (PROJECT, None) — so same_visibility was False and every mutation rung
was unreachable for those fixtures.

This is a fixture artefact, not a production bug — verified on live data: all 14952
approved/conflict memories on the stand are project visibility with team_id NULL, so
same_visibility behaves correctly in production.

Consequence worth stating: no test now exercises _requires_comparison_complete through a
team-bearing entry. Risk is low — in production the verdict and the facts are always produced by
the same derive_decision — but the gap is real.

Verification

pytest -m "not transactional" → 3514 passed. pytest -m transactional → 89 passed.
engram_curator_eval → all thresholds at bound, no threshold weakened. ruff check,
ruff format --check → clean.

Tests were proven to bind by nine mutations, including: making the helper always return True,
making _settle_revalidation unconditionally stale, and deleting the in-transaction guard —
each producing a specific, named failure. Both hashes are pinned in opposite directions
(test_revalidation_hash_ignores_comparison_completeness and
test_manifest_hash_still_covers_comparison_completeness).

One unattributed flake was seen: a single StopIteration in the transactional split, followed by
six green runs. Mechanism is plausibly unrelated to this diff but was not attributed — worth
remembering on a first red CI here.

[Barsoomx]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[codecov-commenter]

Bundle Report

Bundle size has no change ✅

Affected Assets, Files, and Routes:

view changes for bundle: engram-frontend-client-array-push

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
server/app/(admin)/model-*.js	-105 bytes	18.94kB	-0.55%
server/app/(admin)/model-*.js	105 bytes	19.05kB	0.55%
server/middleware-*.js	-2.72kB	495 bytes	-84.59%
server/middleware-*.js	2.72kB	3.21kB	548.89% ⚠️
static/gCHH5ObaZCYaBr478lkY_/_buildManifest.js (New)	1.51kB	1.51kB	100.0% 🚀
static/gCHH5ObaZCYaBr478lkY_/_ssgManifest.js (New)	77 bytes	77 bytes	100.0% 🚀
static/dMFrKL7ufIJJxEdHTkpWq/_buildManifest.js (Deleted)	-1.51kB	0 bytes	-100.0% 🗑️
static/dMFrKL7ufIJJxEdHTkpWq/_ssgManifest.js (Deleted)	-77 bytes	0 bytes	-100.0% 🗑️

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.91667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.11%. Comparing base (e582ef3) to head (c8ef246).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
apps/backend/engram/memory/curation.py	96.77%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #308   +/-   ##
=======================================
  Coverage   91.10%   91.11%           
=======================================
  Files         223      223           
  Lines       23320    23356   +36     
=======================================
+ Hits        21246    21281   +35     
- Misses       2074     2075    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.