Skip to content

feat: ask the curation judge only for indexed per-target relations (v2 contract) - #307

Merged
Barsoomx merged 3 commits into
masterfrom
feat/curation-next-lever
Jul 30, 2026
Merged

feat: ask the curation judge only for indexed per-target relations (v2 contract)#307
Barsoomx merged 3 commits into
masterfrom
feat/curation-next-lever

Conversation

@Barsoomx

Copy link
Copy Markdown
Owner

What

curation_decision_v2. The judge is asked for only an indexed per-target relation plus a
reason:

{"schema_version": 2,
 "comparisons": [{"index": 1, "relation": "equivalent", "applicability": "same"}],
 "reason": "..."}

Everything the code already derived since #304 (outcome, top-level relation,
target_memory_version_id, temporal_order, reason_code) and everything it only
echo-validated (candidate_evidence_refs, target_evidence_refs) is gone from the wire. The
prompt envelope drops both UUIDs per entry too — with index addressing nothing reads them.

Why, measured

An audit attributed every judge_invalid_output on the stand by traceback frame across 9h of
worker logs (105 parse failures):

raise site share cause
_validate_comparisons (curation_judge.py:376) 74% tuple(parsed ids) != version_ids — the model got the membership or order of the comparisons wrong
_parse_uuid (curation_judge.py:373) 26% the model wrote something that is not a UUID at all — it mangles 36-char ids, and current_transition_id sits right beside memory_version_id in the envelope
JSON parse / _TOP_KEYS / reason length 0 not a factor; 1 truncation in 30h

v2 attacks both directly: set(index) == {1..N} accepts any permutation and sorts server-side,
and the model never transcribes a UUID again.

Correction to an earlier figure. judge_invalid_output is ~5.4% of calls (26 of 478 over
5h), not the 32% quoted from a 22-minute window. The larger paid leak is stale_decision: 45%
of those are raised after the call, from _revalidate_shortlist — ~12% of calls. That is
slice B, not this PR.

Per-target applicability

applicability moves from one top-level value into each comparison. This closes the narrow
unsafety recorded in the #304 spec's Risks: the ladder could act on target B while the model's
single applicability: same was about target A. _apply_evidence_policy is kept as the
assertion and reads the selected target's value; the exhaustive property test still proves it
never fires, now enumerating applicability per target (2^N instead of 2).

Prompt size

v1 v2
schema instruction block ~4187 chars ~1475 chars
envelope, 12-entry shortlist ~7976 chars 5946 chars

−40% of input on a full shortlist. Output shrinks further — 12 echoed UUIDs, 12 ref arrays
and 5 dead top-level fields disappear — but that cannot be quantified without a live call.

DEPLOY STEP THAT IS NOT OPTIONAL

Adversarial review caught an inert seam in the original plan, and it is worth stating loudly.

The implementer expected the judge to drop to reasoning_effort: minimal automatically after
deploy, because the stand policy pins curation_decision_v1: "medium" and that key would stop
matching. That is false. resolve_reasoning_effort (services.py:1616-1621) reads effort
only from policy metadata; a key miss returns '', and services.py:1764 sends the parameter
only when non-empty. With nothing sent, OpenAI applies its own default — medium.
REASONING_EFFORT_BY_RESPONSE_KIND is baked into metadata at policy creation, not read at
runtime.

Since ~89% of the call cost is output tokens dominated by reasoning, deploying without acting
would leave this slice's main economic effect inert while appearing to have run a canary.

Required on deploy: set request_shape.reasoning_effort.curation_decision_v2 = "minimal" in the
active curation policy's metadata (or recreate the policy from the preset). That is the canary,
and it is a deliberate step. If the invalid-rate rises on minimal, the correction is a metadata
edit to "low".

For the record: the stand policy was created 2026-07-29 04:10 and never updated, with
curation_decision_v1: "medium" while every other kind is minimal. The rise in cost per call
observed last night predates the derivation work by 13 hours and was not caused by it.

Verification

pytest -m "not transactional" → 3500 passed. pytest -m transactional → 89 passed.
engram_curator_eval (fixture and responses engines) → all 12 thresholds at bound, 126 cases,
no threshold weakened, contract.py untouched. ruff check, ruff format --check,
makemigrations --check → clean.

Tests were proven to bind by mutation: removing the duplicate-index guard, reverting
applicability to global, and re-imposing sorted order each produce specific failures — including
the property test independently detecting the cross-target applicability unsafety.

Known dead weight left behind

  • curation.py:1606/1639 still raise judge_reference_invalid, now doubly unreachable (the
    parser resolves targets from facts.targets, so it cannot produce one outside the shortlist).
    Harmless, but the commit that dropped the unreachable branch only cleaned the judge.
  • The envelope still carries body_hash, content_hash, comparison_manifest_hash,
    authorized_corpus_count and comparison_complete — ~200-300 input tokens the model can no
    longer act on. Candidate for the next trim.
  • CurationDecision.applicability now stores 'not_applicable' for targetless outcomes instead
    of whatever global value the model happened to send. Consumed only by a passthrough serializer.

@Barsoomx

Copy link
Copy Markdown
Owner Author

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@codecov-commenter

codecov-commenter commented Jul 30, 2026

Copy link
Copy Markdown

Bundle Report

Bundle size has no change ✅

Affected Assets, Files, and Routes:

view changes for bundle: engram-frontend-client-array-push

Assets Changed:

Asset Name Size Change Total Size Change (%)
static/pYxKUchmdhJIoBOgD2FBY/_buildManifest.js (New) 1.51kB 1.51kB 100.0% 🚀
static/pYxKUchmdhJIoBOgD2FBY/_ssgManifest.js (New) 77 bytes 77 bytes 100.0% 🚀
static/6_MJrBp1IBhlMIncFp1Tf/_buildManifest.js (Deleted) -1.51kB 0 bytes -100.0% 🗑️
static/6_MJrBp1IBhlMIncFp1Tf/_ssgManifest.js (Deleted) -77 bytes 0 bytes -100.0% 🗑️

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.10%. Comparing base (6179d90) to head (983e598).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #307   +/-   ##
=======================================
  Coverage   91.10%   91.10%           
=======================================
  Files         223      223           
  Lines       23350    23320   -30     
=======================================
- Hits        21273    21246   -27     
+ Misses       2077     2074    -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@Barsoomx
Barsoomx merged commit e582ef3 into master Jul 30, 2026
14 checks passed
@Barsoomx
Barsoomx deleted the feat/curation-next-lever branch July 30, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants