feat: ask the curation judge only for indexed per-target relations (v2 contract) - #307
Merged
Conversation
Owner
Author
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Bundle ReportBundle size has no change ✅ Affected Assets, Files, and Routes:view changes for bundle: engram-frontend-client-array-pushAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #307 +/- ##
=======================================
Coverage 91.10% 91.10%
=======================================
Files 223 223
Lines 23350 23320 -30
=======================================
- Hits 21273 21246 -27
+ Misses 2077 2074 -3 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
curation_decision_v2. The judge is asked for only an indexed per-target relation plus areason:
{"schema_version": 2, "comparisons": [{"index": 1, "relation": "equivalent", "applicability": "same"}], "reason": "..."}Everything the code already derived since #304 (
outcome, top-levelrelation,target_memory_version_id,temporal_order,reason_code) and everything it onlyecho-validated (
candidate_evidence_refs,target_evidence_refs) is gone from the wire. Theprompt envelope drops both UUIDs per entry too — with index addressing nothing reads them.
Why, measured
An audit attributed every
judge_invalid_outputon the stand by traceback frame across 9h ofworker logs (105 parse failures):
_validate_comparisons(curation_judge.py:376)tuple(parsed ids) != version_ids— the model got the membership or order of the comparisons wrong_parse_uuid(curation_judge.py:373)current_transition_idsits right besidememory_version_idin the envelope_TOP_KEYS/ reason lengthv2 attacks both directly:
set(index) == {1..N}accepts any permutation and sorts server-side,and the model never transcribes a UUID again.
Correction to an earlier figure.
judge_invalid_outputis ~5.4% of calls (26 of 478 over5h), not the 32% quoted from a 22-minute window. The larger paid leak is
stale_decision: 45%of those are raised after the call, from
_revalidate_shortlist— ~12% of calls. That isslice B, not this PR.
Per-target applicability
applicabilitymoves from one top-level value into each comparison. This closes the narrowunsafety recorded in the #304 spec's Risks: the ladder could act on target B while the model's
single
applicability: samewas about target A._apply_evidence_policyis kept as theassertion and reads the selected target's value; the exhaustive property test still proves it
never fires, now enumerating applicability per target (
2^Ninstead of 2).Prompt size
≈ −40% of input on a full shortlist. Output shrinks further — 12 echoed UUIDs, 12 ref arrays
and 5 dead top-level fields disappear — but that cannot be quantified without a live call.
DEPLOY STEP THAT IS NOT OPTIONAL
Adversarial review caught an inert seam in the original plan, and it is worth stating loudly.
The implementer expected the judge to drop to
reasoning_effort: minimalautomatically afterdeploy, because the stand policy pins
curation_decision_v1: "medium"and that key would stopmatching. That is false.
resolve_reasoning_effort(services.py:1616-1621) reads effortonly from policy metadata; a key miss returns
'', and services.py:1764 sends the parameteronly when non-empty. With nothing sent, OpenAI applies its own default — medium.
REASONING_EFFORT_BY_RESPONSE_KINDis baked into metadata at policy creation, not read atruntime.
Since ~89% of the call cost is output tokens dominated by reasoning, deploying without acting
would leave this slice's main economic effect inert while appearing to have run a canary.
Required on deploy: set
request_shape.reasoning_effort.curation_decision_v2 = "minimal"in theactive curation policy's metadata (or recreate the policy from the preset). That is the canary,
and it is a deliberate step. If the invalid-rate rises on minimal, the correction is a metadata
edit to
"low".For the record: the stand policy was created 2026-07-29 04:10 and never updated, with
curation_decision_v1: "medium"while every other kind isminimal. The rise in cost per callobserved last night predates the derivation work by 13 hours and was not caused by it.
Verification
pytest -m "not transactional"→ 3500 passed.pytest -m transactional→ 89 passed.engram_curator_eval(fixture and responses engines) → all 12 thresholds at bound, 126 cases,no threshold weakened,
contract.pyuntouched.ruff check,ruff format --check,makemigrations --check→ clean.Tests were proven to bind by mutation: removing the duplicate-index guard, reverting
applicability to global, and re-imposing sorted order each produce specific failures — including
the property test independently detecting the cross-target applicability unsafety.
Known dead weight left behind
curation.py:1606/1639still raisejudge_reference_invalid, now doubly unreachable (theparser resolves targets from
facts.targets, so it cannot produce one outside the shortlist).Harmless, but the commit that dropped the unreachable branch only cleaned the judge.
body_hash,content_hash,comparison_manifest_hash,authorized_corpus_countandcomparison_complete— ~200-300 input tokens the model can nolonger act on. Candidate for the next trim.
CurationDecision.applicabilitynow stores'not_applicable'for targetless outcomes insteadof whatever global value the model happened to send. Consumed only by a passthrough serializer.