runtime: enforce target, fallback, and evaluation boundary - #12
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94bacbc684
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| """Run the experimental architecture and return an attributed receipt.""" | ||
|
|
||
| _validate_text(text) | ||
| neural_receipt = self.neural.infer(text) |
There was a problem hiding this comment.
This is the only inference execution call: self.core is never given output from the neural, circle, or seed layers, and the receipt merely appends _core_state captured during construction. Consequently, evaluate() can report a four-layer PTCNA verdict even though no path into the core or fiq propagation ran; connect inference through the constructed core before attributing this backend as PTCNA.
AGENTS.md reference: AGENTS.md:L3-L7
Useful? React with 👍 / 👎.
| self.target = target if target is not None else PTCNAEngine() | ||
| self.fallback = fallback if fallback is not None else HashedLinearFallback() |
There was a problem hiding this comment.
When normal PTCNARuntime() target construction raises—for example, if the large default build_core allocation exhausts available memory—the exception occurs here before the fallback is constructed and before infer(..., fallback_on_error=True) can handle it. Even an explicit backend="fallback" call is then unreachable, so isolate or lazily initialize the target to preserve fallback operation during target initialization failures.
AGENTS.md reference: AGENTS.md:L32-L34
Useful? React with 👍 / 👎.
| target_factory: BackendFactory = PTCNAEngine, | ||
| comparator_factory: BackendFactory = HashedLinearFallback, |
There was a problem hiding this comment.
These factories are supplied only at execution time and are absent from EvaluationPlan.to_dict() and its digest; identity validation also accepts any implementation that copies the expected identity string. The same preserved plan digest can therefore produce arbitrary different verdicts by passing different factories or target configurations after the freeze, so include an immutable implementation/configuration receipt in the plan or restrict overrides to a separate test-only surface.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| return _receipt( | ||
| plan, | ||
| started, | ||
| status=UNRESOLVED, |
There was a problem hiding this comment.
When target_factory or comparator_factory raises, a plan that preselected backend_error_status="FALSIFIED" still receives the hard-coded UNRESOLVED status. This changes the frozen failure propagation based solely on whether the backend error occurs during construction rather than inference, so use the plan's selected backend-error status for this path as well.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| target_result = target.infer(case.text) | ||
| comparator_result = comparator.infer(case.text) |
There was a problem hiding this comment.
The deadline is checked only before these potentially long-running calls. If either inference on the final case crosses max_seconds, there is no subsequent time check before the evaluator emits SURVIVED — not proved or FALSIFIED, so an execution that exceeded its frozen resource bound can receive a substantive verdict instead of resource_limit_status; recheck elapsed time after backend work and before the final verdict.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| """Reward the neural owner; structural layers remain non-differentiating.""" | ||
|
|
||
| outcome = _validate_reward(winner, outcome) | ||
| result = dict(self.neural.reward(winner, outcome)) |
There was a problem hiding this comment.
Although this wrapper forwards winner, PCNAEngine.reward() ignores that argument and applies the same outcome nudge to every ring, whereas HashedLinearFallback.reward() updates the winner-specific weight row. With the labeled evaluation workload, changing every expected label therefore leaves the target's trained state unchanged while changing comparator training, so the reported accuracies do not compare the two backends on equivalent learning semantics; make the target update winner-specific or evaluate a task interface it actually implements.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| try: | ||
| target.infer(case.text) | ||
| comparator.infer(case.text) | ||
| target.reward(case.expected_winner, plan.reward_outcome) |
There was a problem hiding this comment.
For the documented positive reward, this call reaches MemoryCore.flush_to(), whose _reset() seeds its RNG from time.time() in ptcna/neural/memory_core.py, but the plan contains no random seed or initial-state receipt. Re-running the same plan digest can consequently train different target states and potentially produce different terminal verdicts, so make the training randomness deterministic and include its seed or initial-state identity in the frozen plan.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| self._weights[winner_index] += ( | ||
| self.learning_rate * outcome * self._last_features | ||
| ) |
There was a problem hiding this comment.
The fallback stores only the most recent input's features, so if callers retain two fallback receipts and then reward the first one, this update combines the first receipt's winner with the second inference's features. The public PTCNARuntime.reward(receipt, ...) API does not require rewards to be immediate, making delayed or out-of-order feedback silently train the wrong example; retain features by inference identifier or bind them to a validated receipt.
AGENTS.md reference: AGENTS.md:L32-L34
Useful? React with 👍 / 👎.
| target_winner = target_result.get("winner") | ||
| comparator_winner = comparator_result.get("winner") |
There was a problem hiding this comment.
These values are never checked against WINNER_RINGS, so a comparator that returns {}, None, or an out-of-domain winner is counted as simply incorrect rather than as a backend failure. If the target meets its threshold, that broken comparator contributes zero accuracy and can let the evaluation emit SURVIVED — not proved even when the plan selected UNRESOLVED for backend errors; validate both result schemas and route invalid winners through the frozen failure status.
AGENTS.md reference: AGENTS.md:L35-L38
Useful? React with 👍 / 👎.
| { | ||
| "requested_backend": requested_backend, | ||
| "backend_used": backend_used, | ||
| "fallback_used": backend_used == FALLBACK_BACKEND, |
There was a problem hiding this comment.
For a caller-supplied fallback whose legitimate identity differs from the built-in FALLBACK_BACKEND constant, an explicitly requested fallback receipt sets fallback_used to false even though self.fallback.infer() ran and backend_used names that fallback. Since the constructor publicly accepts any InferenceBackend, derive this flag from the selected runtime role rather than one concrete implementation identity so fallback use remains accurately attributed.
AGENTS.md reference: AGENTS.md:L32-L34
Useful? React with 👍 / 👎.
1 participant
What changed
PTCNAEngine, an attributed target receipt that constructs the complete local core and reports neural, circle, seed, and core separatelyHashedLinearFallback, a deterministic 53-feature online linear learner under its own non-PTCNA identityPTCNARuntime; target failure raises by default, while explicit failover records the requested backend, actual backend, routing reason, and error typeEvaluationPlanand terminalEvaluationReceiptsrc.coremodulesWhy
PR #11 corrected the governing contract but left it descriptive. The repository still lacked an executable target/fallback boundary and a mechanism that freezes the critical “does it work?” verdict inputs before outcome inspection.
Construction remains independent of the verdict. The fallback preserves operation without silently replacing PTCNA. Passing contract tests does not establish representative utility.
Conceptual boundary consulted
b4234ca29529f56526541df8deb58c2c1957079253315e30c54aba881a5b48cbf395890e83ab05c5File plan
ptcna/runtime.pyptcna/evaluation.pyptcna/tests/test_runtime.pyptcna/tests/test_evaluation.pyptcna/__init__.pyptcna/neural/main.pyREADME.mdCHANGELOG.mddocs/architecture.mddocs/work-graphs/repository-plan-report.jsonptcna_msdmd.tsValidation
166 passedCLOSEDEvidence boundary
This PR establishes executable construction, fallback, and evaluation separation. It does not establish that PTCNA or the fallback is useful on a representative workload. That workload and its externally justified frozen settings remain unresolved.
hmmm