fix(live): verify simulator teaching across fixed domains - #246
Merged
Merged
Conversation
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed-domain Phase 3 evaluations exposed malformed generation, misleading circuitry, and visual reviews that sometimes approved incorrect diagrams. The factory now repairs malformed successful responses within its existing candidate limit, includes rendering source in independent review, and assembles a tested series-resistor component for supported voltage/resistance instruments. Publication requires that component's verified rendered structure, topology, and state labels; a positive visual verdict cannot override structural rejection.
The evaluation report retains all 45 development builds: 21 engine approvals, 13 rejections, 11 unsuitable decisions, at least seven later manual rejections, actual usage/latency, and unknown-cost failures. Three selected domains—algorithms, physics, economics—pass fixed numeric oracles and independent manual source/screenshot review. Real tutor command states and negative publication cases are also verified. The critic is explicitly not claimed infallible.
Validation: 3,182 offline Python tests passed (18 skipped), 27 real browser tests passed; Ruff/format; Python/style/test/security reviews. Browser coverage includes circuit mutation rejection and deterministic replay of all three committed real-model artifacts. Paid generation tests remain intentionally capable of failing on a new run; the report separates selected successes from overall reliability.
Closes #244. Refs #237 and #219; production rollout and Pouyan's learner acceptance remain tracked separately.