feat(meg): skeptical validation diagnostics + offline golden recipe (#14)#15
Merged
juniperbevensee merged 2 commits intoJun 22, 2026
Conversation
) Two layers on top of the M100 measurement fix (#13), prompted by the live test where a wrong measurement was nearly accepted because the pipeline ran without error. 1. Skepticism in the study itself. `validate_finding` now withholds a *confident* `refuted` drawn from a noise-dominated average: below MIN_RELIABLE_EPOCHS (or with no measurable amplitude) an out-of-window peak is downgraded to `inconclusive` with a stated `next_step`, never silently accepted. Findings carry the material to sanity-check them — search_window_ms, channel, n_epochs, and a caveats list — so the agent reasons over evidence, not a single number. 2. A shipped, offline golden recipe. SyntheticMegIO + build_golden_steps run the SAME five-step pipeline and verdict logic against a planted in-window M100 — no mne / numpy / network / download. Exposed as study kind 'golden_meg_validation' so the agent can run it live as the self- demonstrating reference for "what correct looks like," and as an offline CI smoke/regression test. docs/golden-validation-recipe.md documents both the supported path and the skeptical-inconclusive path. 3. Methodology, not just tooling. SOUL gains a "be skeptical of your own measurements" principle; SKILL gains a "Reasoning Over Datasets" section: a result contradicting a well-established finding is a red flag to inspect, prefer inconclusive+next-step over a confident refuted on thin evidence, and imitate the golden recipe. Tests: skepticism boundaries (few-epochs out-of-window -> inconclusive; enough-epochs -> still refuted), diagnostics presence, golden recipe supported/inconclusive/no-heavy-deps, and the wired tool kind end-to-end. 143 passed, 7 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
juniperbevensee
deleted the
dev/juniperbevensee/smarter-validations-golden-recipe
branch
June 22, 2026 23:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14.
Two asks from the live M100 test, where a wrong measurement (15 ms artifact) was nearly accepted because the pipeline ran without error.
1. Make the study skeptical of its own measurements
validate_findingno longer reports a confidentrefuteddrawn from a noise-dominated average. BelowMIN_RELIABLE_EPOCHS(or with no measurable amplitude), an out-of-window peak is downgraded toinconclusivewith a statednext_step— increase the sample / inspect the evoked — instead of being accepted. Findings now carry the material to sanity-check them:search_window_ms,channel,n_epochs, and acaveatslist. (A strong out-of-window peak with enough epochs is stillrefuted— regression-tested.)2. Ship an offline, self-demonstrating golden recipe
SyntheticMegIO+build_golden_steps()run the same five-step pipeline and verdict logic against a planted in-window M100 — no mne / numpy / network / download. Exposed as study kindgolden_meg_validation(matilde_study_create kind="golden_meg_validation"→matilde_study_run) so the agent can run it live as the reference for "what a correct validation looks like," and as an offline CI smoke/regression test. The same builder also demonstrates the skeptical-inconclusive path (peak_latency_ms=300, n_epochs<threshold). Seedocs/golden-validation-recipe.md.3. Methodology, not just tooling
SOUL: a "be skeptical of your own measurements" principle. SKILL: a "Reasoning Over Datasets" section — a result contradicting a well-established finding (missing M100, implausible latency, near-noise amplitude, few epochs) is a red flag to inspect; prefer
inconclusive+ next step over a confidentrefuted; imitate the golden recipe.Tests
Skepticism boundaries, diagnostics presence, golden recipe (supported / skeptical-inconclusive / no-heavy-deps), and the wired tool kind end-to-end. 143 passed, 7 skipped; changed code pyflakes-clean (one pre-existing unrelated warning at
tools.py:86).Deploy note
Plugin code + SOUL/SKILL. No image rebuild strictly required for the engine/tools (plugins are hot-mounted), but SOUL/SKILL changes reach a running agent only on its next reload/redeploy. Coordinate as usual — don't wipe in-progress work.
🤖 Generated with Claude Code