feat(meg): skeptical validation diagnostics + offline golden recipe (#14)

docker/SOUL.Matilde.md

@@ -17,6 +17,13 @@ not verify this" than present an unverified claim as fact.
   exists. Unknown is a valid, respectable answer.
 - **Evidence over fluency.** A well-written paragraph with a fabricated citation is
   a failure, not a success. Sourcing beats eloquence.
+- **Be skeptical of your own measurements.** A number a tool hands you is a claim,
+  not a fact. When a measured result *contradicts a well-established finding* — a
+  missing M100 in a dataset famous for it, a peak at an implausible latency, an
+  amplitude near noise, a verdict drawn from too few epochs — treat it as a red
+  flag to inspect, not a result to report. Re-run with a stronger sample or open
+  the intermediates before you conclude; prefer "inconclusive, here's the next
+  step" over a confident "refuted" built on thin evidence.
 - **Reproducibility.** Another researcher should be able to follow your steps and
   reach the same conclusion. Record what you checked, against which source, and when.
 - **Respect the scholarly record.** Retractions, corrections, and uncertainty are

docs/golden-validation-recipe.md

@@ -0,0 +1,76 @@
+# Golden validation recipe (offline, self-demonstrating)
+
+Builds on the [MEG data-sample validation study](meg-validation-study.md) and the
+[stateful study pipeline](stateful-study-pipeline.md). The **golden recipe** is a
+fully worked, dependency-free validation that runs the *same* five-step pipeline
+and the *same* verdict logic as the real `bst_auditory` study — but against a
+synthetic backend that plants a known in-window M100. No `mne`, no `numpy`, no
+network, no multi-GB download.
+
+It exists for two reasons:
+
+1. **A reference the agent imitates.** A fresh Matilde can run it to see what a
+   correct, well-evidenced validation looks like before judging a real result —
+   the shape of a `supported` finding, with clean diagnostics.
+2. **An offline regression smoke test.** It exercises the pipeline + skepticism
+   logic deterministically in CI, with nothing heavy installed
+   (`tests/test_golden_meg.py`, `tests/test_study_tools.py`).
+
+## Why a golden recipe
+
+The original M100 study *ran without error* yet measured the wrong number — a
+15 ms stimulus artifact reported as the peak, calling a textbook finding
+`refuted`. "The pipeline ran" is not evidence the science is right. A golden run
+with a **known** answer gives both the agent and CI a fixed point: if the planted
+in-window peak does not come back `supported` with clean diagnostics, something
+regressed in the pipeline or the verdict logic.
+
+## Run it live
+
+```
+matilde_study_create  slug="golden-demo"  kind="golden_meg_validation"
+matilde_study_run     study_id=<id>
+matilde_study_status  study_id=<id>
+```
+
+Expected: status `done`, one finding, `verdict: "supported"`, `latency_ms: 100.0`
+inside the 80–120 ms window, a healthy `n_epochs`, and `caveats: []`.
+
+## What it demonstrates
+
+A correct finding carries the material to **sanity-check** it, not just a verdict:
+
+| field | golden value | why it matters |
+|---|---|---|
+| `verdict` | `supported` | in-window peak from a reliable sample |
+| `latency_ms` | `100.0` | the planted peak |
+| `expected_window_ms` | `[80.0, 120.0]` | the auditory M100 window |
+| `search_window_ms` | in-window bounds | provenance of the measurement (not the old ±100 ms widening) |
+| `n_epochs` | `40` | above `MIN_RELIABLE_EPOCHS`, so the average is trustworthy |
+| `channel` | the peak channel | which sensor carried it |
+| `caveats` | `[]` | nothing to flag |
+
+## The skepticism path
+
+The same recipe demonstrates the *defensive* behavior too. Built in code with a
+thin, out-of-window sample:
+
+```python
+from matilde_plugin.engine.meg_study import build_golden_steps, MIN_RELIABLE_EPOCHS
+steps = build_golden_steps(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS - 1)
+```
+
+This does **not** return a confident `refuted`. Because the epoch count is below
+`MIN_RELIABLE_EPOCHS`, the evoked average is noise-dominated, so the study
+downgrades to `inconclusive` and attaches a `next_step` ("increase the sample /
+larger crop, or inspect the evoked intermediate, before concluding refuted").
+That is the rule a real validation must follow: a result contradicting a
+well-established finding is a red flag to inspect, never a confident refutation
+built on weak evidence.
+
+## Where it lives
+
+- Backend + builder: `matilde_plugin/engine/meg_study.py`
+  (`SyntheticMegIO`, `build_golden_steps`, `GOLDEN_PEAK_MS`, `MIN_RELIABLE_EPOCHS`).
+- Tool kind: `golden_meg_validation` (`matilde_plugin/tools.py`).
+- Tests: `tests/test_golden_meg.py`, `tests/test_study_tools.py`.

hermes-skill/SKILL.md

@@ -116,6 +116,40 @@ Deliver the answer or draft with:
 - an explicit "could not verify" section for anything that came back `not_found`,
   `retracted`, `unverifiable`, or `warnings` you could not resolve.
 
+## Reasoning Over Datasets — Be Skeptical of Measured Results
+
+Verifying a finding against an open dataset (e.g. the bounded-sample MEG study,
+`matilde_study_create kind="meg_validation"`) produces a **measured number**, and
+a number from a tool is a *claim*, not a fact. The failure mode to guard against
+is accepting a wrong measurement because the pipeline ran without error — a study
+once reported a 15 ms "peak" (a stimulus artifact) and called a textbook M100
+`refuted`. The pipeline worked; the science was wrong.
+
+**Treat a result that contradicts a well-established finding as a red flag, not an
+answer.** A missing M100 in a dataset famous for it, a peak at an implausible
+latency, an amplitude near noise, or a verdict drawn from very few epochs all mean
+*inspect*, not *report*. Before concluding:
+
+- **Read the diagnostics, not just the verdict.** Study findings carry the
+  material to sanity-check them — `latency_ms`, `amplitude`, `n_epochs`, the
+  `expected_window_ms` and the `search_window_ms` actually used, the `channel`,
+  and any `caveats`. A `low_evidence` caveat or a thin `n_epochs` is your cue.
+- **Never report a confident `refuted` on weak evidence.** The study already
+  withholds it — a refutation from too few epochs is downgraded to `inconclusive`
+  with a `next_step`. Honor that: say "inconclusive, here is the next step"
+  (increase the sample / a larger crop, or open the evoked intermediate), not
+  "refuted."
+- **Cross-check against the literature.** If a measured value disagrees with a
+  well-replicated result, the prior is that *your sample or parameters* are off,
+  not that the literature is wrong. Adjust and re-run before claiming a refutation.
+
+**Know what correct looks like.** Run the offline golden recipe —
+`matilde_study_create kind="golden_meg_validation"` then `matilde_study_run` — for
+a fully worked, dependency-free example: a planted in-window M100 that yields
+`supported` with clean diagnostics. It needs no dataset download and doubles as
+the package's smoke test; imitate its shape when judging a real result. See
+`docs/golden-validation-recipe.md`.
+
 ## Iteration Pattern
 
 1. **Gather** candidate sources

matilde_plugin/engine/meg_study.py

@@ -88,6 +88,16 @@
 # artifact (~15 ms) instead of the in-window auditory peak (~100-120 ms).
 _PEAK_SEARCH_MARGIN_MS = 10.0
 
+# Below this many epochs the evoked average is noise-dominated, so an out-of-
+# window (or absent) peak is more likely a weak-sample artifact than a real
+# refutation. We therefore WITHHOLD a confident ``refuted`` under this count and
+# return ``inconclusive`` with a stated next step instead (#14 skepticism: a
+# result that contradicts a well-established finding — e.g. a missing M100 in a
+# dataset famous for it — is a red flag to inspect, not to accept). A genuine
+# bst_auditory standards average has dozens of epochs; a handful means the crop
+# was too small. Tune conservatively — this only gates the *refuted* downgrade.
+MIN_RELIABLE_EPOCHS = 8
+
 
 def _peak_search_bounds(window_ms: Tuple[float, float],
                         times_lo: float, times_hi: float) -> Tuple[float, float]:
@@ -259,7 +269,10 @@ def measure_peak(self, evoked_handle: dict, *,
         ch, latency_s, amp = ev.get_peak(tmin=lo, tmax=hi, mode="abs",
                                          return_amplitude=True)
         return {"latency_ms": float(latency_s) * 1000.0,
-                "amplitude": float(amp), "n_epochs": n_epochs, "channel": ch}
+                "amplitude": float(amp), "n_epochs": n_epochs, "channel": ch,
+                # The actual in-window search bounds (ms) used for this peak — a
+                # diagnostic so the verdict's provenance is inspectable (#14).
+                "search_ms": [lo * 1000.0, hi * 1000.0]}
 
 
 # ---------------------------------------------------------------------------
@@ -349,19 +362,49 @@ def fn(ctx: StepContext) -> StepResult:
             window_ms=window_ms)
         latency = peak.get("latency_ms")
         amplitude = peak.get("amplitude")
+        n_epochs = peak.get("n_epochs")
         verdict = _classify(latency, window_ms)
+
+        # Skepticism layer (#14): surface the material an agent needs to sanity-
+        # check the number, and refuse to report a *confident* refutation drawn
+        # from too thin a sample. A `refuted` that rests on a noise-dominated
+        # average (few epochs, or no measurable amplitude) is downgraded to
+        # `inconclusive` with a next step — never silently accepted.
+        caveats: List[str] = []
+        next_step = None
+        low_evidence = isinstance(n_epochs, int) and n_epochs < MIN_RELIABLE_EPOCHS
+        if low_evidence:
+            caveats.append(
+                f"low_evidence: {n_epochs} epochs is below the reliable "
+                f"threshold ({MIN_RELIABLE_EPOCHS}); the evoked average is "
+                f"noise-dominated and a single peak number is not yet trustworthy")
+        if verdict == "refuted" and (low_evidence or amplitude is None):
+            verdict = "inconclusive"
+            next_step = (
+                "Refutation withheld on weak evidence: a missing or displaced "
+                "M100 in a dataset known for it is more likely too few epochs "
+                "than a real absence. Increase the sample (more epochs / a "
+                "larger crop) and re-run, or inspect the evoked intermediate, "
+                "before concluding refuted.")
+
         claim = (f"auditory M100 peak for {dataset_id} falls within "
                  f"{window_ms[0]:.0f}-{window_ms[1]:.0f} ms")
+        evidence = {
+            "latency_ms": latency,
+            "amplitude": amplitude,
+            "expected_window_ms": list(window_ms),
+            "search_window_ms": peak.get("search_ms"),
+            "channel": peak.get("channel"),
+            "n_epochs": n_epochs,
+            "caveats": caveats,
+        }
+        if next_step:
+            evidence["next_step"] = next_step
         finding = {
             "claim": claim,
             "verdict": verdict,
             "score": None,
-            "evidence": {
-                "latency_ms": latency,
-                "amplitude": amplitude,
-                "expected_window_ms": list(window_ms),
-                "n_epochs": peak.get("n_epochs"),
-            },
+            "evidence": evidence,
         }
         return StepResult(data={"verdict": verdict, "latency_ms": latency,
                                 "amplitude": amplitude},
@@ -397,3 +440,82 @@ def build_steps(*, dataset_id: str = "bst_auditory",
         _evoked_step(io),
         _validate_finding_step(io, dataset_id, window),
     ]
+
+
+# ---------------------------------------------------------------------------
+# Golden recipe (#14): a shipped, offline, dependency-free worked example.
+#
+# The real study needs mne + a multi-GB download; that is the wrong thing to run
+# in CI or to point a fresh agent at as "what correct looks like." The golden
+# recipe runs the SAME five-step pipeline and the SAME verdict/skepticism logic
+# against a synthetic backend that plants a known in-window M100 — deterministic,
+# stdlib-only, instant. It doubles as a regression smoke test and a reference the
+# agent can imitate. See docs/golden-validation-recipe.md.
+# ---------------------------------------------------------------------------
+
+GOLDEN_DATASET_ID = "synthetic-golden"
+GOLDEN_PEAK_MS = 100.0          # planted peak latency — squarely inside 80-120
+GOLDEN_N_EPOCHS = 40            # a healthy, reliable epoch count
+
+
+class SyntheticMegIO:
+    """A real, dependency-free MEG backend that plants a known M100 peak.
+
+    Mirrors the :class:`MegIO` contract exactly, but fabricates small dict
+    handles instead of touching mne / numpy / the filesystem / the network — so
+    the full study runs deterministically and instantly. The planted peak
+    (``peak_latency_ms``, default :data:`GOLDEN_PEAK_MS`) sits inside the default
+    80-120 ms window with a healthy epoch count, so a correct pipeline yields
+    ``supported``. The parameters let a test also plant an out-of-window or
+    thin-sample peak to exercise the skepticism path.
+    """
+
+    def __init__(self, *, peak_latency_ms: float = GOLDEN_PEAK_MS,
+                 peak_amp: float = 4.2e-13, n_epochs: int = GOLDEN_N_EPOCHS):
+        self.peak_latency_ms = float(peak_latency_ms)
+        self.peak_amp = float(peak_amp)
+        self.n_epochs = int(n_epochs)
+
+    def fetch_sample(self, *, dataset_id: str, bounds: dict) -> dict:
+        return {"dataset_id": dataset_id, "bounds": dict(bounds),
+                "path": f"synthetic://{dataset_id}/raw"}
+
+    def preprocess(self, raw_handle: dict, *, l_freq: float, h_freq: float) -> dict:
+        return {**raw_handle, "filtered": [l_freq, h_freq],
+                "path": "synthetic://filtered"}
+
+    def epoch(self, filtered_handle: dict, *, tmin: float, tmax: float,
+              event_id: Any = None) -> dict:
+        return {**filtered_handle, "n_epochs": self.n_epochs,
+                "window": [tmin, tmax], "event_id": event_id,
+                "path": "synthetic://epochs"}
+
+    def evoked(self, epochs_handle: dict) -> dict:
+        return {**epochs_handle, "averaged": True, "path": "synthetic://evoked"}
+
+    def measure_peak(self, evoked_handle: dict, *,
+                     window_ms: Tuple[float, float]) -> dict:
+        if self.n_epochs <= 0:
+            return {"latency_ms": None, "amplitude": None, "n_epochs": 0,
+                    "channel": None, "search_ms": list(window_ms)}
+        lo, hi = _peak_search_bounds(window_ms, -0.1, 0.3)
+        return {"latency_ms": self.peak_latency_ms, "amplitude": self.peak_amp,
+                "n_epochs": self.n_epochs, "channel": "MEG 1631",
+                "search_ms": [lo * 1000.0, hi * 1000.0]}
+
+
+def build_golden_steps(*, peak_latency_ms: float = GOLDEN_PEAK_MS,
+                       n_epochs: int = GOLDEN_N_EPOCHS,
+                       expected_window_ms: Tuple[float, float] = DEFAULT_WINDOW_MS
+                       ) -> List[Step]:
+    """Build the offline *golden* meg_validation pipeline (synthetic backend).
+
+    No mne / numpy / network — a fully deterministic worked example that yields
+    ``supported`` for a planted in-window M100. Doubles as the package's smoke
+    test and the reference recipe (see docs/golden-validation-recipe.md). The
+    defaults plant a correct peak; override ``peak_latency_ms`` / ``n_epochs`` to
+    demonstrate the refuted / skeptical-inconclusive paths.
+    """
+    io = SyntheticMegIO(peak_latency_ms=peak_latency_ms, n_epochs=n_epochs)
+    return build_steps(dataset_id=GOLDEN_DATASET_ID, io=io,
+                       expected_window_ms=expected_window_ms)

matilde_plugin/tools.py

@@ -509,7 +509,7 @@ def _coerce_plan(plan: Any) -> list:
             "title": {"type": "string", "description": "Human-readable title."},
             "plan": {"type": "array", "items": {"type": "string"},
                      "description": "Ordered step names (e.g. ['parse_refs','verify_each','summarize'])."},
-            "kind": {"type": "string", "description": "Optional study kind. 'bibliography' wires the built-in reference-validation steps; 'meg_validation' wires the memory-bounded MEG evoked-peak (M100) validation steps."},
+            "kind": {"type": "string", "description": "Optional study kind. 'bibliography' wires the built-in reference-validation steps; 'meg_validation' wires the memory-bounded MEG evoked-peak (M100) validation steps; 'golden_meg_validation' runs the offline, dependency-free worked example (a planted in-window M100 -> 'supported') — the self-demonstrating reference recipe, no mne or download needed."},
             "bibtex": {"type": "string", "description": "For kind='bibliography': the BibTeX to validate (stored on the study)."},
             "dataset_id": {"type": "string", "description": "For kind='meg_validation': the open MEG dataset to validate against (e.g. 'bst_auditory')."},
             "expected_window_ms": {"type": "array", "items": {"type": "number"}, "description": "For kind='meg_validation': the [low, high] expected peak-latency window in ms (default [80, 120] for the auditory M100)."},
@@ -531,7 +531,7 @@ def _handle_study_create(args: dict, **kwargs: Any) -> str:
         bibtex = args.get("bibtex")
         if kind == "bibliography" and not plan:
             plan = ["parse_refs", "verify_each", "summarize"]
-        if kind == "meg_validation" and not plan:
+        if kind in ("meg_validation", "golden_meg_validation") and not plan:
             plan = ["fetch_sample", "preprocess", "epoch", "evoked",
                     "validate_finding"]
         meta: dict = {}
@@ -600,6 +600,12 @@ def _handle_study_run(args: dict, **kwargs: Any) -> str:
         if kind == "bibliography":
             from .engine.bibliography_study import build_steps
             steps = build_steps(bibtex=meta.get("bibtex", ""))
+        elif kind == "golden_meg_validation":
+            # Offline, dependency-free worked example — plants a known in-window
+            # M100 and yields `supported`. The self-demonstrating reference
+            # recipe; needs no mne / download (docs/golden-validation-recipe.md).
+            from .engine import meg_study
+            steps = meg_study.build_golden_steps()
         elif kind == "meg_validation":
             from .engine import meg_study
             window = meta.get("expected_window_ms") or list(
@@ -614,7 +620,8 @@ def _handle_study_run(args: dict, **kwargs: Any) -> str:
             return _tool_error(
                 f"Study {sid} has no runnable step implementations "
                 f"(kind={kind!r}). Built-in runner currently supports "
-                f"kind='bibliography' and kind='meg_validation'.", study_id=sid)
+                f"kind='bibliography', 'meg_validation', and "
+                f"'golden_meg_validation'.", study_id=sid)
         summary = run(store, sid, steps)
         return _tool_result(
             study_id=sid, status=summary["status"], steps=summary["steps"],