OpenNeuro/BIDS dataset client + agent tools (discovery, metadata, files)#3
Merged
Merged
Conversation
Matilde's first scientific-data capability — the realistic-first-capability the research identified (discovery + metadata + files), not heavy compute. - engine/openneuro.py: stdlib-only client over the OpenNeuro GraphQL API + public no-auth S3 mirror (no datalad/git-annex/openneuro-py/AWS SDK). get_dataset, list_datasets, list_files, download_file; injected I/O for offline tests. Schema validated live against ds000246. - Tools: matilde_openneuro_dataset_info / _search / _list_files. - Fix: _tool_result dropped the message kwarg when passed a positional dict (caught by a live smoke); dataset_info now merges message into the payload. - 11 new offline tests + 3 live; 74 offline total green; gate clean. - Heavy paths (DataLad clone, fMRIPrep, NiMARE) explicitly scoped to the Docker layer / aspirational, per the research. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deterministic gate flags real-looking emails as PII; the placeholder contact-email example in engine/cli.py now uses an allowlisted example domain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
juniperbevensee
added a commit
that referenced
this pull request
Jun 22, 2026
…phy study) (#11) * feat(pipeline): stateful, resumable study store + runner (+ bibliography study) Long analyses currently run inline in one agent turn — they die on OOM and lose all state on container rebuild. This adds durable, checkpointed analysis state so a study resumes from the last completed step instead of vanishing. - engine/store.py: StudyStore (stdlib sqlite3, WAL, atomic). studies/steps/ artifacts/findings tables; create/get/list studies, add_steps, step status + result, artifacts, findings, study_summary. Injected db path (MATILDE_STUDY_DB / HERMES_HOME / repo engagements dir). State survives a fresh StudyStore instance on the same file (the container-rebuild guarantee, under test). - engine/pipeline.py: resumable runner. Step/StepContext/StepResult; run()/resume() skip done steps, persist result+artifacts+findings per step, stop on failure leaving the study blocked + resumable. Pure/injected. - engine/bibliography_study.py: first concrete study (parse_refs -> verify_each -> summarize) reusing the citations/parsing engine, verifier injected. verify_each is mid-step resumable (skips refs that already have findings). - tools.py / __init__.py: matilde_study_create/run/status/list, matching the existing tool-dict + thin-handler conventions. Tests (TDD): store CRUD + status transitions + persistence across a new instance; runner skip-done / stop-on-failure / resume-without-rerun; bibliography happy path and fail-on-ref-#3-then-resume (refs 1-2 NOT re-verified); tool arg coercion + output shape. Full suite: 123 passed, 6 skipped. Engine stays stdlib-only (no mne/numpy) — the heavy MEG data-analysis study is a documented follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pipeline): generalize motivation for public repo (drop dated incident specifics); note plan-evolution semantics Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Juniper Bevensee <juniperbevensee@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Matilde's first scientific-data capability — built from the adversarially-verified OpenNeuro research thread.
Scope: the realistic first capability
Read-only discovery → metadata → file listing/download, not heavy compute. Per the research, DataLad full-clones, fMRIPrep, and NiMARE meta-analysis are deferred to the Docker layer / aspirational.
What's here
get_dataset,list_datasets,list_files,download_file; injected I/O for offline tests. Schema validated live againstds000246.matilde_openneuro_dataset_info,matilde_openneuro_search,matilde_openneuro_list_files.Proven (live)
ds000246: 'MEG-BIDS Brainstorm data sample' — modalities ['meg','mri'], 1 subject(s), 2457671893 bytes (snapshot 1.0.1).Tests
11 new offline + 3 live (
MATILDE_LIVE=1). 74 offline total green. Sanitization gate clean. Also fixes a_tool_resultenvelope bug (droppedmessageon positional dict) caught by a live smoke.🤖 Generated with Claude Code