Add pre_run/post_run hooks; make zipping an optional built-in hook

[asmacdo]

Implements the hooks proposal discussed in #365 — the splice-point mechanism plus its first real consumer: output zipping moves out of the generated run script into an optional built-in post_run hook, making zipping opt-in (#327, #364).
(Originally split as two stacked PRs; combined per the 2026-06-10 community meeting, since a release will be cut before this merges and the split bought nothing.)

Part 1 — the hooks mechanism

A new optional hooks: section in the container config YAML splices user-supplied shell commands into each participant job at two splice points that bracket the BIDS App run:

• pre_run — after job setup, just before the datalad run that executes the BIDS App.
• post_run — after that datalad run completes (its outputs committed), before results are pushed to the output RIA.

hooks:
  pre_run:
    - echo "starting ${subid}"               # snippet: spliced verbatim, runs inline
    - script: /path/to/validate-inputs.sh    # script: copied into code/hooks/ at init
  post_run:
    - script: /path/to/validate-outputs.sh
    - builtin: zip                           # built-in: ships with BABS

Three entry forms:

• snippet — a bare string, spliced verbatim into participant_job.sh.
• script — {script: <absolute path>}; copied into analysis/code/hooks/<basename>.sh at babs init (the same mechanism as imported_files, so the hook is git-tracked in the analysis dataset) and run as bash ./code/hooks/<name>.sh.
• built-in — {builtin: <name>}; a static script shipped inside BABS, copied in like a script hook. Its params become shell arguments at the splice site, resolved at config time — several instances of one built-in share a single script and differ only in args. zip is the first built-in.

Design points:

• Hooks splice outside the datalad run wrapper. The hook author owns commit semantics: the safe default use is validation that fails the job (a non-zero post_run hook aborts before push, so bad results never leave the node); a hook that persists output runs its own datalad run/save — which is exactly what the zip built-in does. This is also what lets a future preprocessing hook (NORDIC) produce an intermediate without committing it.
• A subshell-scoped exported runtime contract. Each splice region is a subshell exporting subid (+ sesid at session level), BRANCH, PROJECT_ROOT, JOB_SCRATCH_DIR, so a separate-process hook can read them; a hook's cd/variable changes can't leak into the rest of the job, and the export can't leak into the BIDS App environment. set -e is preserved: a failing hook fails the job.
• Resolution is a pure function. babs/hooks.py:resolve_hooks() classifies config entries (no I/O); bootstrap executes the copies through the existing _init_import_files. Collisions key on conflicting content, not name reuse: the same hook used at both splice points is copied once; two different sources claiming the same destination raise at init. Built-in names and params are validated against a registry, so a typo fails babs init, not the job.

Part 2 — zip as a built-in hook, and the config swap

The zip/output config is redefined around one new top-level key, output_dir — the folder the app writes into (relative to the dataset root), carrying the full versioned derivative name. It replaces both zip_foldernames (the name was derived from key + version) and all_results_in_one_zip:

# before                                   # after
all_results_in_one_zip: true               output_dir: outputs/fmriprep_anat-24-1-1
zip_foldernames:                           hooks:
    fmriprep_anat: "24-1-1"                  post_run:
                                               - builtin: zip   # zips `output_dir`

output_dir is the single source for the versioned name — app write dir = datalad run -o declaration = zip source = zip name, one string that can't drift. Zipping itself becomes just a hook: no zip hook = no zipping.

• The zip hook takes two optional parameters: path: (the folder to zip; defaults to the top-level output_dir) and name: (the archive-name stem — the X in ${subid}[_${sesid}]_X.zip; defaults to path's basename). In the old schema the map value only ever versioned the archive name; a free-form stem keeps that without any derivation.

• Multi-zip (what the multi-entry zip_foldernames map did): one zip hook per folder, each with its own path: (+ name: when the app controls the folder name and the archive should stay versioned):

  output_dir: outputs    # the app writes here, creating the subfolders below
  hooks:
    post_run:
      - builtin: zip
        path: outputs/fmriprep_func
        name: fmriprep_func-24-1-1     # -> ${subid}_fmriprep_func-24-1-1.zip
      - builtin: zip
        path: outputs/freesurfer
        name: freesurfer-24-1-1        # -> ${subid}_freesurfer-24-1-1.zip

• The zip built-in is a static script taking its params as args at the splice site (visible in participant_job.sh); all instances share one code/hooks/zip.sh. Nothing zip varies on needs render time — sesid presence encodes the processing level at runtime, per the splice contract. (An init-time rendered built-in form was prototyped during this work and removed as unconsumed; it returns alongside the container-running built-ins only if runtime arg/env composition turns out not to suffice there.)

• The hook owns its own commit: datalad unlock → 7z inside its own datalad run --explicit → a separate git rm of the granular outputs (separate because datalad run --explicit doesn't track deletions — datalad run --explicit does not save file deletions in --output` paths datalad/datalad#7822, since fixed upstream; TODO in-script to fold it back in once babs's minimum datalad has the fix).

• The generated run script no longer zips: <container>_zip.sh → <container>_run.sh, the in-script 7z + rm -rf outputs tail is gone, and the participant job's datalad run -o declares the granular output_dir.

• All shipped example configs (notebooks/eg_*.yaml), test fixtures, and docs are migrated; the legacy derivation survives only for pipeline mode, which the NORDIC follow-up deletes.

Breaking change, on purpose: the legacy keys hard-error with a migration message — a clean break, no deprecation shim (accepted at the 2026-06-10 community meeting; a release is cut before this merges).

Demo: what babs init produces

From a config with output_dir: outputs/fmriprep_anat-24-1-1 and an argless {builtin: zip} at post_run (simbids, in the slurm-docker-ci container):

code/hooks/zip.sh — the static zip built-in, copied in verbatim (git-tracked, so you can read exactly what will run)

#!/bin/bash
# Built-in babs `zip` hook: archive a BIDS App output folder, commit the
# archive, and remove the granular outputs it replaces.
#
# Copied verbatim at `babs init` into `code/hooks/zip.sh`. Runs at the
# `post_run` splice point (cwd = the job's dataset clone) as a separate
# process: `subid` (and `sesid` at session level) arrive via the exported
# splice-point contract; what to zip arrives as arguments:
#
#   zip.sh <path> [<name>]
#
# <path> is the folder to zip, relative to the dataset root. <name> is the
# archive-name stem (the X in ${subid}[_${sesid}]_X.zip), defaulting to
# <path>'s basename. The archive is written to the dataset root and contains
# the folder itself (not its parents), matching the layout of babs zips to
# date.
set -e -u -x

path="$1"
name="${2:-$(basename "$path")}"
zip_dir="$(dirname "$path")"
zip_folder="$(basename "$path")"

# subid is exported by the splice-point subshell in participant_job.sh;
# sesid only at session level, so its presence encodes the processing level.
# shellcheck disable=SC2154
ZIP_ID="${subid}${sesid:+_${sesid}}"
ZIP_NAME="${ZIP_ID}_${name}.zip"

# Zip real file content, not annex symlinks:
datalad unlock "${path}"

# cd into the parent so the archive contains the folder at its top level;
# OLDPWD (the dataset root) is where the archive lands.
datalad run \
	--explicit \
	--output "${ZIP_NAME}" \
	-m "Zip ${path} for ${ZIP_ID}" \
	-- \
	"cd ${zip_dir} && 7z a \"\${OLDPWD}/${ZIP_NAME}\" ${zip_folder}"

# `datalad run --explicit` does not track deletions, so the granular outputs
# are removed in a separate commit (workaround for datalad/datalad#7822,
# since fixed upstream).
# TODO research which datalad version shipped the datalad/datalad#7822 fix;
# once babs's minimum supported datalad is at or above it, fold this removal
# into the datalad run above and drop this step.
git rm -rf -q --sparse "${path}"
git commit -m "Remove ${path} for ${ZIP_ID} (zipped)"

code/participant_job.sh — the run's -o declares the granular output_dir; the post_run splice runs the zip hook with its resolved argument

# datalad run:
datalad run \
	-i "code/simbids-0-0-3_run.sh" \
	-i "inputs/data/BIDS/${subid}" \
	-i "inputs/data/BIDS/dataset_description.json" \
	-i "containers/.datalad/environments/simbids-0-0-3/image" \
	--expand inputs \
	--explicit \
	-o "outputs/fmriprep_anat-24-1-1" \
	-m "simbids-0-0-3 ${subid}" \
    "bash ./code/simbids-0-0-3_run.sh ${subid} "
# post_run hooks: spliced after the run, before push; subshell exports the contract.
(
  export subid BRANCH PROJECT_ROOT JOB_SCRATCH_DIR
bash ./code/hooks/zip.sh outputs/fmriprep_anat-24-1-1
)

# Finish up:
# push result file content to output RIA storage:
echo '# Push result file content to output RIA storage:'
datalad push --to output-storage

analysis dataset history after babs init — the static hook rides the imported-files commit

80bdd80 Save anything in folder code/ that hasn't been saved
2f4b876 Template for job submission
dc34509 Record of inclusion/exclusion of participants/sessions
695f62a Import files
3d4cab1 Generate script of running container
1e4f86f [DATALAD] Added subdataset
465ea59 Register input data dataset 'BIDS' as a subdataset
059b1da Initial save of babs_proj_config.yaml
62358f3 Save .gitignore file
580eaf7 Apply YODA dataset setup
45194cd [DATALAD] new dataset

What this deliberately does not do

• Pipeline mode is untouched (its call sites stay hooks-unaware; NORDIC-as-hook then deletes the whole codepath — see roadmap).
• No user-supplied jinja templates, no rendered built-ins. User content is only ever copied, never rendered.

Testing

• Resolver unit tests: form classification, content-collision semantics, builtin registry/param validation, arg resolution + shell quoting.
• Render tests: no-hooks output unchanged; hooks spliced in order at the right positions; sesid exported only at session level.
• Init-level tests: hook materialized + spliced (positive), no-hooks ⟹ no code/hooks/ (negative), zip builtin copied verbatim with its resolved arg visible.
• The raw_bids e2e splices a contract-guard script hook at both points (: "${subid:?}" etc. fails the job at runtime if the exported contract breaks) and runs post_run: [guard, {builtin: zip}] — guard before zip because hook ordering is load-bearing (a validator must see the outputs before zip flattens them).
• Known state at last push: 99/103 unit-level green; 4 session-level render-shellcheck failures on non-*prep apps (suspected: sesid="$2" now unused once in-script zip is gone) — to fix, plus a fresh full-e2e run.

Follow-ups (next PRs)

1. A generic singularity built-in (container-running hook, no datalad) — NORDIC then becomes that hook plus its specifics as additional hooks, after which pipeline mode is deleted (NORDIC is its only user).
2. babs merge as a dependent slurm compute job (upstream issue to come) — makes the two-commit zip's heavier merge acceptable at ~1k-subject scale.
3. babs status when zipping is off: merged-results accounting currently assumes zips; likely derivable from branch ancestry in git history instead of new persisted state.