POC: zip as a built-in post_run hook (optional zipping)

[asmacdo]

Draft / WIP — stacked on #381 (the first 9 commits are that PR; the 6 on top are this one).

Moves output zipping out of the generated run script and into a built-in post_run hook, using the splice-point machinery from #381.
Zipping becomes opt-in: no zip hook configured = no zipping (#327, #364).
This PR introduces the built-in hook form (zip is the first, shipped as a static script), i.e.
it is partly a demonstration that the hooks design generalizes — flag anything that doesn't.

Config shape

The zip/output config is redefined around one new top-level key, output_dir — the folder the app writes into (relative to the dataset root), carrying the full versioned derivative name. It replaces both zip_foldernames (the name was derived from key + version) and all_results_in_one_zip:

# today                                    # this PR
all_results_in_one_zip: true               output_dir: outputs/fmriprep_anat-24-1-1
zip_foldernames:                           hooks:
    fmriprep_anat: "24-1-1"                  post_run:
                                               - builtin: zip   # zips `output_dir`

output_dir is the single source for the versioned name — app write dir = zip source = zip name, one string that can't drift. Zipping itself becomes just a hook: no zip hook = no zipping.

• The zip hook takes two optional parameters: path: (the folder to zip, relative to the dataset root; defaults to the top-level output_dir) and name: (the archive-name stem — the X in ${subid}[_${sesid}]_X.zip; defaults to path's basename). The archive contains the zipped folder at its top level. name: exists because in the old schema the map value only ever versioned the archive name; making it a free-form stem keeps that without any derivation.

• Non-default case — not all output in one zip (what the multi-entry zip_foldernames map did): one zip hook per folder, each with its own path: (+ name: when the app controls the folder name and the archive should stay versioned):

  output_dir: outputs    # the app writes here, creating the subfolders below
  hooks:
    post_run:
      - builtin: zip
        path: outputs/fmriprep_func
        name: fmriprep_func-24-1-1     # -> ${subid}_fmriprep_func-24-1-1.zip
      - builtin: zip
        path: outputs/freesurfer
        name: freesurfer-24-1-1        # -> ${subid}_freesurfer-24-1-1.zip

  The zip built-in is a static script taking its params as args at the splice site (bash ./code/hooks/zip.sh outputs/freesurfer freesurfer-24-1-1 — visible in participant_job.sh), so all instances share one code/hooks/zip.sh. Nothing zip varies on needs render time (sesid presence encodes the processing level at runtime, per the splice contract); the Render seam from the splice-points PR stays reserved for the container-running built-ins (NORDIC), which genuinely need init-time composition.

  Question for review: do any of your configs actually use multiple zip_foldernames entries? Every public config has exactly one.

• Built-in names and their params are validated against a registry at resolve time (a typo'd builtin or unknown param fails babs init, not the job).

Done so far in this draft

• The config swap is in: top-level output_dir replaces zip_foldernames / all_results_in_one_zip, which hard-error with a migration message — a clean break, no deprecation shim (question for review below). All shipped example configs (notebooks/eg_*.yaml), test fixtures, and the docs (zip_foldernames section → output_dir section) are migrated; the legacy derivation survives only for pipeline mode, which PR 3 deletes.
• {builtin: zip} materializes the packaged static code/hooks/zip.sh (git-tracked, copied verbatim) and splices it with its resolved arguments.
• The hook owns its own commit: datalad unlock → 7z inside its own datalad run --explicit → a separate git rm of the granular outputs (separate because datalad run --explicit doesn't track deletions — datalad run --explicit does not save file deletions in --output` paths datalad/datalad#7822, since fixed upstream; TODO in-script to fold it back in once babs's minimum datalad has the fix).
• The generated run script no longer zips: <container>_zip.sh → <container>_run.sh, the in-script 7z + rm -rf outputs tail is gone, and the participant job's datalad run -o declares the granular output_dir (the run commits granular outputs; the zip hook replaces them with the archive).
• The raw_bids e2e runs post_run: [<contract guard>, {builtin: zip}] — guard before zip, because hook ordering is load-bearing (a validator must see the outputs before zip flattens them) — so the runtime zip path is exercised end-to-end in CI.
• Unit + init-level test coverage and user docs throughout.

Demo: what babs init produces

Artifacts below are produced by this PR's init-level test (simbids, in the slurm-docker-ci container), from a config with output_dir: outputs/fmriprep_anat-24-1-1 and an argless {builtin: zip} at post_run.

code/hooks/zip.sh — the static zip built-in, copied in verbatim (git-tracked, so you can read exactly what will run)

#!/bin/bash
# Built-in babs `zip` hook: archive a BIDS App output folder, commit the
# archive, and remove the granular outputs it replaces.
#
# Copied verbatim at `babs init` into `code/hooks/zip.sh`. Runs at the
# `post_run` splice point (cwd = the job's dataset clone) as a separate
# process: `subid` (and `sesid` at session level) arrive via the exported
# splice-point contract; what to zip arrives as arguments:
#
#   zip.sh <path> [<name>]
#
# <path> is the folder to zip, relative to the dataset root. <name> is the
# archive-name stem (the X in ${subid}[_${sesid}]_X.zip), defaulting to
# <path>'s basename. The archive is written to the dataset root and contains
# the folder itself (not its parents), matching the layout of babs zips to
# date.
set -e -u -x

path="$1"
name="${2:-$(basename "$path")}"
zip_dir="$(dirname "$path")"
zip_folder="$(basename "$path")"

# subid is exported by the splice-point subshell in participant_job.sh;
# sesid only at session level, so its presence encodes the processing level.
# shellcheck disable=SC2154
ZIP_ID="${subid}${sesid:+_${sesid}}"
ZIP_NAME="${ZIP_ID}_${name}.zip"

# Zip real file content, not annex symlinks:
datalad unlock "${path}"

# cd into the parent so the archive contains the folder at its top level;
# OLDPWD (the dataset root) is where the archive lands.
datalad run \
	--explicit \
	--output "${ZIP_NAME}" \
	-m "Zip ${path} for ${ZIP_ID}" \
	-- \
	"cd ${zip_dir} && 7z a \"\${OLDPWD}/${ZIP_NAME}\" ${zip_folder}"

# `datalad run --explicit` does not track deletions, so the granular outputs
# are removed in a separate commit (workaround for datalad/datalad#7822,
# since fixed upstream).
# TODO research which datalad version shipped the datalad/datalad#7822 fix;
# once babs's minimum supported datalad is at or above it, fold this removal
# into the datalad run above and drop this step.
git rm -rf -q --sparse "${path}"
git commit -m "Remove ${path} for ${ZIP_ID} (zipped)"

code/participant_job.sh — the run's -o declares the granular output_dir; the post_run splice runs the zip hook with its resolved argument

# datalad run:
datalad run \
	-i "code/simbids-0-0-3_run.sh" \
	-i "inputs/data/BIDS/${subid}" \
	-i "inputs/data/BIDS/dataset_description.json" \
	-i "containers/.datalad/environments/simbids-0-0-3/image" \
	--expand inputs \
	--explicit \
	-o "outputs/fmriprep_anat-24-1-1" \
	-m "simbids-0-0-3 ${subid}" \
    "bash ./code/simbids-0-0-3_run.sh ${subid} "
# post_run hooks: spliced after the run, before push; subshell exports the contract.
(
  export subid BRANCH PROJECT_ROOT JOB_SCRATCH_DIR
bash ./code/hooks/zip.sh outputs/fmriprep_anat-24-1-1
)

# Finish up:
# push result file content to output RIA storage:
echo '# Push result file content to output RIA storage:'
datalad push --to output-storage

analysis dataset history after babs init — the static hook rides the imported-files commit

80bdd80 Save anything in folder code/ that hasn't been saved
2f4b876 Template for job submission
dc34509 Record of inclusion/exclusion of participants/sessions
695f62a Import files
3d4cab1 Generate script of running container
1e4f86f [DATALAD] Added subdataset
465ea59 Register input data dataset 'BIDS' as a subdataset
059b1da Initial save of babs_proj_config.yaml
62358f3 Save .gitignore file
580eaf7 Apply YODA dataset setup
45194cd [DATALAD] new dataset

Remaining before review-ready

• The clean break needs your sign-off: legacy keys hard-error with a migration message, no deprecation shim. Acceptable, or do you need a deprecation path?
• babs status with zipping off: OutputDataset currently hardcodes is_zipped=True, so merged-results accounting silently reports nothing when no zip hook is configured — needs the zip-presence persisted; follow-up, design open.
• Remove the TODO-fenced demo block from the init test (it exists to produce the artifacts above).
• Test state at last push: 99/103 green; 4 session-level render-shellcheck failures on non-*prep apps (suspected: sesid="$2" now unused once the in-script zip is gone → SC2034) — to fix, plus a fresh full-e2e run.

Known trade-off to weigh at review: post-run zip vs within-run zip

With the hook owning its own datalad run, each job commits granular outputs first, then the zip (two commits). The granular annex keys stay in history and the git-annex branch even after git rm — so babs merge at ~1000-subject scale gets strictly heavier than today's within-run zip. Is the two-commit shape tolerable at your scale?

If it isn't, the design answer we'd propose is not coupling zip back into the run script, but a second seam: command decoration — a configurable prefix/suffix on the command inside the datalad run wrapper (prepend_run/append_run), so zip happens within the run and only the archive is ever committed. That seam already has a second independent consumer: per-job resource monitoring with duct (#356), which also needs to wrap the wrapped command and can't be expressed as a pre_run/post_run hook. Caveat to weigh: a decoration lands in the recorded rerun command, so datalad rerun then requires the decorating tool on the host.

[asmacdo]

Folded into #381 — per today's community meeting we're combining the splice-points PR and the zip-as-hook PR into one (a release will be cut before merge, so the additive/behavioral split bought nothing). #381 now carries the full stack including the config swap (output_dir + optional zip) and the static built-in form.