Add `extra_paths` to input dataset config to retain non-subject files (e.g. `nidm.ttl`)

[asmacdo]

Add extra_paths to input dataset config to retain non-subject files (e.g. nidm.ttl)

Problem

babs init-generated participant_job.sh hard-codes the per-subject sparse-checkout
of each non-zipped input dataset to only ${subid} (and ${sesid}) plus
dataset_description.json:

babs/templates/participant_job.sh.jinja2:102

That means any other dataset-root files needed by the BIDS app are dropped from
the working tree.

Concrete case discovered with @djarecka: an NIDM input dataset has
sourcedata/NIDM/nidm.ttl which is symlinked into each subject. After the
per-subject sparse-checkout runs, nidm.ttl is no longer in the working tree,
the symlinks dangle, and datalad run fails with "no such file".

There should be a way to configure extra_paths so these files are included
in the sparse-checkout set.

The existing required_files field in the input dataset YAML doesn't help.
It's a per-subject inclusion filter, and per
docs/preparation_config_yaml_file.rst:115 it's not implemented yet anyway.

Proposal

Add an optional extra_paths list to each entry in input_datasets, e.g.:

input_datasets:
  NIDM:
    is_zipped: false
    origin_url: ...
    path_in_babs: sourcedata/NIDM
    extra_paths:
      - nidm.ttl

In babs/templates/participant_job.sh.jinja2, each extra_path should be:

1. Added to the per-subject git sparse-checkout set --stdin list (line 102).
   This is the load-bearing change — it's what makes the file visible to
   the BIDS app.
2. Optionally added to datalad run -i so annex content (if any) is fetched
   and provenance is recorded.

[djarecka]

Want to add one comment that I created the symlinks, becouse babs requires all input_datasets to have data per participant. My nidm.ttl file had already content for all the subjects, but to meet the requirements I created the subdirectories for each subject, but I didn't want to copy the file.

Perhaps, in addition to what @asmacdo wrote, we can release the requirement for input_datasets?

[tien-tong]

@asmacdo

Add an optional extra_paths list to each entry in input_datasets

This sounds good to me

@djarecka

we can release the requirement for input_datasets

I have not worked with nidm.ttl before, I took a quick look at http://nidm.nidash.org, but could you share an example of your current setup and what you’d like to change it to?

[djarecka]

I don't think it is important what is in the nidm.ttl file, basically the situation could be more common. Let's say we have some group level statistics, for nidm it is a file that annotate metedata for all subjects.
From the data structure it looks:

derivatives/
    nidm/
        nidm.ttl

And for some time, my workaround was to create symlinks and have:

derivatives/
    nidm/
        nidm.ttl
        sub-1/
            nidm.ttl -> ../nidm.ttl (symlink)
        sub-2/
            nidm.ttl -> ../nidm.ttl (symlink)

but after recent changes to babs, i.e., use sparse-checkout it stopped working.

[asmacdo]

We landed at "common_paths"

Behavior
default to ["dataset_description.json"]

common_paths: [] # would disable dataset_description for the case it doesnt exist
common_paths:
   - nidm.ttl         # include nidm.ttl but not dataset description

common_paths:
   - nidm.ttl 
   - dataset_description.json # for dorota's case. 

This would use existing machinery, and quite "hardcoding" a special case for dataset_description.json