Dev 1.5.0

.agents/skills/monsda/SKILL.md

@@ -0,0 +1,80 @@
+---
+name: monsda
+description: Use MONSDA (Modular Organizer of Nextflow and Snakemake driven hts Data Analysis) to plan and run HTS/NGS pipelines — QC, trimming, mapping, dedup, counting, tracks, peak calling, DE/DEU/DAS/DTU, circRNA, fusion detection — from a single config.json that MONSDA turns into Snakemake or Nextflow workflows. Use whenever the user wants to analyze FASTQ/BAM sequencing data with this repo, needs a MONSDA config built or edited, needs an experimental design mapped to a condition-tree, or wants to generate/dry-run/execute the resulting workflows instead of doing it by hand.
+---
+
+# MONSDA
+
+MONSDA is a config-driven wrapper: one JSON config describes samples, genomes
+and tool choices; MONSDA compiles it into per-condition Snakemake or Nextflow
+sub-workflows and can run them. Your job as the agent is to turn the user's
+experiment description into a correct config, validate it cheaply before
+spending compute, and only then run it — instead of the user hand-editing
+JSON and guessing at tool names.
+
+Read the reference files below on demand (do not load all of them up front):
+
+- `reference/config-guide.md` — config.json anatomy, condition-tree, OPTIONS/TOOLS, samplesheet shortcut. Read before writing or editing any config.
+- `reference/workflows.md` — available `WORKFLOWS` steps and the tools each one accepts. Read when choosing which steps/tools to enable.
+- `reference/cli.md` — `monsda` / `monsda_configure` CLI flags, the generate-then-dry-run validation recipe, cluster profiles. Read before invoking any command.
+
+## Decision flow
+
+1. **Clarify the experiment, not the tool syntax.** Ask the user (or infer from
+   what they already said): experimental design/conditions, where the
+   FASTQ/BAM files and genome/annotation live, sequencing type (single/paired/
+   single-cell, stranded?), and which analysis steps they want
+   (QC/TRIMMING/MAPPING/DEDUP/COUNTING/TRACKS/PEAKS/DE/DEU/DAS/DTU/CIRCS/
+   FUSIONS). Don't ask about MONSDA internals they aren't expected to know.
+2. **Map the design to a condition-tree** (`ID -> CONDITION -> SETTING`,
+   see `reference/config-guide.md`). This fixes the expected input directory
+   layout (`FASTQ/<ID>/<CONDITION>/...`) and the output tool-key directories.
+3. **Build the config.** Prefer, in order:
+   - If the user has (or can export) a sample table, use the built-in
+     `--samplesheet file.csv` support (`configs/samplesheet_template.csv`
+     shows the columns) instead of hand-writing the nested `SETTINGS` block.
+   - Otherwise copy the closest starting point — `configs/template_clean.json`
+     (empty skeleton) or `configs/tutorial_quick.json` (small worked example)
+     — and edit it. Several other files under `configs/` (`template.json`,
+     `tutorial_toolmix.json`, `tutorial_exhaustive.json`,
+     `tutorial_postprocess.json`) carry trailing `#comments` for human
+     documentation and are **not valid JSON**; read them for explanations but
+     don't copy them as-is — see `reference/config-guide.md` for the full list.
+   - Keep `WORKFLOWS` as a single comma-separated string in the **order**
+     steps should run, and give every listed step a corresponding top-level
+     key (`QC`, `MAPPING`, ...), each with a `TOOLS` map and per-condition
+     `OPTIONS` — MONSDA errors out otherwise.
+   - Set `VERSION` to the exact installed MONSDA version the run will use
+     (`python -c "import MONSDA; print(MONSDA.__version__)"` with the same
+     interpreter/env as the run) — a mismatch aborts the run.
+4. **Validate before executing anything expensive.** Run
+   `scripts/validate_config.py <config.json>` (does structural/JSON checks and
+   cross-checks referenced sample/reference files) and then the
+   generate-only + dry-run recipe in `reference/cli.md` (`monsda --save` then
+   `snakemake -n` / nextflow equivalent on the generated sub-workflows). This
+   catches missing files, bad tool names and condition-tree mistakes without
+   running a single real job.
+5. **Run it.** Only after a clean dry run, execute for real (local, or via a
+   cluster profile — see `reference/cli.md`). Report the actual command used
+   and where outputs land (tool-key directories under e.g. `MAPPING/<id>/
+   <condition>/<combo>/`).
+
+## What this buys the user over doing it by hand
+
+- No need to memorize the condition-tree <-> directory-layout mapping or the
+  per-tool `OPTIONS` schema — you look it up in the reference files.
+- `scripts/validate_config.py` catches config mistakes before a conda env
+  solve or a multi-hour mapping job starts.
+- You generate-and-dry-run (`--save` + `snakemake/nextflow -n`) before
+  committing to a real run, which is the step people skip when working by
+  hand and MONSDA doesn't enforce.
+
+## Notes
+
+- `monsda_configure` is an interactive terminal UI meant for humans; it is
+  not practical to drive from an agent. Build/edit the JSON directly (or via
+  `--samplesheet`) instead of trying to script the TUI.
+- If you are working inside a MONSDA git checkout rather than an installed
+  release, see this repo's `AGENTS.md` for the dev-only gotchas (symlinks
+  under `MONSDA/MONSDA`, PATH for `snakemake`/`nextflow`, golden-file tests).
+  None of that applies to an end user running an installed `monsda`.

.agents/skills/monsda/reference/cli.md

@@ -0,0 +1,112 @@
+# CLI and validation recipe
+
+Full docs: `docs/source/runsmk.rst`, `docs/source/cluster.rst`. This covers
+what an agent needs to generate, dry-run, and then execute a config safely.
+
+## Environment
+
+```bash
+conda activate monsda   # or whatever env MONSDA was installed into
+```
+
+`monsda` (Snakemake backend, default) and `monsda --nextflow` both need
+`snakemake`/`nextflow` on PATH respectively; MONSDA aborts immediately if the
+selected engine's binary is missing.
+
+## `monsda` flags that matter for an agent
+
+```
+monsda -c CONFIG.json -d DIRECTORY -j PROCS [options]
+```
+
+- `-c/--config/--configfile CONFIG [CONFIG ...]`: one JSON, or JSON + an
+  extra Nextflow-specific config.
+- `-d/--directory`: the run/project root (where `FASTQ/`, `GENOMES/`,
+  `SubSnakes|SubFlows/`, `JOBS/`, `LOGS/` live or will be created).
+- `-j/--procs`: parallelism, capped by the config's `MAXTHREADS`.
+- `--save`: **generate only** — write the sub-workflow files and the CLI
+  calls to `JOBS/` without executing anything. This is the key step for
+  validating a config cheaply.
+- `-s/--skeleton`: just create the minimal directory hierarchy, no config
+  processing.
+- `--snakemake` (default) / `--nextflow`: pick the execution engine.
+- `-u/--use-conda`: use conda envs (default true).
+- `-l/--unlock`: unlock a Snakemake working directory after an interrupted run.
+- `--clean`: Nextflow work-dir cleanup (`-n` to preview, `-f` to actually delete).
+- `--loglevel {WARNING,ERROR,INFO,DEBUG}`.
+- `--samplesheet FILE`: populate `SETTINGS` from a CSV/TSV when the config
+  lacks one (see `reference/config-guide.md`); writes
+  `<config>_with_settings.json` on first use.
+- `--oras-registry HOST` / `--oras-namespace NAMESPACE`: override the
+  container registry/namespace for ORAS image pulls (defaults
+  `ghcr.io` / `jfallmann/monsda`).
+- `--version`: print and exit.
+
+Additional unrecognized arguments are passed through to Snakemake/Nextflow
+verbatim, e.g. `-n` (Snakemake dry-run), `--omit-from <rule>` (skip a rule and
+everything downstream without removing it from `WORKFLOWS`), `--profile
+<dir>` (cluster profile).
+
+## Validate-before-run recipe (do this before a real execution)
+
+1. Run the repo's lightweight structural check first:
+   `python scripts/validate_config.py CONFIG.json` (see this skill's
+   `scripts/` directory) — catches invalid JSON, missing step blocks, a
+   `VERSION` mismatch, and `SETTINGS`/`REFERENCE`/`ANNOTATION` paths that
+   don't exist on disk, before anything talks to Snakemake/Nextflow.
+2. Generate without executing:
+   ```bash
+   monsda -c CONFIG.json -d DIRECTORY -j 2 --save
+   ```
+   This writes the per-condition sub-workflows (`SubSnakes/*.smk` or
+   `SubFlows/*.nf`) and the exact CLI invocations MONSDA would run into
+   `JOBS/MONSDA.commands`.
+3. Dry-run every generated sub-workflow using the commands recorded in
+   `JOBS/MONSDA.commands`, adding `-n` (Snakemake) — e.g.:
+   ```bash
+   snakemake --snakefile SubSnakes/<name>.smk --configfile <subconfig>.json -n
+   ```
+   For Nextflow, `nextflow run <subflow>.nf -preview` only validates syntax,
+   not the script bodies — a real `-n`-equivalent dry run is not available,
+   so inspect the generated `.nf`/its `params` instead, or do a tiny real run
+   on a minimal subset of samples first.
+4. Fix whatever the dry run reports (commonly `MissingInputException` for
+   files that don't exist yet, or a bad tool/env name) and repeat from step 2
+   before launching the full, real run.
+
+## Running for real
+
+```bash
+monsda -c CONFIG.json -d DIRECTORY -j THREADS --conda-frontend mamba \
+  --conda-prefix PATH_TO_CONDA_ENVS
+```
+
+or, for Nextflow:
+
+```bash
+monsda --nextflow -c CONFIG.json -d DIRECTORY -j THREADS
+```
+
+### On a cluster (SLURM example)
+
+```bash
+monsda -c CONFIG.json -d DIRECTORY -j THREADS --conda-frontend mamba \
+  --profile profile_snakemake --conda-prefix PATH_TO_CONDA_ENVS
+```
+
+```bash
+export NXF_EXECUTOR=slurm
+monsda --nextflow -c CONFIG.json -d DIRECTORY -j THREADS
+```
+
+`profile_snakemake/` and `profile_nextflow/` in this repo ship example
+profiles to adapt.
+
+### Resuming a run while skipping a step
+
+Keep the full original `WORKFLOWS` list (see `reference/config-guide.md`),
+and pass Snakemake's own skip flag instead of editing the config:
+
+```bash
+monsda -c CONFIG.json -d DIRECTORY -j THREADS --omit-from multiqc
+```

.agents/skills/monsda/reference/config-guide.md

@@ -0,0 +1,169 @@
+# config.json anatomy
+
+Full human docs: `docs/source/config.rst`, `docs/source/conditiontree.rst`,
+`docs/source/preparation.rst`. This is the condensed, agent-facing version.
+
+## Top level
+
+```json
+{
+  "WORKFLOWS": "QC,TRIMMING,MAPPING,DEDUP",
+  "BINS": "",
+  "MAXTHREADS": "20",
+  "VERSION": "1.5.0",
+  "SETTINGS": { ... },
+  "QC": { ... },
+  "TRIMMING": { ... },
+  "MAPPING": { ... },
+  "DEDUP": { ... }
+}
+```
+
+- `WORKFLOWS`: comma-separated step names, in run order. Every step listed
+  here needs a matching top-level key with its own settings, or MONSDA errors.
+  Available steps: see `reference/workflows.md`.
+- `VERSION` must equal the installed `MONSDA.__version__` for the interpreter
+  that will run the job, or the run aborts.
+- `MAXTHREADS` caps `-j` passed on the CLI.
+- `BINS`: path to custom user scripts; advanced/rare, leave `""` otherwise.
+
+### Rerunning on an existing output directory
+
+Keep the **full, original** `WORKFLOWS` list even if you only want to rerun a
+later step (e.g. keep `QC,TRIMMING,MAPPING,DEDUP` even if only `DE` changed).
+The tool-key/`combo` directory name is derived from which stages are listed;
+dropping an earlier stage changes that name and makes existing outputs look
+missing, forcing a full rerun. To skip actually re-executing one rule, add
+Snakemake's own `--omit-from <rule>` on the command line instead of editing
+`WORKFLOWS`.
+
+## The condition-tree (`SETTINGS`)
+
+```
+'ID' -> 'CONDITION' -> 'SETTING' (optional third level)
+```
+
+```json
+"SETTINGS": {
+  "LabA": {
+    "wild-type": {
+      "SAMPLES": ["Sample_1", "Sample_2"],
+      "SEQUENCING": "paired",
+      "REFERENCE": "GENOMES/Dm6/dm6.fa.gz",
+      "ANNOTATION": {
+        "GTF": "GENOMES/Dm6/dm6.gtf.gz",
+        "GFF": "GENOMES/Dm6/dm6.gff3.gz"
+      }
+    }
+  }
+}
+```
+
+- Each leaf of the tree is one independent analysis unit. The tree also
+  dictates the **input** directory layout: `FASTQ/<ID>/<CONDITION>/...`
+  (MONSDA looks one level above the deepest leaf for input, so adding a
+  `SETTING` sub-level for e.g. benchmarking different tool params does not
+  require duplicating the FASTQ files).
+- `SAMPLES`: list of sample basenames, no file extension. For paired-end,
+  list the name once without `_R1`/`_R2` — MONSDA appends those
+  automatically and requires files to end in `_R1.fastq.gz`/`_R2.fastq.gz`
+  (single-end: `<name>.fastq.gz`).
+- `SEQUENCING`: `single`, `paired`, or `singlecell`; optionally append
+  comma-separated strandedness (`rf` or `fr`, see RSeQC `infer_experiment.py`
+  convention) e.g. `"paired,fr"`. Leave strandedness empty if unstranded —
+  MONSDA configures per-tool strandedness flags automatically from this.
+- `REFERENCE` / `ANNOTATION.GTF` / `ANNOTATION.GFF` / `INDEX` / `PREFIX` /
+  `DECOY`: per-condition defaults, can be overridden per workflow step.
+- `GROUPS` / `BATCHES` / `TYPES`: optional, same length as `SAMPLES`, used
+  only by DE/DEU/DAS/DTU design matrices (batch correction, type covariate).
+- `IP`: only for `PEAKS` with CLIP-type protocols (`CLIP`, `iCLIP`, `revCLIP`).
+
+Output directories are named by a derived **tool-key/combo**, e.g.
+`MAPPING/LabA/wild-type/fastqc-cutadapt-star-umitools/`, built from the
+enabled workflow steps + tools so different tool combinations never collide.
+
+## Per-workflow-step blocks
+
+Each step listed in `WORKFLOWS` needs:
+
+```json
+"MAPPING": {
+  "TOOLS": { "star": "STAR" },
+  "LabA": {
+    "wild-type": {
+      "ENV": "star",
+      "BIN": "STAR",
+      "star": {
+        "OPTIONS": {
+          "INDEX": "--some-star-index-flag value",
+          "MAP": "--some-star-mapping-flag value"
+        }
+      }
+    }
+  }
+}
+```
+
+- `TOOLS`: map of `conda-env-name: executable-name` for every tool used in
+  this step (env name usually equals the bioconda package, see `envs/`).
+- `ENV`/`BIN` at the condition level override the step-level default (useful
+  to benchmark two installs of the same tool, or point at a local binary for
+  tools without a conda package, e.g. `guppy`, `dorado`).
+- Inside each tool's block, `OPTIONS` holds one key per sub-step (e.g.
+  `INDEX` vs `MAP` for a mapper) with a **raw command-line string** of extra
+  flags for that tool — exactly as you'd type them, minus anything MONSDA
+  derives automatically (paired/stranded flags, threads). Leave `OPTIONS: {}`
+  if no extra flags are needed; never invent a flag that isn't in the tool's
+  own `--help`.
+- `COMPARABLE` (DE/DEU/DAS/DTU and MACS peak calling): if omitted/empty,
+  MONSDA generates all-vs-all comparisons from `GROUPS`. Otherwise:
+  ```json
+  "COMPARABLE": { "comparison-name": ["Group1", "Group2"] }
+  ```
+
+## Fastest path: samplesheet instead of hand-written SETTINGS
+
+If the user can provide a CSV/TSV, skip writing `SETTINGS` by hand. Columns
+(see `configs/samplesheet_template.csv`):
+
+```
+CONDITION,SAMPLE,GROUP,SEQUENCING,REFERENCE,GTF,GFF,INDEX,PREFIX,DECOY,TYPE,BATCH,IP
+```
+
+`CONDITION` is a `/`-separated path that becomes the condition-tree branch
+(e.g. `FGUMI/WT/dummylevel`). Rows sharing a `CONDITION` are grouped into one
+leaf's `SAMPLES` list; only the first row of a group needs the
+REFERENCE/GTF/GFF/etc. columns filled in. Pass it to `monsda` directly with
+`--samplesheet file.csv` (it populates `SETTINGS` in the given config JSON on
+the first run and writes `<config>_with_settings.json` for reuse), or let
+`monsda_configure --samplesheet file.csv` do it when scaffolding a brand-new
+project.
+
+## PostDE (optional enrichment on top of DE)
+
+Only for the `DE` workflow with `deseq2`/`edger` (not DEU/DAS/DTU). Enabled
+via a top-level `POSTDE` key pointing at a separate analysis JSON (gProfiler/
+clusterProfiler/GSVA/decoupleR/dream). See `docs/source/config.rst` section
+"PostDE" and the shipped example `configs/postde_analysis.json` before
+building one — it has its own schema and conda env (`postde`).
+
+## Starting points in this repo
+
+Valid JSON, safe to copy/parse directly:
+
+- `configs/template_clean.json` — empty skeleton with every workflow step's
+  key already scaffolded.
+- `configs/tutorial_quick.json` — small worked example (FETCH+MAPPING on the
+  bundled `tests/data` Ecoli set).
+- `configs/template_umicollapse.json` — UMI-aware dedup example.
+- `configs/postde_analysis.json` — example PostDE analysis JSON.
+
+**Documentation-only, NOT valid JSON** — these have trailing `#comment`
+annotations after values and will fail `json.load`; read them for the
+explanations but copy structure from the valid files above instead:
+`configs/template.json`, `configs/template_base_commented.json`,
+`configs/tutorial_toolmix.json`, `configs/tutorial_exhaustive.json`,
+`configs/tutorial_postprocess.json`. Strip the `#...` trailing comments
+before using one of these as a base, or diff it against
+`configs/template_clean.json` to see the annotation-free version of the same
+keys.