diff --git a/.agents/skills/monsda/SKILL.md b/.agents/skills/monsda/SKILL.md new file mode 100644 index 00000000..1dbd2f57 --- /dev/null +++ b/.agents/skills/monsda/SKILL.md @@ -0,0 +1,80 @@ +--- +name: monsda +description: Use MONSDA (Modular Organizer of Nextflow and Snakemake driven hts Data Analysis) to plan and run HTS/NGS pipelines — QC, trimming, mapping, dedup, counting, tracks, peak calling, DE/DEU/DAS/DTU, circRNA, fusion detection — from a single config.json that MONSDA turns into Snakemake or Nextflow workflows. Use whenever the user wants to analyze FASTQ/BAM sequencing data with this repo, needs a MONSDA config built or edited, needs an experimental design mapped to a condition-tree, or wants to generate/dry-run/execute the resulting workflows instead of doing it by hand. +--- + +# MONSDA + +MONSDA is a config-driven wrapper: one JSON config describes samples, genomes +and tool choices; MONSDA compiles it into per-condition Snakemake or Nextflow +sub-workflows and can run them. Your job as the agent is to turn the user's +experiment description into a correct config, validate it cheaply before +spending compute, and only then run it — instead of the user hand-editing +JSON and guessing at tool names. + +Read the reference files below on demand (do not load all of them up front): + +- `reference/config-guide.md` — config.json anatomy, condition-tree, OPTIONS/TOOLS, samplesheet shortcut. Read before writing or editing any config. +- `reference/workflows.md` — available `WORKFLOWS` steps and the tools each one accepts. Read when choosing which steps/tools to enable. +- `reference/cli.md` — `monsda` / `monsda_configure` CLI flags, the generate-then-dry-run validation recipe, cluster profiles. Read before invoking any command. + +## Decision flow + +1. **Clarify the experiment, not the tool syntax.** Ask the user (or infer from + what they already said): experimental design/conditions, where the + FASTQ/BAM files and genome/annotation live, sequencing type (single/paired/ + single-cell, stranded?), and which analysis steps they want + (QC/TRIMMING/MAPPING/DEDUP/COUNTING/TRACKS/PEAKS/DE/DEU/DAS/DTU/CIRCS/ + FUSIONS). Don't ask about MONSDA internals they aren't expected to know. +2. **Map the design to a condition-tree** (`ID -> CONDITION -> SETTING`, + see `reference/config-guide.md`). This fixes the expected input directory + layout (`FASTQ///...`) and the output tool-key directories. +3. **Build the config.** Prefer, in order: + - If the user has (or can export) a sample table, use the built-in + `--samplesheet file.csv` support (`configs/samplesheet_template.csv` + shows the columns) instead of hand-writing the nested `SETTINGS` block. + - Otherwise copy the closest starting point — `configs/template_clean.json` + (empty skeleton) or `configs/tutorial_quick.json` (small worked example) + — and edit it. Several other files under `configs/` (`template.json`, + `tutorial_toolmix.json`, `tutorial_exhaustive.json`, + `tutorial_postprocess.json`) carry trailing `#comments` for human + documentation and are **not valid JSON**; read them for explanations but + don't copy them as-is — see `reference/config-guide.md` for the full list. + - Keep `WORKFLOWS` as a single comma-separated string in the **order** + steps should run, and give every listed step a corresponding top-level + key (`QC`, `MAPPING`, ...), each with a `TOOLS` map and per-condition + `OPTIONS` — MONSDA errors out otherwise. + - Set `VERSION` to the exact installed MONSDA version the run will use + (`python -c "import MONSDA; print(MONSDA.__version__)"` with the same + interpreter/env as the run) — a mismatch aborts the run. +4. **Validate before executing anything expensive.** Run + `scripts/validate_config.py ` (does structural/JSON checks and + cross-checks referenced sample/reference files) and then the + generate-only + dry-run recipe in `reference/cli.md` (`monsda --save` then + `snakemake -n` / nextflow equivalent on the generated sub-workflows). This + catches missing files, bad tool names and condition-tree mistakes without + running a single real job. +5. **Run it.** Only after a clean dry run, execute for real (local, or via a + cluster profile — see `reference/cli.md`). Report the actual command used + and where outputs land (tool-key directories under e.g. `MAPPING// + //`). + +## What this buys the user over doing it by hand + +- No need to memorize the condition-tree <-> directory-layout mapping or the + per-tool `OPTIONS` schema — you look it up in the reference files. +- `scripts/validate_config.py` catches config mistakes before a conda env + solve or a multi-hour mapping job starts. +- You generate-and-dry-run (`--save` + `snakemake/nextflow -n`) before + committing to a real run, which is the step people skip when working by + hand and MONSDA doesn't enforce. + +## Notes + +- `monsda_configure` is an interactive terminal UI meant for humans; it is + not practical to drive from an agent. Build/edit the JSON directly (or via + `--samplesheet`) instead of trying to script the TUI. +- If you are working inside a MONSDA git checkout rather than an installed + release, see this repo's `AGENTS.md` for the dev-only gotchas (symlinks + under `MONSDA/MONSDA`, PATH for `snakemake`/`nextflow`, golden-file tests). + None of that applies to an end user running an installed `monsda`. diff --git a/.agents/skills/monsda/reference/cli.md b/.agents/skills/monsda/reference/cli.md new file mode 100644 index 00000000..012a5951 --- /dev/null +++ b/.agents/skills/monsda/reference/cli.md @@ -0,0 +1,112 @@ +# CLI and validation recipe + +Full docs: `docs/source/runsmk.rst`, `docs/source/cluster.rst`. This covers +what an agent needs to generate, dry-run, and then execute a config safely. + +## Environment + +```bash +conda activate monsda # or whatever env MONSDA was installed into +``` + +`monsda` (Snakemake backend, default) and `monsda --nextflow` both need +`snakemake`/`nextflow` on PATH respectively; MONSDA aborts immediately if the +selected engine's binary is missing. + +## `monsda` flags that matter for an agent + +``` +monsda -c CONFIG.json -d DIRECTORY -j PROCS [options] +``` + +- `-c/--config/--configfile CONFIG [CONFIG ...]`: one JSON, or JSON + an + extra Nextflow-specific config. +- `-d/--directory`: the run/project root (where `FASTQ/`, `GENOMES/`, + `SubSnakes|SubFlows/`, `JOBS/`, `LOGS/` live or will be created). +- `-j/--procs`: parallelism, capped by the config's `MAXTHREADS`. +- `--save`: **generate only** — write the sub-workflow files and the CLI + calls to `JOBS/` without executing anything. This is the key step for + validating a config cheaply. +- `-s/--skeleton`: just create the minimal directory hierarchy, no config + processing. +- `--snakemake` (default) / `--nextflow`: pick the execution engine. +- `-u/--use-conda`: use conda envs (default true). +- `-l/--unlock`: unlock a Snakemake working directory after an interrupted run. +- `--clean`: Nextflow work-dir cleanup (`-n` to preview, `-f` to actually delete). +- `--loglevel {WARNING,ERROR,INFO,DEBUG}`. +- `--samplesheet FILE`: populate `SETTINGS` from a CSV/TSV when the config + lacks one (see `reference/config-guide.md`); writes + `_with_settings.json` on first use. +- `--oras-registry HOST` / `--oras-namespace NAMESPACE`: override the + container registry/namespace for ORAS image pulls (defaults + `ghcr.io` / `jfallmann/monsda`). +- `--version`: print and exit. + +Additional unrecognized arguments are passed through to Snakemake/Nextflow +verbatim, e.g. `-n` (Snakemake dry-run), `--omit-from ` (skip a rule and +everything downstream without removing it from `WORKFLOWS`), `--profile +` (cluster profile). + +## Validate-before-run recipe (do this before a real execution) + +1. Run the repo's lightweight structural check first: + `python scripts/validate_config.py CONFIG.json` (see this skill's + `scripts/` directory) — catches invalid JSON, missing step blocks, a + `VERSION` mismatch, and `SETTINGS`/`REFERENCE`/`ANNOTATION` paths that + don't exist on disk, before anything talks to Snakemake/Nextflow. +2. Generate without executing: + ```bash + monsda -c CONFIG.json -d DIRECTORY -j 2 --save + ``` + This writes the per-condition sub-workflows (`SubSnakes/*.smk` or + `SubFlows/*.nf`) and the exact CLI invocations MONSDA would run into + `JOBS/MONSDA.commands`. +3. Dry-run every generated sub-workflow using the commands recorded in + `JOBS/MONSDA.commands`, adding `-n` (Snakemake) — e.g.: + ```bash + snakemake --snakefile SubSnakes/.smk --configfile .json -n + ``` + For Nextflow, `nextflow run .nf -preview` only validates syntax, + not the script bodies — a real `-n`-equivalent dry run is not available, + so inspect the generated `.nf`/its `params` instead, or do a tiny real run + on a minimal subset of samples first. +4. Fix whatever the dry run reports (commonly `MissingInputException` for + files that don't exist yet, or a bad tool/env name) and repeat from step 2 + before launching the full, real run. + +## Running for real + +```bash +monsda -c CONFIG.json -d DIRECTORY -j THREADS --conda-frontend mamba \ + --conda-prefix PATH_TO_CONDA_ENVS +``` + +or, for Nextflow: + +```bash +monsda --nextflow -c CONFIG.json -d DIRECTORY -j THREADS +``` + +### On a cluster (SLURM example) + +```bash +monsda -c CONFIG.json -d DIRECTORY -j THREADS --conda-frontend mamba \ + --profile profile_snakemake --conda-prefix PATH_TO_CONDA_ENVS +``` + +```bash +export NXF_EXECUTOR=slurm +monsda --nextflow -c CONFIG.json -d DIRECTORY -j THREADS +``` + +`profile_snakemake/` and `profile_nextflow/` in this repo ship example +profiles to adapt. + +### Resuming a run while skipping a step + +Keep the full original `WORKFLOWS` list (see `reference/config-guide.md`), +and pass Snakemake's own skip flag instead of editing the config: + +```bash +monsda -c CONFIG.json -d DIRECTORY -j THREADS --omit-from multiqc +``` diff --git a/.agents/skills/monsda/reference/config-guide.md b/.agents/skills/monsda/reference/config-guide.md new file mode 100644 index 00000000..ed823c83 --- /dev/null +++ b/.agents/skills/monsda/reference/config-guide.md @@ -0,0 +1,169 @@ +# config.json anatomy + +Full human docs: `docs/source/config.rst`, `docs/source/conditiontree.rst`, +`docs/source/preparation.rst`. This is the condensed, agent-facing version. + +## Top level + +```json +{ + "WORKFLOWS": "QC,TRIMMING,MAPPING,DEDUP", + "BINS": "", + "MAXTHREADS": "20", + "VERSION": "1.5.0", + "SETTINGS": { ... }, + "QC": { ... }, + "TRIMMING": { ... }, + "MAPPING": { ... }, + "DEDUP": { ... } +} +``` + +- `WORKFLOWS`: comma-separated step names, in run order. Every step listed + here needs a matching top-level key with its own settings, or MONSDA errors. + Available steps: see `reference/workflows.md`. +- `VERSION` must equal the installed `MONSDA.__version__` for the interpreter + that will run the job, or the run aborts. +- `MAXTHREADS` caps `-j` passed on the CLI. +- `BINS`: path to custom user scripts; advanced/rare, leave `""` otherwise. + +### Rerunning on an existing output directory + +Keep the **full, original** `WORKFLOWS` list even if you only want to rerun a +later step (e.g. keep `QC,TRIMMING,MAPPING,DEDUP` even if only `DE` changed). +The tool-key/`combo` directory name is derived from which stages are listed; +dropping an earlier stage changes that name and makes existing outputs look +missing, forcing a full rerun. To skip actually re-executing one rule, add +Snakemake's own `--omit-from ` on the command line instead of editing +`WORKFLOWS`. + +## The condition-tree (`SETTINGS`) + +``` +'ID' -> 'CONDITION' -> 'SETTING' (optional third level) +``` + +```json +"SETTINGS": { + "LabA": { + "wild-type": { + "SAMPLES": ["Sample_1", "Sample_2"], + "SEQUENCING": "paired", + "REFERENCE": "GENOMES/Dm6/dm6.fa.gz", + "ANNOTATION": { + "GTF": "GENOMES/Dm6/dm6.gtf.gz", + "GFF": "GENOMES/Dm6/dm6.gff3.gz" + } + } + } +} +``` + +- Each leaf of the tree is one independent analysis unit. The tree also + dictates the **input** directory layout: `FASTQ///...` + (MONSDA looks one level above the deepest leaf for input, so adding a + `SETTING` sub-level for e.g. benchmarking different tool params does not + require duplicating the FASTQ files). +- `SAMPLES`: list of sample basenames, no file extension. For paired-end, + list the name once without `_R1`/`_R2` — MONSDA appends those + automatically and requires files to end in `_R1.fastq.gz`/`_R2.fastq.gz` + (single-end: `.fastq.gz`). +- `SEQUENCING`: `single`, `paired`, or `singlecell`; optionally append + comma-separated strandedness (`rf` or `fr`, see RSeQC `infer_experiment.py` + convention) e.g. `"paired,fr"`. Leave strandedness empty if unstranded — + MONSDA configures per-tool strandedness flags automatically from this. +- `REFERENCE` / `ANNOTATION.GTF` / `ANNOTATION.GFF` / `INDEX` / `PREFIX` / + `DECOY`: per-condition defaults, can be overridden per workflow step. +- `GROUPS` / `BATCHES` / `TYPES`: optional, same length as `SAMPLES`, used + only by DE/DEU/DAS/DTU design matrices (batch correction, type covariate). +- `IP`: only for `PEAKS` with CLIP-type protocols (`CLIP`, `iCLIP`, `revCLIP`). + +Output directories are named by a derived **tool-key/combo**, e.g. +`MAPPING/LabA/wild-type/fastqc-cutadapt-star-umitools/`, built from the +enabled workflow steps + tools so different tool combinations never collide. + +## Per-workflow-step blocks + +Each step listed in `WORKFLOWS` needs: + +```json +"MAPPING": { + "TOOLS": { "star": "STAR" }, + "LabA": { + "wild-type": { + "ENV": "star", + "BIN": "STAR", + "star": { + "OPTIONS": { + "INDEX": "--some-star-index-flag value", + "MAP": "--some-star-mapping-flag value" + } + } + } + } +} +``` + +- `TOOLS`: map of `conda-env-name: executable-name` for every tool used in + this step (env name usually equals the bioconda package, see `envs/`). +- `ENV`/`BIN` at the condition level override the step-level default (useful + to benchmark two installs of the same tool, or point at a local binary for + tools without a conda package, e.g. `guppy`, `dorado`). +- Inside each tool's block, `OPTIONS` holds one key per sub-step (e.g. + `INDEX` vs `MAP` for a mapper) with a **raw command-line string** of extra + flags for that tool — exactly as you'd type them, minus anything MONSDA + derives automatically (paired/stranded flags, threads). Leave `OPTIONS: {}` + if no extra flags are needed; never invent a flag that isn't in the tool's + own `--help`. +- `COMPARABLE` (DE/DEU/DAS/DTU and MACS peak calling): if omitted/empty, + MONSDA generates all-vs-all comparisons from `GROUPS`. Otherwise: + ```json + "COMPARABLE": { "comparison-name": ["Group1", "Group2"] } + ``` + +## Fastest path: samplesheet instead of hand-written SETTINGS + +If the user can provide a CSV/TSV, skip writing `SETTINGS` by hand. Columns +(see `configs/samplesheet_template.csv`): + +``` +CONDITION,SAMPLE,GROUP,SEQUENCING,REFERENCE,GTF,GFF,INDEX,PREFIX,DECOY,TYPE,BATCH,IP +``` + +`CONDITION` is a `/`-separated path that becomes the condition-tree branch +(e.g. `FGUMI/WT/dummylevel`). Rows sharing a `CONDITION` are grouped into one +leaf's `SAMPLES` list; only the first row of a group needs the +REFERENCE/GTF/GFF/etc. columns filled in. Pass it to `monsda` directly with +`--samplesheet file.csv` (it populates `SETTINGS` in the given config JSON on +the first run and writes `_with_settings.json` for reuse), or let +`monsda_configure --samplesheet file.csv` do it when scaffolding a brand-new +project. + +## PostDE (optional enrichment on top of DE) + +Only for the `DE` workflow with `deseq2`/`edger` (not DEU/DAS/DTU). Enabled +via a top-level `POSTDE` key pointing at a separate analysis JSON (gProfiler/ +clusterProfiler/GSVA/decoupleR/dream). See `docs/source/config.rst` section +"PostDE" and the shipped example `configs/postde_analysis.json` before +building one — it has its own schema and conda env (`postde`). + +## Starting points in this repo + +Valid JSON, safe to copy/parse directly: + +- `configs/template_clean.json` — empty skeleton with every workflow step's + key already scaffolded. +- `configs/tutorial_quick.json` — small worked example (FETCH+MAPPING on the + bundled `tests/data` Ecoli set). +- `configs/template_umicollapse.json` — UMI-aware dedup example. +- `configs/postde_analysis.json` — example PostDE analysis JSON. + +**Documentation-only, NOT valid JSON** — these have trailing `#comment` +annotations after values and will fail `json.load`; read them for the +explanations but copy structure from the valid files above instead: +`configs/template.json`, `configs/template_base_commented.json`, +`configs/tutorial_toolmix.json`, `configs/tutorial_exhaustive.json`, +`configs/tutorial_postprocess.json`. Strip the `#...` trailing comments +before using one of these as a base, or diff it against +`configs/template_clean.json` to see the annotation-free version of the same +keys. diff --git a/.agents/skills/monsda/reference/workflows.md b/.agents/skills/monsda/reference/workflows.md new file mode 100644 index 00000000..4a44ee7e --- /dev/null +++ b/.agents/skills/monsda/reference/workflows.md @@ -0,0 +1,39 @@ +# Workflow steps and tools + +Full tables with descriptions/links: `docs/source/workflows.rst`. Condensed +reference below: `WORKFLOWS` step name -> `conda-env: executable` pairs you +can put under that step's `TOOLS` key. Env names match a `.yaml` in `envs/`. + +| Step | Purpose | `TOOLS` env : bin examples | +|---|---|---| +| `FETCH` | Download from SRA | `sra: fasterq-dump` | +| `BASECALL` | FAST5/POD5 -> FASTQ | `guppy: $PATH_TO_LOCAL_BIN` (no conda pkg), `dorado: $PATH_TO_LOCAL_BIN` (no conda pkg) | +| `QC` | QC of FASTQ/BAM (runs standalone pre-processing, or again post-processing for any later step) | `fastqc: fastqc` (FASTQ/BAM, includes MultiQC), `rustqc: rustqc` (BAM only, includes MultiQC, has a strandedness guard — aborts the run if inferred strandedness conflicts with `SEQUENCING`) | +| `TRIMMING` | Adapter/quality trimming | `trimgalore: trim_galore`, `cutadapt: cutadapt`, `bbduk: bbduk`, `fastp: fastp` | +| `MAPPING` | Align to reference | `hisat2: hisat2`, `star: STAR` (also used for `STARsolo` single-cell mode), `rustar: rustar` (Rust STAR reimplementation), `segemehl2`/`segemehl3`/`segemehl2bisulfite`/`segemehl3bisulfite: segemehl.x`, `bwa: bwa mem`, `bwa2: bwa2-mem`, `bwameth: bwameth.py`, `minimap: minimap2` (long-read, use `-x map-ont`/`-x map-pb` in OPTIONS), `salmonalign: salmon` (alignment mode), `rammap: rammap` (long-read, not on bioconda), `piscem: piscem` (single-cell only, emits RAD for `alevinfry`) | +| `DEDUP` | Remove/mark duplicates | `umitools: umi_tools`, `fgumi: fgumi` (UMI extraction + dedup), `picard: picard` (position-based) | +| `COUNTING` | Quantify reads/transcripts | `countreads: featureCounts`, `salmon: salmon` (FASTQ or, via `OPTIONS.BAM = "transcriptome"\|"genome"`, directly from BAM), `kallisto: kallisto`, `oarfish: oarfish` (long-read, from transcriptome BAM), `alevinfry: alevin-fry` (needs a piscem RAD file), `simpleaf: simpleaf` (self-contained single-cell index+map+quant) | +| `TRACKS` | trackdb.txt + BIGWIG for UCSC/genome browsers | `ucsc: ucsc` | +| `PEAKS` | Peak calling (ChIP/RIP/CLIP) | `piranha: piranha` (CLIP/RIP), `macs: macs` (ChIP TF binding; use `COMPARABLE` for signal-vs-background pairs), `scyphy: piranha` (cyPhyRNA-Seq), `peaks: peaks` (quick/unpublished window scan) | +| `DE` | Differential gene expression | `deseq2`, `edger` (R scripts under `Analysis/DE/`) | +| `DEU` | Differential exon usage | `dexseq`, `edger` | +| `DAS` | Differential alternative splicing | `edger`, `diego` | +| `DTU` | Differential transcript usage | `dexseq`, `drimseq`, `spit` (all three accept an optional `TERMINUS` option to pre-group transcripts) | +| `CIRCS` | circRNA detection | `ciri2: $Path_to_CIRI2.pl` (install locally, no conda pkg) | +| `FUSIONS` | Fusion gene detection | `starfusion: STAR-Fusion` — needs `MAPPING` with `star` and chimeric output enabled (`--chimSegmentMin 12 --chimOutType Junctions` in the STAR `MAP` OPTIONS), or set `FUSIONS.OPTIONS.FASTQ = "true"` to run directly on trimmed FASTQ instead. Needs a CTAT genome resource dir in `INDEX` (auto-built from `REFERENCE`+`ANNOTATION` if missing — slow). | + +Notes: + +- QC runs automatically again after any processing step if enabled, in + addition to the optional pre-processing QC pass. +- Single-cell: `piscem` (MAPPING) + `alevinfry` (COUNTING), or the + self-contained `simpleaf` (COUNTING only); generic post-mapping QC/DEDUP is + skipped for the RAD-file path. +- Long-read (Nanopore/PacBio): `minimap`/`rammap` (MAPPING) + `oarfish` + (COUNTING, from transcriptome BAM). +- Bisulfite: `bwameth` or `segemehl{2,3}bisulfite` for MAPPING. +- Tools marked "no conda pkg" need `ENV`/`BIN` pointing at a locally + installed executable instead of a bioconda environment name. +- `PostDE` (gProfiler/clusterProfiler/GSVA/decoupleR/dream enrichment on top + of `DE` with `deseq2`/`edger`) is a separate top-level `POSTDE` config, not + a `WORKFLOWS` entry — see `reference/config-guide.md`. diff --git a/.agents/skills/monsda/scripts/validate_config.py b/.agents/skills/monsda/scripts/validate_config.py new file mode 100644 index 00000000..16c797ff --- /dev/null +++ b/.agents/skills/monsda/scripts/validate_config.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 +"""Cheap, offline sanity checks for a MONSDA config.json. + +This does NOT replace MONSDA's own validation (``monsda --save`` + a +Snakemake/Nextflow dry run, see reference/cli.md) — it only catches obvious +mistakes (bad JSON, missing step blocks, a stale VERSION, sample/reference +files that don't exist) before you spend time on a real generate/run. + +Usage: + python validate_config.py CONFIG.json [--base DIR] + +``--base`` is the run/project directory paths in the config are resolved +against (defaults to the config file's own directory). +""" +import argparse +import json +import os +import sys + +KNOWN_WORKFLOW_STEPS = { + "FETCH", "BASECALL", "QC", "TRIMMING", "MAPPING", "DEDUP", "COUNTING", + "TRACKS", "PEAKS", "DE", "DEU", "DAS", "DTU", "CIRCS", "FUSIONS", +} + + +def fail(msg, errors): + errors.append(msg) + + +def walk_settings_leaves(node, path=()): + """Yield (path, leaf_dict) for every SETTINGS leaf that has a SAMPLES key.""" + if not isinstance(node, dict): + return + if "SAMPLES" in node: + yield path, node + return + for key, val in node.items(): + if isinstance(val, dict): + yield from walk_settings_leaves(val, path + (key,)) + + +def check_installed_version(declared_version, errors, warnings): + try: + import MONSDA + installed = getattr(MONSDA, "__version__", None) + except Exception: + warnings.append( + "Could not import MONSDA to check VERSION against the installed " + f"release (declared VERSION={declared_version!r}); verify manually " + "with: python -c \"import MONSDA; print(MONSDA.__version__)\"" + ) + return + if installed and declared_version != installed: + fail( + f"VERSION in config ({declared_version!r}) does not match the " + f"installed MONSDA ({installed!r}). MONSDA refuses to run on a " + "mismatch.", + errors, + ) + + +def resolve(base, path): + if not path: + return None + expanded = os.path.expandvars(os.path.expanduser(path)) + return expanded if os.path.isabs(expanded) else os.path.join(base, expanded) + + +def check_leaf_paths(leaf, base, label, errors, warnings): + for key in ("REFERENCE", "INDEX"): + val = leaf.get(key) + if isinstance(val, str) and val: + full = resolve(base, val) + if not os.path.exists(full): + warnings.append(f"{label}: {key} path not found: {val}") + ann = leaf.get("ANNOTATION") + if isinstance(ann, dict): + for key, val in ann.items(): + if isinstance(val, str) and val: + full = resolve(base, val) + if not os.path.exists(full): + warnings.append(f"{label}: ANNOTATION.{key} path not found: {val}") + samples = leaf.get("SAMPLES") + if not isinstance(samples, list) or not samples: + fail(f"{label}: SAMPLES is missing or empty", errors) + for extra_key in ("GROUPS", "BATCHES", "TYPES"): + extra = leaf.get(extra_key) + if extra is not None and isinstance(samples, list) and len(extra) != len(samples): + fail( + f"{label}: {extra_key} has {len(extra)} entries but SAMPLES " + f"has {len(samples)}", + errors, + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("config", help="Path to the MONSDA config JSON") + parser.add_argument( + "--base", default=None, + help="Directory to resolve relative paths against (default: config's own directory)", + ) + args = parser.parse_args() + + errors, warnings = [], [] + + try: + with open(args.config) as fh: + config = json.load(fh) + except json.JSONDecodeError as exc: + print(f"ERROR: {args.config} is not valid JSON: {exc}", file=sys.stderr) + print( + "Hint: configs/template.json has trailing '#comments' for " + "documentation and is NOT valid JSON on its own; start from " + "configs/template_clean.json instead.", + file=sys.stderr, + ) + sys.exit(2) + except OSError as exc: + print(f"ERROR: could not read {args.config}: {exc}", file=sys.stderr) + sys.exit(2) + + base = args.base or os.path.dirname(os.path.abspath(args.config)) or "." + + for required in ("WORKFLOWS", "VERSION", "SETTINGS"): + if required not in config: + fail(f"Missing top-level key: {required}", errors) + + workflows_raw = config.get("WORKFLOWS", "") + steps = [s.strip() for s in str(workflows_raw).split(",") if s.strip()] + if not steps: + fail("WORKFLOWS is empty: no workflow step selected", errors) + for step in steps: + if step not in KNOWN_WORKFLOW_STEPS: + warnings.append( + f"WORKFLOWS lists {step!r}, which is not in the known step " + f"list ({sorted(KNOWN_WORKFLOW_STEPS)}); check reference/workflows.md" + ) + elif step not in config: + fail( + f"WORKFLOWS includes {step!r} but there is no top-level " + f"{step!r} block with its TOOLS/OPTIONS", + errors, + ) + + version = config.get("VERSION") + if version is not None: + check_installed_version(version, errors, warnings) + + settings = config.get("SETTINGS") + if isinstance(settings, dict): + leaves = list(walk_settings_leaves(settings)) + if not leaves: + fail("SETTINGS has no condition-tree leaf with a SAMPLES list", errors) + for path, leaf in leaves: + label = "SETTINGS." + ".".join(path) if path else "SETTINGS" + check_leaf_paths(leaf, base, label, errors, warnings) + + print(f"Checked {args.config} (base dir: {base})") + if warnings: + print(f"\n{len(warnings)} warning(s):") + for w in warnings: + print(f" - {w}") + if errors: + print(f"\n{len(errors)} error(s):") + for e in errors: + print(f" - {e}") + sys.exit(1) + print("\nNo structural errors found. This does not replace a real " + "`monsda --save` + dry-run (see reference/cli.md).") + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/monsda b/.claude/skills/monsda new file mode 120000 index 00000000..f0e7e4f8 --- /dev/null +++ b/.claude/skills/monsda @@ -0,0 +1 @@ +../../.agents/skills/monsda \ No newline at end of file diff --git a/.github/workflows/build-containers.yml b/.github/workflows/build-containers.yml new file mode 100644 index 00000000..a9452d17 --- /dev/null +++ b/.github/workflows/build-containers.yml @@ -0,0 +1,86 @@ +name: Build MONSDA containers + +on: + pull_request_target: + types: [closed] + release: + types: [published] + workflow_dispatch: + +permissions: + contents: read + packages: write + +concurrency: + group: containers-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: false + +jobs: + prepare: + if: github.event_name != 'pull_request_target' || github.event.pull_request.merged == true + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.matrix.outputs.matrix }} + version: ${{ steps.version.outputs.version }} + revision: ${{ steps.revision.outputs.revision }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.merge_commit_sha || github.event_name == 'release' && github.event.release.tag_name || github.sha }} + - id: matrix + run: echo "matrix=$(python scripts/container_matrix.py)" >> "$GITHUB_OUTPUT" + - id: version + run: python scripts/container_matrix.py --version >> "$GITHUB_OUTPUT" + - id: revision + run: echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + + build: + needs: prepare + runs-on: ubuntu-latest + strategy: + fail-fast: false + max-parallel: 8 + matrix: ${{ fromJSON(needs.prepare.outputs.matrix) }} + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ needs.prepare.outputs.revision }} + - uses: docker/setup-buildx-action@v3 + - uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - uses: docker/build-push-action@v6 + with: + context: . + file: docker/Dockerfile + push: true + build-args: | + ENVIRONMENT=${{ matrix.environment }} + MONSDA_VERSION=${{ needs.prepare.outputs.version }} + VCS_REF=${{ needs.prepare.outputs.revision }} + tags: ghcr.io/jfallmann/monsda:${{ matrix.environment }}-${{ needs.prepare.outputs.version }} + cache-from: type=gha,scope=${{ matrix.environment }} + cache-to: type=gha,mode=max,scope=${{ matrix.environment }} + + verify: + needs: [prepare, build] + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ needs.prepare.outputs.revision }} + - uses: docker/setup-buildx-action@v3 + - uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - env: + VERSION: ${{ needs.prepare.outputs.version }} + run: | + for environment in $(python scripts/container_matrix.py | python -c 'import json,sys; print(" ".join(json.load(sys.stdin)["environment"]))'); do + docker buildx imagetools inspect "ghcr.io/jfallmann/monsda:${environment}-${VERSION}" >/dev/null + done diff --git a/.github/workflows/python-app.yaml b/.github/workflows/python-app.yaml index 3ba2edd8..33fadca7 100644 --- a/.github/workflows/python-app.yaml +++ b/.github/workflows/python-app.yaml @@ -1,6 +1,10 @@ name: MONSDA CI on: + push: + branches: [main, develop] + tags: + - "v*" pull_request: types: [opened, synchronize, reopened, labeled, unlabeled] workflow_dispatch: @@ -11,10 +15,35 @@ on: default: false type: boolean +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: + lint: + name: Lint (flake8, isort) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + - name: Install linting tools + run: | + python -m pip install --upgrade pip + python -m pip install flake8 isort + - name: Check import sorting with isort + run: isort --check-only --diff MONSDA/ tests/ + continue-on-error: true + - name: Lint with flake8 + run: flake8 MONSDA/ tests/ --count --statistics --show-source + continue-on-error: true + tests: name: Unit tests (pytest) runs-on: ubuntu-latest + needs: lint defaults: run: shell: bash -el {0} @@ -34,16 +63,58 @@ jobs: show-channel-urls: true use-only-tar-bz2: false auto-activate-base: false - - name: build and install MONSDA + - name: Install dependencies run: | cd ${{ github.workspace }} python -m pip install --upgrade pip python -m pip install -e . - python -m pip install pytest - - name: run unit tests + python -m pip install pytest pytest-cov + - name: Run unit tests with coverage run: | cd ${{ github.workspace }} - pytest -q tests/test_Utils.py + pytest tests/ -v --cov=MONSDA --cov-report=xml --cov-report=html --cov-fail-under=25 + - name: Upload coverage reports + uses: codecov/codecov-action@v4 + if: always() + with: + files: ./coverage.xml + fail_ci_if_error: false + verbose: true + + build: + name: Build package + runs-on: ubuntu-latest + needs: tests + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + - name: Install build tools + run: | + python -m pip install --upgrade pip + python -m pip install build wheel setuptools versioneer + - name: Sync config versions + run: | + python scripts/update_config_versions.py configs/*.json tests/data/config_*.json + - name: Build distribution + run: | + python -m build + - name: Check build artifacts + run: | + ls -lh dist/ + python -m pip install twine + twine check dist/* + - name: Upload build artifacts + uses: actions/upload-artifact@v4 + with: + name: python-package-distributions + path: dist/ + retention-days: 7 full-pipeline: name: Full pipeline integration test @@ -59,6 +130,7 @@ jobs: - uses: actions/checkout@v4 with: submodules: recursive + fetch-depth: 0 - uses: conda-incubator/setup-miniconda@v3 with: miniconda-version: "latest" @@ -71,23 +143,22 @@ jobs: show-channel-urls: true use-only-tar-bz2: false auto-activate-base: false - - name: build and install MONSDA + - name: Build and install MONSDA run: | cd ${{ github.workspace }} python -m pip install --upgrade pip python -m pip install -e . - - name: prepare integration test config + - name: Prepare integration test config run: | cd ${{ github.workspace }} - VERSION=$(monsda --version 2>&1 | sed 's/MONSDA version //g') - sed -i "s/\"VERSION\": \"FIXME\"/\"VERSION\": \"${VERSION}\"/g" tests/data/config_Test.json - - name: run full pipeline test + python scripts/update_config_versions.py tests/data/config_Test.json + - name: Run full pipeline test run: | cd ${{ github.workspace }}/tests ln -fs data/* . mkdir -p CONDALIB monsda -j 6 -c config_Test.json --directory ${PWD} --use-conda --conda-prefix CONDALIB --save - - name: upload test logs on failure + - name: Upload test logs on failure if: failure() uses: actions/upload-artifact@v4 with: @@ -100,4 +171,25 @@ jobs: ${{ github.workspace }}/tests/QC ${{ github.workspace }}/tests/COUNTING ${{ github.workspace }}/tests/PEAKS - ${{ github.workspace }}/tests/TRACKS \ No newline at end of file + ${{ github.workspace }}/tests/TRACKS + + publish: + name: Publish to PyPI + runs-on: ubuntu-latest + needs: build + if: startsWith(github.ref, 'refs/tags/') + environment: + name: pypi + url: https://pypi.org/project/MONSDA + permissions: + id-token: write + steps: + - name: Download build artifacts + uses: actions/download-artifact@v4 + with: + name: python-package-distributions + path: dist/ + - name: Publish to PyPI + uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://upload.pypi.org/legacy/ diff --git a/.gitignore b/.gitignore index d1f3fcb2..e4bbd90a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,11 @@ #temps *~ \#* + +# generated tool-audit run reports (tool_manifest.yaml and the *.txt dumps +# under tests/tool_audit/ ARE tracked and reviewed, this is not) +tests/tool_audit/*.json +tests/tool_audit/.last_report.json # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] @@ -108,3 +113,6 @@ venv.bak/ # singularity *.sif + +# AI +.abacusai/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 00000000..03f766a0 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,36 @@ +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.6.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-added-large-files + args: ['--maxkb=1000'] + - id: check-merge-conflict + - id: check-json + - id: mixed-line-ending + + - repo: https://github.com/PyCQA/isort + rev: 5.13.2 + hooks: + - id: isort + name: isort + language_version: python3.12 + + - repo: https://github.com/PyCQA/flake8 + rev: 7.1.1 + hooks: + - id: flake8 + name: flake8 + language_version: python3.12 + + - repo: https://github.com/pre-commit/mirrors-mypy + rev: v1.11.1 + hooks: + - id: mypy + name: mypy + language_version: python3.12 + additional_dependencies: ['types-PyYAML', 'types-setuptools'] + args: ['--ignore-missing-imports', '--show-error-codes'] + stages: [manual] diff --git a/.vscode/settings.json b/.vscode/settings.json index b3c483e6..b04e345e 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -28,5 +28,6 @@ "[python]": { "editor.defaultFormatter": "ms-python.black-formatter" }, - "python.formatting.provider": "none" + "python.formatting.provider": "none", + "makefile.configureOnOpen": false } diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..2876616e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,90 @@ +# Notes for AI agents + +## Keep the user updated + +Report progress **while** working, not only at the end. Before/while running a +step, say what is being done and why, and show the relevant command output or +diff. A silent stretch of work followed by one large summary makes the change +impossible to judge and is not acceptable. + +State clearly what was actually verified and what was not. + +## Test environment + +`snakemake` and `nextflow` live in the `monsda` conda env, they are *not* on the +default PATH: + + export PATH=/home/fall/anaconda3/envs/monsda/bin:$PATH + +Confirmed there: snakemake 9.20.0, nextflow 26.04.0.12031. `multiqc` is **not** +installed in that env, MultiQC behaviour cannot be verified locally. + +That env contains an installed copy of MONSDA in `site-packages`, *not* an +editable install. It only picks up the working tree when the cwd is the repo, so +runs from anywhere else need `PYTHONPATH=` or they silently test the +installed release instead of the change. + +MONSDA aborts early if `nextflow` is missing, so `--nextflow` runs need that +PATH. + +### Gotchas when running MONSDA from a checkout + +- `envpath`/`workflowpath` resolve to `/MONSDA/MONSDA/{envs,workflows}`, + which does not exist in a checkout. Create the symlinks before running and + remove them afterwards, they must not be committed: + + mkdir -p MONSDA/MONSDA + ln -sfn ../../workflows MONSDA/MONSDA/workflows + ln -sfn ../../envs MONSDA/MONSDA/envs + ln -sfn ../../scripts MONSDA/MONSDA/scripts + +- `VERSION` in the config has to match the installed version exactly, otherwise + MONSDA refuses to run. Get it from `MONSDA.__version__` of the interpreter + used for the run, it differs between envs. + +- Generating without executing: run from the target directory with + `python -m MONSDA.RunMONSDA -c -d -j 2 --save`. The declared + console script is `monsda`, not `MONSDA`, and it is not on PATH in the + `monsda` env, so use the module form and set `PYTHONPATH=` if the + interpreter does not already import MONSDA from the tree. Then + dry run every generated workflow via the calls in `JOBS/MONSDA.commands` + (`snakemake ... -n`). Note that `tests/data` ships no BAMs, so most sub + workflows fail with `MissingInputException` even on an unmodified tree, + compare the failure set against a pristine worktree instead of expecting zero + failures. + +- Workflow templates are snapshot tested. On intended changes regenerate with + `UPDATE_GOLDEN=1 python -m pytest` in `tests/` and check the golden diff only + contains the intended lines. + +- `${HOME}/MONSDA` is a symlink to this repo, the same tree under a second path. + Editor diagnostics may report paths there, they refer to these files. + +## Test data and configs + +`${HOME}/Work2/Tests/MONSDATest` is the user's test directory. It holds real +input data (`FASTQ/{Ecoli,SIMPLE,FGUMI}`, `GENOMES/Ecoli` incl. prebuilt +indices, 122 BAMs in `MAPPED/`), their own configs at the top level and outputs +of earlier runs. Use it instead of `tests/data`, which ships no BAMs. It is not +a git repo, so nothing there is protected by version control. + +Put configs written for a test into `TESTCONFIGS/` so they can be reused: + +- `config_multiqc_fastqc.json`, `config_multiqc_rustqc.json`: QC, TRIMMING, + DEDUP, MAPPING on `Ecoli/KO` with a `MULTI` option set. The rustqc one selects + the `multiqc_rustqc` template, the fastqc one `multiqc`. +- `monsda_gen.sh [--nextflow]`: generates only (`--save`). + Creates the `MONSDA/MONSDA` symlinks, stamps `VERSION` with the tree's + `MONSDA.__version__` and links `FASTQ`/`GENOMES` into a sandbox `outdir`, so + the outputs in the test root are not overwritten. MONSDA resolves input data + relative to its working directory, hence the links. +- `mqc_script_test.sh [combo] [condition] [params]`: runs + the generated nextflow `mqc` process with a fake `multiqc` that records its + argv, and checks the module list from `LOGS/versions.txt`, the scanned + directories and the published files. `nextflow lint` and `-preview` do not + render script bodies, so this is the only local check of that bash. +- `mqc_smk_test.py [combo] [condition]`: same idea for + the snakemake side, formats the generated `multiqc` rule's `shell:` string and + runs it against a fake `multiqc`. + +Do not run tests in the test root itself, generate into a subdirectory. diff --git a/CICD.md b/CICD.md new file mode 100644 index 00000000..1c3efdb7 --- /dev/null +++ b/CICD.md @@ -0,0 +1,354 @@ +# CI/CD Pipeline for MONSDA + +This document describes the Continuous Integration/Continuous Deployment (CI/CD) pipeline for MONSDA. + +## Overview + +The CI/CD pipeline is implemented using GitHub Actions and includes the following stages: + +1. **Linting** - Code quality checks with flake8 and isort +2. **Testing** - Unit tests with pytest and coverage reporting +3. **Building** - Package building and artifact verification +4. **Integration Testing** - Full pipeline integration tests (optional) +5. **Publishing** - Automatic deployment to PyPI on releases + +## Pipeline Stages + +### 1. Linting (Runs on all PRs and pushes) + +**Job:** `lint` + +Automatically checks code quality using: +- **isort** - Validates import statement ordering +- **flake8** - Checks PEP 8 compliance and code style + +**Configuration:** +- Line length: 110 characters +- Ignores: E203, E266, E501, W503 +- Continues on error (warnings don't block) + +Run locally: +```bash +make lint-check # Check without fixing +make lint # Check and fix issues +make format # Format imports only +``` + +### 2. Unit Testing (Runs after linting) + +**Job:** `tests` + +Runs pytest test suite with coverage reporting: +- Tests located in `tests/test_Utils.py` +- Generates coverage reports (XML and HTML) +- Uploads coverage to Codecov + +**Dependencies:** +- Conda environment from `environment.yml` +- Python 3.12.2 +- pytest and pytest-cov + +Run locally: +```bash +make test # Run unit tests +make test-cov # Run with coverage report +``` + +### 3. Building (Runs after tests) + +**Job:** `build` + +Builds distribution packages: +- Creates wheel and source distributions +- Validates build artifacts with twine +- Stores artifacts for 7 days + +**Tools:** +- setuptools +- wheel +- versioneer (for version management) +- twine (for package validation) + +Run locally: +```bash +make build # Build distribution packages +make clean-build # Remove build artifacts +``` + +### 4. Full Pipeline Integration Test (Optional) + +**Job:** `full-pipeline` + +Runs comprehensive integration tests of the Snakemake/Nextflow pipeline functionality. + +**Trigger conditions:** +- Manual workflow dispatch with `run_full_pipeline=true` +- PR with `full-pipeline` label + +**Process:** +1. Sets up Conda environment from `environment.yml` +2. Installs MONSDA package +3. Updates test configuration with current version +4. Runs test workflow with sample data +5. Uploads debug artifacts on failure + +Run locally: +```bash +RUN_INTEGRATION_TESTS=1 bash tests/cicd_test.sh +``` + +### 5. Publishing to PyPI (Runs on version tags) + +**Job:** `publish` + +Automatically publishes releases to PyPI when tags are pushed. + +**Trigger:** Tags matching pattern `v*` (e.g., `v1.0.0`) + +**Requirements:** +- Build job must succeed +- PyPI environment configured with trusted publishing +- Token in GitHub secrets (automated via Trusted Publisher) + +**Manual deployment:** +```bash +make deploy # Build and upload to PyPI +``` + +## Workflow Triggers + +The pipeline runs automatically on: + +- **Pull requests** - All PR types (opened, synchronize, reopened, labeled, unlabeled) +- **Push to main/develop** - On code pushes to main and develop branches +- **Version tags** - On tags matching `v*` (triggers PyPI publishing) +- **Manual dispatch** - Triggered via GitHub Actions UI with options + +## Local Development Setup + +### Prerequisites + +- Python 3.12+ +- Conda/Mamba (for integration tests) +- Git + +### Quick Setup + +```bash +# Install development dependencies +make install-dev + +# Set up pre-commit hooks +make pre-commit +``` + +### Available Commands + +```bash +# Code quality +make lint # Run isort and flake8 (auto-fix) +make lint-check # Check linting (don't fix) +make format # Format imports with isort + +# Testing +make test # Run unit tests +make test-cov # Run tests with coverage +make test-integration # Run full pipeline tests + +# Building +make build # Build packages +make clean # Clean all artifacts +make clean-pyc # Remove Python cache files + +# Versioning +make sync-config-versions # Stamp config VERSION fields with the git-tag version +make check-config-versions # Verify config VERSION fields are in sync + +# Useful +make help # Show all commands +``` + +## Pre-commit Hooks + +Pre-commit hooks are configured to automatically check code before commits. + +### Setup + +```bash +make pre-commit +``` + +### Hooks Included + +- Trailing whitespace removal +- End-of-file fixer +- YAML validation +- Large file detection (>1MB) +- Merge conflict detection +- JSON validation +- Line ending normalization +- Import sorting (isort) +- Code linting (flake8) +- Type checking (mypy, optional) + +Run manually: +```bash +pre-commit run --all-files +``` + +## Configuration Files + +### GitHub Actions Workflow +- **File:** `.github/workflows/python-app.yaml` +- Defines all CI/CD pipeline stages +- Configures job dependencies and triggers + +### Pre-commit Config +- **File:** `.pre-commit-config.yaml` +- Defines hooks run before commits +- Includes linting and formatting tools + +### Makefile +- **File:** `Makefile` +- Provides convenient commands for local development +- Mirrors CI/CD pipeline steps + +### Package Configuration +- **File:** `pyproject.toml` - Modern Python project metadata +- **File:** `setup.py` - Package installation script +- **File:** `setup.cfg` - Tool configuration + - isort settings + - flake8 settings + - pytest configuration + - coverage settings + +### Environment +- **File:** `environment.yml` - Conda environment for testing +- Includes all runtime and development dependencies + +## Troubleshooting + +### Linting Failures + +If linting fails locally or in CI: + +```bash +# Fix imports +make format + +# Fix other issues manually (requires code changes) +make lint-check # See what needs fixing +``` + +### Test Failures + +```bash +# Run tests with verbose output +make test-cov + +# Check coverage gaps +# Open htmlcov/index.html in browser +``` + +### Build Issues + +```bash +# Clean previous builds +make clean + +# Rebuild +make build + +# Check for issues +twine check dist/* +``` + +### Integration Test Failures + +```bash +# Debug locally +cd tests +RUN_INTEGRATION_TESTS=1 bash cicd_test.sh + +# Check logs in tests/LOGS directory +``` + +## Version Management + +MONSDA uses **versioneer** for automatic version management: + +1. Version is defined in git tags +2. Versioneer automatically derives version from git state +3. Package version matches git tag (e.g., tag `v1.0.0` → package version `1.0.0`) + +**Release process:** +1. Create release tag: `git tag v1.0.0` +2. Push tag: `git push --tags` +3. GitHub Actions automatically builds and publishes to PyPI + +### Config Version Synchronization + +The `VERSION` field in the config files under `configs/` and `tests/data/` +is kept in sync with the versioneer-derived version automatically, so that +`monsda`'s runtime version check (`monsda_check_version`) always passes. + +This is handled by `scripts/update_config_versions.py`, which mirrors the same +version source used at runtime (`MONSDA.__version__`, falling back to +`python setup.py --version`). The script is comment-safe (config files contain +`#` comments and are not strict JSON) and only rewrites empty, `FIXME`, or +version-like `VERSION` values, leaving documentation strings untouched. + +It runs automatically in CI: +- In the **build** job, before `python -m build`, so the published package ships + templates matching the release version. +- In the **full-pipeline** job, before the integration test, replacing the old + `sed`-based patching. + +Both jobs use `fetch-depth: 0` on checkout so versioneer can resolve git tags. + +Run locally: +```bash +make sync-config-versions # Stamp config VERSION fields with the git-tag version +make check-config-versions # Verify config VERSION fields are in sync (CI mode) + +# Or invoke the script directly with an explicit version +python scripts/update_config_versions.py --version 1.5.0 configs/*.json +``` + +## Best Practices + +1. **Always run linting before committing:** + ```bash + make pre-commit # or rely on pre-commit hooks + ``` + +2. **Run tests locally before pushing:** + ```bash + make test-cov + ``` + +3. **Use meaningful commit messages** that reference issue numbers + +4. **Add tests for new features** to maintain coverage + +5. **Keep dependencies updated** by regularly running `pip install --upgrade` + +6. **Review CI logs** if builds fail to understand what needs fixing + +## Contact & Support + +For issues or questions about the CI/CD pipeline: +- Check GitHub Actions logs for specific failures +- Review this documentation +- Open an issue on GitHub +- Contact the maintainers + +## References + +- [GitHub Actions Documentation](https://docs.github.com/en/actions) +- [pytest Documentation](https://docs.pytest.org/) +- [flake8 Documentation](https://flake8.pycqa.org/) +- [isort Documentation](https://pycqa.github.io/isort/) +- [Pre-commit Framework](https://pre-commit.com/) +- [Setuptools Documentation](https://setuptools.pypa.io/) +- [Versioneer](https://github.com/python-versioneer/python-versioneer) diff --git a/MONSDA/Configurator.py b/MONSDA/Configurator.py index 97bd7be8..95fe0e3b 100755 --- a/MONSDA/Configurator.py +++ b/MONSDA/Configurator.py @@ -17,6 +17,7 @@ from snakemake.common.configfile import load_configfile from MONSDA.Logger import * +from MONSDA.PostDE import load_postde_config from . import _version @@ -27,14 +28,14 @@ installpath = os.path.dirname(__file__).replace( os.sep.join(["lib", pythonversion, "site-packages", "MONSDA"]), "share" ) -except: +except Exception: installpath = os.getcwd() configpath = os.path.join(installpath, "MONSDA", "configs") current_path = os.getcwd() template = load_configfile(os.sep.join([configpath, "template_base_commented.json"])) -none_workflow_keys = ["WORKFLOWS", "BINS", "MAXTHREADS", "SETTINGS", "VERSION"] +none_workflow_keys = ["WORKFLOWS", "BINS", "MAXTHREADS", "SETTINGS", "VERSION", "POSTDE"] comparable_workflows = ["DE", "DEU", "DAS", "DTU", "PEAKS"] quantifying_workflows = ["COUNTING"] IP_workflows = ["PEAKS"] @@ -108,6 +109,7 @@ def __init__(self): self.workflowsDict = NestedDefaultDict() self.settingsDict = NestedDefaultDict() self.settingsList = [] + self.postde = {"enabled": False, "config": ""} class GUIDE: @@ -179,7 +181,7 @@ def proof_input(self, whitespace, proof=None, spec=None): [float(x) for x in a.split(",")] self.answer = a break - except: + except ValueError: self.clear(2) prRed("please enter integer or float") continue @@ -187,10 +189,10 @@ def proof_input(self, whitespace, proof=None, spec=None): if not a: self.answer = a break - ending = proof.replace("end_exist_", "") + "$" - if not re.findall(ending, a): + endings = proof.replace("end_exist_", "").split("|") + if not any(a.endswith(ending) for ending in endings): self.clear(2) - prRed(f"Nope, file has to end with '{ending.replace('$','')}'") + prRed(f"Nope, file has to end with '{' or '.join(endings)}'") continue elif not os.path.isfile(a): self.clear(2) @@ -304,7 +306,7 @@ def setInDict(dataDict, maplist, value): dataDict[first].append(value) else: dataDict[first] = value - except: + except (KeyError, TypeError, AttributeError): dataDict[first] = value @@ -579,7 +581,7 @@ def show_settings(): var = json.dumps(var, indent=4) try: print(f"{name} = {var}") - except: + except Exception: print(f"{name} = NestedDefaultDict()") print("\n============") @@ -753,10 +755,12 @@ def add_workflows(existing_workflows=None): prGreen("\nADD WORKFLOWS\n") possible_workflows = list(project.baseDict.keys()) for e in none_workflow_keys: - possible_workflows.remove(e) + if e in possible_workflows: + possible_workflows.remove(e) if existing_workflows: for e in existing_workflows: - possible_workflows.remove(e) + if e in possible_workflows: + possible_workflows.remove(e) posWorkDict = NestedDefaultDict() counter = 1 for w in possible_workflows: @@ -1140,7 +1144,7 @@ def set_settings(): try: if project.samplesDict[k]["cond"] == ":".join(maplist): seq = project.samplesDict[k]["seq"] - except: + except (KeyError, TypeError): continue setInDict(project.settingsDict, maplist + ["SEQUENCING"], seq) settings_to_make = ["REFERENCE", "DECOY", "GTF", "GFF"] @@ -1169,7 +1173,10 @@ def set_settings(): else: s = last_answer if key in ["GTF", "GFF"]: - p = f"end_exist_.{key.lower()}.gz" + suffix = f".{key.lower()}" + p = f"end_exist_{suffix}|{suffix}.gz|{suffix}.bgz" + elif key == "REFERENCE": + p = "end_exist_.fa|.fa.gz|.fa.bgz" elif key == "IP": p = None else: @@ -1257,7 +1264,7 @@ def modify(config=None): else: print("") prCyan(f" {config}") - except: + except Exception: er = 2 condition_pathes = getPathesFromDict(modify_config, "SAMPLES") @@ -1272,11 +1279,16 @@ def modify(config=None): active_workflows = modify_config["WORKFLOWS"].split(",") inactive_workflows = list(modify_config.keys()) for e in none_workflow_keys: - inactive_workflows.remove(e) + if e in inactive_workflows: + inactive_workflows.remove(e) for wf in inactive_workflows: project.workflowsDict[wf] = decouple(modify_config[wf]) for e in modify_config["WORKFLOWS"].split(","): - inactive_workflows.remove(e) + if e in inactive_workflows: + inactive_workflows.remove(e) + project.postde = modify_config.get( + "POSTDE", {"enabled": False, "config": ""} + ) prRed("\n Following configuration was found :\n") prRed(" Condition-Tree:\n") print_dict(project.conditionsDict, gap=" ") @@ -1297,7 +1309,8 @@ def modify(config=None): opts["2"] = "remove workflows" opts["3"] = "add conditions" opts["4"] = "remove conditions" - opts["5"] = "choose another file" + opts["5"] = "configure PostDE" + opts["6"] = "choose another file" guide.display(question="choose an option", options=opts, proof=opts.keys()) guide.clear(3) if guide.answer == "1": @@ -1309,6 +1322,8 @@ def modify(config=None): if guide.answer == "4": return remove_conditions() if guide.answer == "5": + return finalize() + if guide.answer == "6": project.settingsDict = NestedDefaultDict() project.conditionsDict = NestedDefaultDict() project.workflowsDict = NestedDefaultDict() @@ -1569,7 +1584,9 @@ def set_workflows(wf=None): if ( "FEATURES" in project.baseDict[workflow].keys() and not project.workflowsDict[workflow]["FEATURES"] + and "countreads" in tools_to_use ): + opt_dict = NestedDefaultDict() number = 1 for k in project.baseDict[workflow]["FEATURES"].keys(): opt_dict[number] = k @@ -1747,6 +1764,43 @@ def set_cores(): return finalize() +def configure_postde(final_dict): + workflows = [ + w.strip() + for w in str(final_dict.get("WORKFLOWS", "")).split(",") + if w.strip() + ] + if "DE" not in workflows: + final_dict["POSTDE"] = {"enabled": False, "config": ""} + return + current = getattr(project, "postde", {"enabled": False, "config": ""}) + guide.display( + question="Enable PostDE analysis (GO/GSEA/GSVA/decoupleR/dream) after DE?", + proof=["yes", "no"], + spec="yes" if current.get("enabled") else "no", + ) + if guide.answer.strip().lower() not in ("yes", "y"): + final_dict["POSTDE"] = {"enabled": False, "config": ""} + return + while True: + guide.display( + question="Enter the absolute path to the PostDE analysis JSON", + whitespace=True, + spec=current.get("config", ""), + ) + path = guide.answer.strip() + if not path: + path = current.get("config", "") + try: + final_dict["POSTDE"] = load_postde_config( + {"enabled": True, "config": path}, config=final_dict + ) + return + except ValueError as err: + prRed("PostDE config invalid: " + str(err)) + continue + + def finalize(): pickle_unfinished("finalize") final_dict = NestedDefaultDict() @@ -1757,6 +1811,7 @@ def finalize(): final_dict["VERSION"] = __version__ final_dict["SETTINGS"] = project.settingsDict final_dict.update(project.workflowsDict) + configure_postde(final_dict) if project.subname: configfile = f"config_{'_'.join([project.name,project.subname])}.json" diff --git a/MONSDA/Containers.py b/MONSDA/Containers.py new file mode 100644 index 00000000..1f5b7d0d --- /dev/null +++ b/MONSDA/Containers.py @@ -0,0 +1,13 @@ +import re + +from MONSDA import _version + + +def normalize_container_version(version=None): + version = version or _version.get_versions()["version"] + version = re.sub(r"^v(?=\d)", "", version) + version = re.sub(r"(?:[.+-]dirty)$", "", version) + version = re.sub(r"[^A-Za-z0-9_.-]+", "-", version).strip(".-") + if not version: + raise ValueError("MONSDA version cannot be converted to an OCI tag") + return version diff --git a/MONSDA/Deprecated.py b/MONSDA/Deprecated.py deleted file mode 100644 index cd5bd546..00000000 --- a/MONSDA/Deprecated.py +++ /dev/null @@ -1,181 +0,0 @@ -######################################## -############## DEPRECATED ############## -######################################## - -@check_run -def sources(config): - logid = scriptname+'.Collection_sources: ' - ret = list() - search = [x[0] for x in keysets_from_dict(config["SOURCE"]) if x[0] != 'last'] - if len(getFromDict(config['SAMPLES'], search)) > 0: - ret.extend(search) - log.debug(logid+str(ret)) - return ret - -@check_run -def genomepath(s, config): - logid=scriptname+'.Collection_genomepath: ' - sa = os.path.basename(str(s)) - cond= s.split(os.sep)[-2] - sk = find_key_for_value(sa, config["SAMPLES"]) - log.debug(logid+'GENOMEPATH: '+str([sa, cond, sk])) - for skey in sk: - klist = value_extract(skey, config["SOURCE"]) - for k in klist: - for x, y in config["GENOME"].items(): - log.debug(logid+'GENOMEPATH: '+str([x, y, k])) - if str(k) == str(y) or str(k) == str(x): - return os.path.join(str(x), str(y)) - -@check_run -def genome(s, config): - logid=scriptname+'.Collection_genome: ' - sa = os.path.basename(str(s)) - sp = source_from_sample(str(s), config) - cond= s.split(os.sep)[-2] - sk = find_key_for_value(sa, config['SAMPLES']) - for skey in sk: - klist = value_extract(skey, config['SOURCE']) - for k in klist: - for x, y in config['GENOME'].items(): - log.debug(logid+str([k, x, y])) - if str(k) == str(x): - return str(y) - -@check_run -def fullgenomepath(sa, config): - ret=list() - for s in sa: - l = config["GENOME"][s] - ret.append(os.path.join(str(s), str(l))) - return ret - -@check_run -def genomename(s, config): - s = os.path.basename(str(s)) - for k, v in config["SAMPLES"].items(): - for g, l in v.items(): - if s in l: - for x, y in config["GENOME"].items(): - if g == y: - return str(x) - -@check_run -def transcriptomepath(s, config): - logid=scriptname+'.Collection_transcriptomepath: ' - sa = os.path.basename(str(s)) - cond= s.split(os.sep)[-2] - sk = find_key_for_value(sa, config["SAMPLES"]) - log.debug(logid+'TRANSCRIPTOMEPATH: '+str([sa, cond, sk])) - for skey in sk: - klist = value_extract(skey, config["SOURCE"]) - for k in klist: - for x, y in config["TRANSCRIPTOME"].items(): - log.debug(logid+'TRANSCRIPTOMEPATH: '+str([x, y, k])) - if str(k) == str(y) or str(k) == str(x): - return os.path.join(str(x), str(y)) - -@check_run -def transcriptome(s, config): - logid=scriptname+'.Collection_transcriptome: ' - sa = os.path.basename(str(s)) - sp = source_from_sample(str(s), config) - cond= s.split(os.sep)[-2] - sk = find_key_for_value(sa, config['SAMPLES']) - for skey in sk: - klist = value_extract(skey, config['SOURCE']) - for k in klist: - for x, y in config['TRANSCRIPTOME'].items(): - log.debug(logid+str([k, x, y])) - if str(k) == str(x): - return str(y) - -@check_run -def fulltranscriptomepath(sa, config): - ret=list() - for s in sa: - l = config["TRANSCRIPTOME"][s] - ret.append(os.path.join(str(s), str(l))) - return ret - -@check_run -def transcriptomename(s, config): - s = os.path.basename(str(s)) - for k, v in config["SAMPLES"].items(): - for g, l in v.items(): - if s in l: - for x, y in config["TRANSCRIPTOME"].items(): - if g == y: - return str(x) - - -@check_run -def namefromfile(s, config): - if 'NAME' not in config: - return '' - else: - sa = os.path.basename(str(s)) - cond= s.split(os.sep)[-2] - sk = find_key_for_value(sa, config["SAMPLES"]) - for skey in sk: - klist = value_extract(skey, config["NAME"]) - for k in klist: - if str(skey) == str(cond): - return str(k) - - -@check_run -def namefrompath(p, config): - p = os.path.dirname(p).split(os.sep) - klist = getFromDict(config["NAME"], p) if 'NAME' in config else list('') - for k in klist: - return str(k) - -@check_run -def pathstogenomes(samples, config): - ret = list() - for s in samples: - s = os.path.basename(s) - for k, v in config["SAMPLES"].items(): - for g, l in v.items(): - if s in l: - for x, y in config["GENOME"].items(): - if g == y: - ret.append(os.path.join(str(x), str(y))) - return sorted(list(set(ret))) - - -@check_run -def source_from_sample(sample, config): - logid=scriptname+'.Collection_source_from_sample: ' - s = os.path.dirname(str(sample)) - cond= s.split(os.sep) - log.debug(logid+str([s, cond])) - ret = getFromDict(config["SOURCE"], cond)[0] - return ret - - -@check_run -def anno_from_file(sample, config, step): - logid = scriptname+'.Collection_anno_from_file: ' - p = os.path.dirname(genomepath(sample, config)) - s = source_from_sample(sample, config) - ret = os.path.join(config["REFERENCE"], p, subDict(config["ANNOTATE"], s)[step]) - log.debug(logid+str(ret)) - return ret - -@check_run -def anno_from_source(source, config, step): - logid = scriptname+'.Collection_anno_from_source: ' - s = source.split(os.sep)[0:-1] - p = s[0] - samp = source.split(os.sep)[-1] - log.debug(logid+str(s)) - runstate = runstate_from_sample([samp], config)[0] - lst = list() - lst.extend(s) - lst.append(runstate) - log.debug(logid+str(lst)) - ret = os.path.join(config["REFERENCE"], p, subDict(config["ANNOTATE"], lst)[step]) - log.debug(logid+str(ret)) - return ret diff --git a/MONSDA/Params.py b/MONSDA/Params.py index 6a62a918..77be14a6 100644 --- a/MONSDA/Params.py +++ b/MONSDA/Params.py @@ -160,9 +160,9 @@ def samplesheet_to_settings(samplesheet_path: str) -> dict: SAMPLE - sample name / accession GROUP - group label for differential analysis SEQUENCING - e.g. ``paired`` or ``single`` - REFERENCE - path to genome FASTA (.fa.gz) - GTF - path to GTF annotation (.gtf.gz) [optional] - GFF - path to GFF annotation (.gff.gz) [optional] + REFERENCE - path to genome FASTA (.fa, .fa.gz or .fa.bgz) + GTF - path to GTF annotation (.gtf, .gtf.gz or .gtf.bgz) [optional] + GFF - path to GFF annotation (.gff, .gff.gz or .gff.bgz) [optional] INDEX - path to pre-built index [optional] PREFIX - mapper index prefix [optional] DECOY - path to decoy file [optional] @@ -923,7 +923,7 @@ def get_cutoff_as_string(config: dict, subwork: str, cf: str) -> str: subwork : str workflow part cf : str - cutoff + cutoff, either 'padj' or 'lfc' Returns ------- @@ -931,11 +931,30 @@ def get_cutoff_as_string(config: dict, subwork: str, cf: str) -> str: configured cutoffs or defaults """ logid = scriptname + ".get_cutoff: " - cutoff = ( - str(config[subwork]["CUTOFFS"].get(cf)) - if config[subwork].get("CUTOFFS") - else ".05" if cf == "pval" else "1.5" - ) + defaults = {"padj": ".05", "lfc": "1.5"} + deprecated = {"padj": ["pvalue", "pval"]} + cutoffs = config[subwork].get("CUTOFFS") or dict() + cutoff = cutoffs.get(cf) + if cutoff is None: + for alias in deprecated.get(cf, []): + if cutoffs.get(alias) is not None: + cutoff = cutoffs[alias] + log.warning( + logid + + "CUTOFFS key '" + + alias + + "' in " + + str(subwork) + + " is deprecated, cutoffs are applied to adjusted p-values, please rename it to '" + + cf + + "'" + ) + break + if cutoff is None: + if cf not in defaults: + log.error(logid + "No default cutoff known for " + str(cf)) + cutoff = defaults.get(cf, "1.5") + cutoff = str(cutoff) log.info(logid + "CUTOFFS: " + str(cf) + ":" + cutoff) return cutoff @@ -1173,7 +1192,7 @@ def get_reps(samples: list, config: dict, analysis: str, process: str = "smk") - if ( "BATCHES" in partconf and len(partconf["BATCHES"]) >= idx + 1 - and partconf.get("BATCHES") != "" + and str(partconf["BATCHES"][idx]) != "" ): ret["batches"].append(str(partconf["BATCHES"][idx]).replace(",", "_")) else: @@ -1181,7 +1200,7 @@ def get_reps(samples: list, config: dict, analysis: str, process: str = "smk") - if ( "TYPES" in partconf and len(partconf["TYPES"]) >= idx + 1 - and partconf.get("TYPE") != "" + and str(partconf["TYPES"][idx]) != "" ): ret["types"].append(str(partconf["TYPES"][idx]).replace(",", "_")) else: @@ -1390,7 +1409,7 @@ def runstate_from_sample(sample: list, config: dict) -> list: log.debug(logid + "SAMPLE: " + s) try: c = mu.get_from_dict(config["SETTINGS"], s.split(os.sep))[0] - except: + except (KeyError, IndexError, TypeError): c = None log.debug(logid + "SETTINGS: " + str(c)) if mu.dict_inst(c): @@ -1698,7 +1717,7 @@ def get_pairing( log.debug(logid + "Match found: " + str(v) + " : " + str(x)) try: matching = samplecond([x], config)[0].replace("MAPPED/", "") - except: + except (KeyError, IndexError): matching = x log.info(logid + "PAIRINGS: " + sample + ": " + str(matching)) if not matching or matching == "": @@ -1819,7 +1838,7 @@ def check_tool_params( return "std" else: return "" - except: + except (KeyError, IndexError, TypeError): if subconf == "MAPPING": return "std" else: @@ -1938,7 +1957,7 @@ def fixRunParameters( return re.sub(paramToFix + " [0-9a-zA-Z\.\_\-\\\/]+", fixedParam, para) else: log.warning( - f"No parameter to fix {paramToFix} found or fix {fixedParam} defined for {para}!" + f"No parameter to fix {paramToFix} in INDEX options or fix {fixedParam} defined for {para}!" ) return para diff --git a/MONSDA/PostDE.py b/MONSDA/PostDE.py new file mode 100644 index 00000000..64931055 --- /dev/null +++ b/MONSDA/PostDE.py @@ -0,0 +1,380 @@ +import hashlib +import json +import math +import os +import shutil + +POSTDE_SECTION_KEYS = frozenset({"enabled", "config"}) +POSTDE_TOP_KEYS = frozenset( + { + "padj", + "lfc", + "seed", + "min_size", + "max_size", + "gprofiler", + "clusterprofiler", + "gsva", + "decoupler", + "dream", + "plots", + "report", + "shiny", + } +) +POSTDE_METHOD_KEYS = { + "gprofiler": frozenset( + { + "enabled", + "organism", + "domain_scope", + "allow_network", + "sources", + "correction_method", + "background", + } + ), + "clusterprofiler": frozenset( + {"enabled", "term2gene", "term2name", "go_expand", "background"} + ), + "gsva": frozenset({"enabled", "method", "min_size", "max_size"}), + "decoupler": frozenset( + { + "enabled", + "network", + "source_col", + "target_col", + "mor_col", + "allow_network", + "resource", + "organism", + "top", + "min_size", + "methods", + "contrast_activity", + } + ), + "dream": frozenset({"enabled", "metadata", "formula", "contrasts"}), + "plots": frozenset({"enabled"}), + "report": frozenset({"enabled"}), + "shiny": frozenset({"enabled"}), +} +POSTDE_RESOURCE_KEYS = { + "gprofiler": ("background",), + "clusterprofiler": ("term2gene", "term2name", "background"), + "decoupler": ("network",), + "dream": ("metadata",), +} +POSTDE_BOOL_FLAGS = { + "gprofiler": ("allow_network",), + "clusterprofiler": ("go_expand",), + "decoupler": ("allow_network", "contrast_activity"), +} +POSTDE_SUPPORTED_DE_TOOLS = frozenset({"deseq2", "edger"}) +POSTDE_DOMAIN_SCOPES = ("annotated", "known", "custom", "custom_annotated") +POSTDE_CORRECTION_METHODS = ( + "g_SCS", + "bonferroni", + "fdr", + "false_discovery_rate", + "gSCS", + "analytical", +) +POSTDE_GSVA_METHODS = ("gsva", "ssgsea") +POSTDE_DECOUPLER_RESOURCES = ("collectri", "progeny") +POSTDE_DECOUPLER_METHODS = ("ulm", "mlm") + + +def _is_number(value): + return ( + isinstance(value, (int, float)) + and not isinstance(value, bool) + and math.isfinite(value) + ) + + +def _is_integral(value): + return _is_number(value) and value == int(value) + + +def _validate_analysis_config(analysis, base_dir): + unknown = set(analysis) - POSTDE_TOP_KEYS + if unknown: + raise ValueError( + "POSTDE config has unknown top-level fields: " + + ", ".join(sorted(unknown)) + ) + if "padj" in analysis and ( + not _is_number(analysis["padj"]) or not 0 < analysis["padj"] < 1 + ): + raise ValueError("POSTDE config padj must be numeric in (0, 1)") + if "lfc" in analysis and ( + not _is_number(analysis["lfc"]) or analysis["lfc"] < 0 + ): + raise ValueError("POSTDE config lfc must be numeric >= 0") + if "seed" in analysis and not _is_integral(analysis["seed"]): + raise ValueError("POSTDE config seed must be a finite integer") + if "min_size" in analysis and ( + not _is_integral(analysis["min_size"]) or analysis["min_size"] < 1 + ): + raise ValueError("POSTDE config min_size must be an integer >= 1") + if "max_size" in analysis and ( + not _is_integral(analysis["max_size"]) + or analysis["max_size"] < analysis.get("min_size", 1) + ): + raise ValueError("POSTDE config max_size must be an integer >= min_size") + for method, keys in POSTDE_METHOD_KEYS.items(): + sub = analysis.get(method) + if sub is None: + continue + if not isinstance(sub, dict): + raise ValueError("POSTDE config " + method + " must be an object") + unknown = set(sub) - keys + if unknown: + raise ValueError( + "POSTDE config " + + method + + " has unknown fields: " + + ", ".join(sorted(unknown)) + ) + if "enabled" in sub and not isinstance(sub["enabled"], bool): + raise ValueError( + "POSTDE config " + method + ".enabled must be a boolean" + ) + for flag in POSTDE_BOOL_FLAGS.get(method, ()): + if flag in sub and not isinstance(sub[flag], bool): + raise ValueError( + "POSTDE config " + method + "." + flag + " must be a boolean" + ) + for vis in ("plots", "report", "shiny"): + if analysis.get(vis, {}).get("enabled"): + raise ValueError( + "POSTDE config " + + vis + + ".enabled=true is not available (reporting.R not yet provided)" + ) + gp = analysis.get("gprofiler", {}) + if gp.get("enabled"): + if not isinstance(gp.get("organism"), str) or not gp["organism"].strip(): + raise ValueError( + "POSTDE config gprofiler.enabled=true requires gprofiler.organism" + ) + if not gp.get("allow_network"): + raise ValueError( + "POSTDE config gprofiler.enabled=true requires gprofiler.allow_network=true" + ) + if "domain_scope" in gp and gp["domain_scope"] not in POSTDE_DOMAIN_SCOPES: + raise ValueError( + "POSTDE config gprofiler.domain_scope must be one of " + + ", ".join(POSTDE_DOMAIN_SCOPES) + ) + if "correction_method" in gp and gp[ + "correction_method" + ] not in POSTDE_CORRECTION_METHODS: + raise ValueError( + "POSTDE config gprofiler.correction_method must be one of " + + ", ".join(POSTDE_CORRECTION_METHODS) + ) + cp = analysis.get("clusterprofiler", {}) + gs = analysis.get("gsva", {}) + if (cp.get("enabled") or gs.get("enabled")) and not cp.get("term2gene"): + raise ValueError( + "POSTDE config clusterprofiler.term2gene required when clusterprofiler/gsva enabled" + ) + if "method" in gs and gs["method"] not in POSTDE_GSVA_METHODS: + raise ValueError( + "POSTDE config gsva.method must be one of " + ", ".join(POSTDE_GSVA_METHODS) + ) + if "min_size" in gs and ( + not _is_integral(gs["min_size"]) or gs["min_size"] < 1 + ): + raise ValueError("POSTDE config gsva.min_size must be an integer >= 1") + if "max_size" in gs and ( + not _is_integral(gs["max_size"]) + or gs["max_size"] < gs.get("min_size", analysis.get("min_size", 1)) + ): + raise ValueError("POSTDE config gsva.max_size must be an integer >= min_size") + dc = analysis.get("decoupler", {}) + if dc.get("enabled"): + if not dc.get("network") and not dc.get("allow_network"): + raise ValueError( + "POSTDE config decoupler.enabled=true requires decoupler.network or decoupler.allow_network=true" + ) + if "resource" in dc and dc["resource"] not in POSTDE_DECOUPLER_RESOURCES: + raise ValueError( + "POSTDE config decoupler.resource must be one of " + + ", ".join(POSTDE_DECOUPLER_RESOURCES) + ) + if "methods" in dc: + methods = ( + dc["methods"] if isinstance(dc["methods"], list) else [dc["methods"]] + ) + if not methods or any( + m not in POSTDE_DECOUPLER_METHODS for m in methods + ): + raise ValueError( + "POSTDE config decoupler.methods must be a list of " + + ", ".join(POSTDE_DECOUPLER_METHODS) + ) + if "min_size" in dc and ( + not _is_integral(dc["min_size"]) or dc["min_size"] < 1 + ): + raise ValueError("POSTDE config decoupler.min_size must be an integer >= 1") + if "top" in dc and (not _is_integral(dc["top"]) or dc["top"] < 1): + raise ValueError("POSTDE config decoupler.top must be an integer >= 1") + dm = analysis.get("dream", {}) + if dm.get("enabled"): + if not isinstance(dm.get("metadata"), str) or not dm["metadata"].strip(): + raise ValueError( + "POSTDE config dream.enabled=true requires dream.metadata" + ) + if not isinstance(dm.get("formula"), str) or not dm["formula"].strip(): + raise ValueError("POSTDE config dream.enabled=true requires dream.formula") + for method, keys in POSTDE_RESOURCE_KEYS.items(): + sub = analysis.get(method, {}) + for key in keys: + path = sub.get(key) + if path is None: + continue + if not isinstance(path, str) or not path.strip(): + raise ValueError( + "POSTDE config " + method + "." + key + " must be a non-empty path" + ) + full = path if os.path.isabs(path) else os.path.join(base_dir, path) + if not os.path.isfile(full): + raise ValueError( + "POSTDE config resource file not found: " + + method + + "." + + key + + " -> " + + full + ) + + +def load_postde_config(section, config=None): + if section is None: + return None + if not isinstance(section, dict): + raise ValueError("POSTDE section must be an object") + unknown = set(section) - POSTDE_SECTION_KEYS + if unknown: + raise ValueError( + "POSTDE section has unknown fields: " + ", ".join(sorted(unknown)) + ) + if "enabled" not in section: + raise ValueError("POSTDE section missing required field 'enabled'") + if not isinstance(section["enabled"], bool): + raise ValueError("POSTDE.enabled must be a boolean") + if not section["enabled"]: + return {"enabled": False, "config": section.get("config", "")} + if not isinstance(section.get("config"), str) or not section["config"].strip(): + raise ValueError("POSTDE.enabled=true requires a non-empty POSTDE.config path") + config_path = os.path.abspath(section["config"]) + if not os.path.isfile(config_path): + raise ValueError("POSTDE.config file not found: " + config_path) + try: + with open(config_path) as fh: + analysis = json.load(fh) + except (OSError, ValueError) as err: + raise ValueError("POSTDE.config is not readable JSON: " + str(err)) + if not isinstance(analysis, dict): + raise ValueError("POSTDE.config must contain a JSON object") + _validate_analysis_config(analysis, os.path.dirname(config_path)) + if config is not None: + workflows = [ + w.strip() + for w in str(config.get("WORKFLOWS", "")).split(",") + if w.strip() + ] + if "DE" not in workflows: + raise ValueError("POSTDE.enabled=true requires DE in WORKFLOWS") + de_tools = config.get("DE", {}).get("TOOLS", {}) + if isinstance(de_tools, dict): + unsupported = set(de_tools) - POSTDE_SUPPORTED_DE_TOOLS + if unsupported: + raise ValueError( + "POSTDE.enabled=true does not support DE tools: " + + ", ".join(sorted(unsupported)) + ) + return {"enabled": True, "config": config_path} + + +def _collect_resources(analysis, base_dir): + resources = [] + for method, keys in POSTDE_RESOURCE_KEYS.items(): + sub = analysis.get(method, {}) + for key in keys: + path = sub.get(key) + if path is None: + continue + full = path if os.path.isabs(path) else os.path.join(base_dir, path) + resources.append((method, key, full)) + return resources + + +def _staged_name(method, key, full): + ext = os.path.splitext(os.path.basename(full))[1] + return method + "_" + key + ext + + +def _same_content(path_a, path_b): + """Compare file contents directly (readbytes), immune to stat-based + caching that can miss same-size/same-mtime content mutations.""" + with open(path_a, "rb") as fh_a, open(path_b, "rb") as fh_b: + return fh_a.read() == fh_b.read() + + +def _same_text(path, text): + """Compare a file's bytes against an expected text string.""" + with open(path, "rb") as fh: + return fh.read() == text.encode("utf-8") + + +def prepare_postde(config, subdir): + section = load_postde_config(config.get("POSTDE"), config=config) + if section is None or not section.get("enabled"): + return None + config_path = section["config"] + with open(config_path) as fh: + analysis = json.load(fh) + base_dir = os.path.dirname(config_path) + resources = _collect_resources(analysis, base_dir) + digest = hashlib.sha256() + digest.update(json.dumps(analysis, sort_keys=True).encode("utf-8")) + for method, key, full in sorted(resources): + with open(full, "rb") as fh: + digest.update(fh.read()) + staged = os.path.join(os.path.abspath(subdir), "POSTDE_" + digest.hexdigest()[:12]) + names = { + (method, key): _staged_name(method, key, full) + for method, key, full in resources + } + staged_analysis = json.loads(json.dumps(analysis)) + for method, key, full in resources: + staged_analysis[method][key] = names[(method, key)] + expected_config = json.dumps(staged_analysis, indent=2) + if os.path.isdir(staged): + staged_config = os.path.join(staged, "config.json") + config_ok = os.path.isfile(staged_config) and _same_text( + staged_config, expected_config + ) + resources_ok = all( + os.path.isfile(os.path.join(staged, names[(m, k)])) + and _same_content(full, os.path.join(staged, names[(m, k)])) + for m, k, full in resources + ) + if config_ok and resources_ok: + return staged + os.makedirs(staged, exist_ok=True) + for method, key, full in resources: + dst = os.path.join(staged, names[(method, key)]) + if not os.path.exists(dst) or not _same_content(full, dst): + shutil.copy2(full, dst) + staged_config = os.path.join(staged, "config.json") + if not os.path.isfile(staged_config) or not _same_text( + staged_config, expected_config + ): + with open(staged_config, "w") as fh: + fh.write(expected_config) + return staged diff --git a/MONSDA/RunMONSDA.py b/MONSDA/RunMONSDA.py index 422c82d3..f174db73 100755 --- a/MONSDA/RunMONSDA.py +++ b/MONSDA/RunMONSDA.py @@ -2,7 +2,6 @@ import argparse import os -import re import shlex import shutil import subprocess @@ -37,6 +36,12 @@ "LOGS" + os.sep + scriptname + ".log", "LOGS" + os.sep + scriptname + "_" + ts + ".log", ) + # keep tool versions of the archived run with the same timestamp as its log + if os.path.isfile(os.path.abspath("LOGS" + os.sep + "versions.txt")): + shutil.copy2( + "LOGS" + os.sep + "versions.txt", + "LOGS" + os.sep + "versions_" + ts + ".txt", + ) log = setup_logger( name=scriptname.lower(), @@ -53,6 +58,7 @@ import MONSDA.Params as mp import MONSDA.Utils as mu +import MONSDA.Versions as mv import MONSDA.Workflows as mw @@ -142,10 +148,17 @@ def parseargs(): parser.add_argument( "--oras-registry", type=str, - default="docker.io", + default="ghcr.io", metavar="HOST", - help="Container registry hostname for ORAS image pulls (default: docker.io). " - "Use e.g. ghcr.io for GitHub Container Registry.", + help="Container registry hostname for ORAS image pulls (default: ghcr.io).", + ) + parser.add_argument( + "--oras-namespace", + type=str, + default="jfallmann/monsda", + metavar="NAMESPACE", + help="Container namespace/repository for ORAS image pulls " + "(default: jfallmann/monsda). Use e.g. myuser/myrepo for a custom registry.", ) if len(sys.argv) == 1: @@ -168,7 +181,7 @@ def run_snakemake( ): try: logid = scriptname + ".run_snakemake: " - config = load_configfile(configfile) + config = mu.link_bgz_to_gz(load_configfile(configfile)) subdir = "SubSnakes" mp.create_skeleton(subdir, skeleton) @@ -207,7 +220,7 @@ def run_snakemake( os.sep.join(["lib", pythonversion, "site-packages", "MONSDA"]), "share", ) - except: + except Exception: installpath = os.getcwd() workflowpath = os.path.join(installpath, "MONSDA", "workflows") @@ -225,6 +238,15 @@ def run_snakemake( preprocess, subworkflows, postprocess = mw.get_processes(config) conditions = mp.get_conditions(config) + # Export versions of all tools configured to run for provenance and MultiQC + mv.write_versions( + config, + (preprocess or []) + (subworkflows or []) + (postprocess or []), + mw.envpath, + __version__, + configfile, + ) + """ START TO PREPROCESS IF WE NEED TO DOWNLOAD FILES WE DO THIS NOW @@ -492,7 +514,7 @@ def run_nextflow( logid = scriptname + ".run_nextflow: " argslist = list() writeout = True - config = load_configfile(configfile) + config = mu.link_bgz_to_gz(load_configfile(configfile)) if useconda and not any( [ x in optionalargs @@ -558,6 +580,15 @@ def run_nextflow( preprocess, subworkflows, postprocess = mw.nf_get_processes(config) conditions = mp.get_conditions(config) + # Export versions of all tools configured to run for provenance and MultiQC + mv.write_versions( + config, + (preprocess or []) + (subworkflows or []) + (postprocess or []), + mw.envpath, + __version__, + configfile, + ) + """ START TO PROCESS IF WE NEED TO DOWNLOAD FILES WE DO THIS NOW @@ -849,19 +880,9 @@ def run_nextflow( def runjob(jobtorun): try: logid = scriptname + ".runjob: " - run_cmd = jobtorun - if ( - re.search(r"(^|\s)nextflow(\s|$)", jobtorun) - and "NXF_SYNTAX_PARSER" not in jobtorun - ): - run_cmd = "NXF_SYNTAX_PARSER=v1 " + jobtorun - log.warning( - logid - + "Nextflow 26.04+ strict parser detected; forcing legacy parser (NXF_SYNTAX_PARSER=v1) for compatibility with generated workflows." - ) # return subprocess.run(jobtorun, shell=True, universal_newlines=True, capture_output=True) # python >= 3.7 job = subprocess.Popen( - run_cmd, + jobtorun, shell=True, universal_newlines=True, stdout=subprocess.PIPE, @@ -890,7 +911,7 @@ def runjob(jobtorun): log.error(logid + "STOPPING: " + str(output)) log.info("PLEASE CHECK LOG AT LOGS/MONSDA.log") job.kill() - sys.exit() + sys.exit(1) else: log.info(logid + str(output)) if outerr and outerr != "": @@ -905,7 +926,7 @@ def runjob(jobtorun): log.error(logid + "STOPPING: " + str(outerr)) log.info("PLEASE CHECK LOG AT LOGS/MONSDA.log") job.kill() - sys.exit() + sys.exit(1) else: log.info(logid + str(outerr)) if job.poll() is not None: @@ -935,7 +956,7 @@ def runjob(jobtorun): exc_tb, ) log.error("".join(tbe.format())) - sys.exit() + sys.exit(1) def main(): @@ -972,6 +993,7 @@ def main(): # --- set ORAS registry for container image pulls --- mw.set_oras_registry(knownargs.oras_registry) + mw.set_oras_namespace(knownargs.oras_namespace) log.debug( f"{logid} ARGS: {args} {type(args)} KNOWNARGS: {knownargs} {type(knownargs)} OPTIONALARGS: {optionalargs} {type(optionalargs)}" @@ -985,7 +1007,8 @@ def main(): + "{}".format(" ".join([shlex.quote(s) for s in sys.argv[1:]])) ) - required_version = load_configfile(knownargs.configfile).get("VERSION") + config = load_configfile(knownargs.configfile) + required_version = config.get("VERSION") if not required_version: sys.exit( "Can not check version needed, please add VERSION key to config file" @@ -1009,6 +1032,13 @@ def main(): else: log.info("Running MONSDA version " + __version__ + " as configured") + postde_section = config.get("POSTDE") + if postde_section is not None: + try: + mw.load_postde_config(postde_section, config=config) + except ValueError as err: + sys.exit("POSTDE config invalid: " + str(err)) + MIN_PYTHON = (3, 11) if sys.version_info < MIN_PYTHON: log.error("This script requires Python version >= 3.7") diff --git a/MONSDA/Utils.py b/MONSDA/Utils.py index 8dd741f5..33e04547 100644 --- a/MONSDA/Utils.py +++ b/MONSDA/Utils.py @@ -411,7 +411,7 @@ def merge_dicts(d:dict, u:dict) -> dict: # python 3.8+ compatibility try: collectionsAbc = collections.abc - except: + except AttributeError: collectionsAbc = collections for k, v in six.iteritems(u): @@ -876,7 +876,9 @@ def toarray(file:str, ulim:int) -> np.array: usecols=(ulim), delimiter="\t", unpack=True, - converters={ulim: lambda s: convertcol(s.decode("utf-8"))}, + converters={ + ulim: lambda s: convertcol(s.decode("utf-8") if isinstance(s, bytes) else s) + }, ) return x @@ -1128,6 +1130,68 @@ def check_ref(reference:str) -> str: return reference elif os.path.exists(os.path.abspath(reference + ".gz")): return reference + ".gz" + elif os.path.exists(os.path.abspath(reference + ".bgz")): + return reference + ".bgz" + + +@check_run +def link_bgz_to_gz(config: dict) -> dict: + """Replace .bgz file paths in config with .gz symlinks + + Walks the config recursively, for every string value pointing to an + existing .bgz file a sibling symlink with .gz ending is created + (bgzip is gzip compatible) and the config value is replaced by the + .gz path, so all tools relying on .gz endings work transparently. + + Parameters + ---------- + config : dict + config to check + + Returns + ------- + dict + config with .bgz paths replaced by .gz symlinks + """ + logid = scriptname + ".Utils_link_bgz_to_gz: " + + def _fix(val): + if isinstance(val, str) and val.endswith(".bgz"): + if not os.path.isfile(val): + log.warning( + logid + + f"{val} is not a readable file (missing or dangling link?), cannot link to .gz" + ) + return val + gz = val[: -len(".bgz")] + ".gz" + if os.path.isfile(gz): + log.info(logid + f"Using existing {gz} instead of {val}") + return gz + if os.path.lexists(gz): + log.warning( + logid + + f"{gz} exists but is no readable file (broken link?), keeping {val}" + ) + return val + try: + os.symlink(os.path.basename(val), gz) + log.info(logid + f"Linked {val} to {gz} for .gz compatibility") + except OSError as e: + log.warning( + logid + f"Could not link {val} to {gz}: {e}, keeping .bgz path" + ) + return val + return gz + return val + + for key, value in config.items(): + if isinstance(value, dict): + link_bgz_to_gz(value) + elif isinstance(value, list): + config[key] = [_fix(v) for v in value] + else: + config[key] = _fix(value) + return config @check_run diff --git a/MONSDA/Versions.py b/MONSDA/Versions.py new file mode 100644 index 00000000..47d5b3dc --- /dev/null +++ b/MONSDA/Versions.py @@ -0,0 +1,379 @@ +# Versions.py --- + +# Export of tool versions of a MONSDA run as deduced from the config file and +# the conda environment yaml files of the configured tools. + +import datetime +import logging +import os +import re + +log = logging.getLogger(__name__) + +scriptname = os.path.basename(__file__).replace(".py", "") + +# Tools/packages that are no analysis tools and only clutter the version list +SKIP_PACKAGES = frozenset( + [ + "pip", + "python", + "setuptools", + "wheel", + "openssl", + "zlib", + ] +) + +# Mapping of tool/package name to the corresponding MultiQC module name(s), +# names have to match the modules of MultiQC exactly, including case, since +# MultiQC 1.35 all module names are lowercase, e.g. featurecounts not +# featureCounts, +# tools without a MultiQC module are not listed, tools reporting the output of +# several other tools, like rustqc, map to all modules they produce output for +MULTIQC_MODULES = { + "bbmap": "bbmap", + "bbduk": "bbduk", + "bismark": "bismark", + "bowtie2": "bowtie2", + "cutadapt": "cutadapt", + "fastp": "fastp", + "fastqc": "fastqc", + "featurecounts": "featurecounts", + "hisat2": "hisat2", + "htseq": "htseq", + "kallisto": "kallisto", + "macs2": "macs2", + "macs3": "macs2", + "picard": "picard", + "qualimap": "qualimap", + "rastqc": "fastqc", + "rseqc": "rseqc", + "rustqc": ( + "custom_content", + "featurecounts", + "preseq", + "qualimap", + "rseqc", + "samtools", + ), + "salmon": "salmon", + "samtools": "samtools", + "star": "star", + "subread": "featurecounts", + "trimgalore": "cutadapt", + "trim-galore": "cutadapt", + "trimmomatic": "trimmomatic", + "umitools": "umitools", + "umi_tools": "umitools", +} + + +def multiqc_module(tool): + """Return the MultiQC module(s) for a tool or None if there is none + + Parameters + ---------- + tool : str + name of tool/conda package + + Returns + ------- + str or None + module name, comma separated if the tool reports for several modules + """ + module = MULTIQC_MODULES.get( + str(tool).lower().replace("_", "").replace("-", "") + ) or MULTIQC_MODULES.get(str(tool).lower()) + if isinstance(module, tuple): + return ",".join(module) + return module + + +def envs_from_config(config, steps): + """Collect conda environments of all tools configured to run + + Parameters + ---------- + config : dict + config json file parsed by snakemake.common.configfile + steps : list + list of workflow steps (e.g. ['QC', 'TRIMMING', 'MAPPING']) that will be run + + Returns + ------- + dict + environment name to set of workflow steps it is used in + """ + logid = scriptname + ".envs_from_config: " + envs = dict() + + for step in steps: + if not step or not config.get(step): + continue + conf = config[step] + if not isinstance(conf, dict): + continue + found = set() + stack = [conf] + while stack: + sub = stack.pop() + for key, val in sub.items(): + if isinstance(val, dict): + if key == "TOOLS": + found.update( + str(x) for x in val.keys() if x and str(x) != "None" + ) + else: + stack.append(val) + elif key.endswith("ENV") and isinstance(val, str) and val: + found.add(str(val)) + for env in found: + envs.setdefault(env, set()).add(step) + + log.debug(logid + str(envs)) + return envs + + +def env_yaml(env, step, envdir): + """Find the conda env yaml belonging to an env name of a workflow step + + Parameters + ---------- + env : str + environment name as configured + step : str + workflow step the env is used in + envdir : str + directory holding the conda env yamls + + Returns + ------- + str or None + path to yaml file + """ + names = [env, "_".join([env, step]), "_".join([env, str(step).lower()])] + # bisulfite mapping runs out of the plain mapper env, see segemehl_bisulfite.smk + if "segemehl" in env and "bisulfite" in env: + names.append(env.replace("bisulfite", "")) + for name in names: + path = os.path.join(envdir, name + ".yaml") + if os.path.isfile(path): + return path + return None + + +def parse_env_yaml(yamlfile): + """Parse pinned versions from the dependencies of a conda env yaml + + Parameters + ---------- + yamlfile : str + path to conda env yaml + + Returns + ------- + dict + package name to version string, 'unpinned' if no version is defined + """ + logid = scriptname + ".parse_env_yaml: " + versions = dict() + indeps = False + + with open(yamlfile, "r") as y: + for line in y: + line = line.rstrip() + if not line or line.lstrip().startswith("#"): + continue + entry = line.strip() + if not entry.startswith("- "): + indeps = line.startswith("dependencies:") + continue + if not indeps: + continue + entry = entry[2:].strip() + if entry.endswith(":"): + continue + match = re.match(r"^([A-Za-z0-9._+-]+)\s*(.*)$", entry) + if not match: + continue + pkg, spec = match.group(1), match.group(2).strip() + if pkg.lower() in SKIP_PACKAGES: + continue + spec = re.sub(r"^[=]{1,2}\s*", "", spec) + versions[pkg] = spec if spec else "unpinned" + + log.debug(logid + yamlfile + ": " + str(versions)) + return versions + + +# Package name prefixes of conda channels that are no part of the tool name +PACKAGE_PREFIXES = ("bioconductor-", "r-", "python-", "perl-") + +# Tools always reported if part of an env, independent of the env name +ALWAYS_REPORT = frozenset(["multiqc"]) + + +def _norm(name): + name = str(name).lower() + for prefix in PACKAGE_PREFIXES: + if name.startswith(prefix) and len(name) > len(prefix): + name = name[len(prefix) :] + break + return re.sub(r"[_\-.]", "", name) + + +def relevant_packages(env, packages): + """Reduce the packages of an env to the tools worth reporting + + Keeps the package the env is named after, plus every package MultiQC has a + module for, so dependencies like perl or matplotlib do not clutter the list. + + Parameters + ---------- + env : str + environment name as configured + packages : dict + package name to version as parsed from the env yaml + + Returns + ------- + dict + package name to version + """ + keep = dict() + nenv = _norm(env) + + for pkg, version in packages.items(): + npkg = _norm(pkg) + if npkg == nenv or npkg.startswith(nenv) or nenv.startswith(npkg): + keep[pkg] = version + elif multiqc_module(pkg) or pkg.lower() in ALWAYS_REPORT: + keep[pkg] = version + + return keep + + +def collect_versions(config, steps, envdir): + """Collect versions of all tools configured to run + + Parameters + ---------- + config : dict + config json file parsed by snakemake.common.configfile + steps : list + list of workflow steps that will be run + envdir : str + directory holding the conda env yamls + + Returns + ------- + list + rows of [tool, version, multiqc module, steps] sorted by tool + """ + logid = scriptname + ".collect_versions: " + tools = dict() + + for env, envsteps in envs_from_config(config, steps).items(): + yamlfile = None + for step in sorted(envsteps): + yamlfile = env_yaml(env, step, envdir) + if yamlfile: + break + if not yamlfile: + log.warning( + logid + "No conda env yaml found for " + str(env) + ", skipping version" + ) + tools.setdefault(env, ["unknown", set()])[1].update(envsteps) + continue + packages = relevant_packages(env, parse_env_yaml(yamlfile)) + if not packages: + log.warning( + logid + + "No version for " + + str(env) + + " deducible from " + + str(yamlfile) + ) + packages = {env: "unknown"} + # tools shipped by a differently named package, e.g. bbduk by bbmap, + # need their own MultiQC module to be reported + envmod = multiqc_module(env) + if envmod and envmod not in [multiqc_module(pkg) for pkg in packages]: + packages[env] = ( + list(packages.values())[0] if len(packages) == 1 else "unknown" + ) + for pkg, version in packages.items(): + tools.setdefault(pkg, [version, set()]) + tools[pkg][0] = version + tools[pkg][1].update(envsteps) + + rows = list() + for tool in sorted(tools): + version, envsteps = tools[tool] + rows.append( + [ + tool, + version, + multiqc_module(tool) or "-", + ",".join(sorted(envsteps)), + ] + ) + + log.debug(logid + str(rows)) + return rows + + +def write_versions(config, steps, envdir, monsda_version, configfile=None, path=None): + """Write versions of all configured tools to LOGS/versions.txt + + Parameters + ---------- + config : dict + config json file parsed by snakemake.common.configfile + steps : list + list of workflow steps that will be run + envdir : str + directory holding the conda env yamls + monsda_version : str + version of MONSDA + configfile : str, optional + config file name to note in the header + path : str, optional + output file, by default LOGS/versions.txt + + Returns + ------- + str + path of the versions file + """ + logid = scriptname + ".write_versions: " + path = path if path else os.path.join("LOGS", "versions.txt") + os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True) + + if not os.path.isdir(envdir): + fallback = os.path.join(os.path.dirname(os.path.dirname(__file__)), "envs") + log.warning( + logid + str(envdir) + " not found, falling back to " + str(fallback) + ) + envdir = fallback + + rows = collect_versions(config, steps, envdir) + + with open(path, "w") as out: + out.write( + "# MONSDA {v}\t{ts}\n".format( + v=monsda_version, + ts=datetime.datetime.now().strftime("%Y%m%d_%H_%M_%S"), + ) + ) + if configfile: + out.write("# config: {c}\n".format(c=str(configfile))) + out.write( + "# versions as pinned in the conda env yamls of the configured tools, not queried from the running binaries\n" + ) + out.write("#tool\tversion\tmultiqc_module\tsteps\n") + for row in rows: + out.write("\t".join(row) + "\n") + + log.info(logid + "Tool versions written to " + str(path)) + return path diff --git a/MONSDA/Workflows.py b/MONSDA/Workflows.py index 2206e7bb..4f84d67e 100755 --- a/MONSDA/Workflows.py +++ b/MONSDA/Workflows.py @@ -37,6 +37,8 @@ import MONSDA.Params as mp import MONSDA.Utils as mu +from MONSDA.Containers import normalize_container_version +from MONSDA.PostDE import load_postde_config, prepare_postde from MONSDA.Utils import check_run as check_run from MONSDA.Utils import dump_if_different, write_if_different @@ -45,7 +47,7 @@ installpath = os.path.dirname(__file__).replace( os.sep.join(["lib", pythonversion, "site-packages", "MONSDA"]), "share" ) -except: +except Exception: installpath = os.getcwd() workflowpath = os.path.join(installpath, "MONSDA", "workflows") @@ -56,7 +58,22 @@ # Matches oras:// NOT already followed by a registry host (i.e. followed directly by a # Docker Hub namespace like "jfallmann/" without dots before the first slash) oraspath = re.compile(r'oras://(?![\w.-]+\.[\w.-]+/)') -oras_registry = "docker.io" +oras_registry = "ghcr.io" +oras_default_namespace = "jfallmann/monsda" +oras_namespace = oras_default_namespace + + +def _version_oras_containers(content: str, version: str = None) -> str: + """Append the MONSDA version to each generated ORAS container reference.""" + tag = normalize_container_version(version) + lines = list() + for line in content.splitlines(keepends=True): + body = line.rstrip("\r\n") + ending = line[len(body):] + if "container" in body and "oras://" in body and f'-{tag}"' not in body: + body += f'+"-{tag}"' + lines.append(body + ending) + return "".join(lines) def set_oras_registry(registry: str): @@ -65,16 +82,261 @@ def set_oras_registry(registry: str): oras_registry = registry.strip().rstrip("/") +def set_oras_namespace(namespace: str): + """Set the ORAS namespace/repository used for container image URIs.""" + global oras_namespace + oras_namespace = namespace.strip().strip("/") + + +def _rewrite_oras(content: str) -> str: + """Version ORAS images and apply the configured namespace and registry.""" + content = _version_oras_containers(content) + if oras_namespace != oras_default_namespace: + content = content.replace( + "oras://" + oras_default_namespace, "oras://" + oras_namespace + ) + return re.sub(oraspath, "oras://" + oras_registry + "/", content) + + def _fix_oras(line: str) -> str: """Insert oras_registry into unqualified oras:// URIs.""" - return re.sub(oraspath, "oras://" + oras_registry + "/", line) + return _rewrite_oras(line) + + +_NF_KW = ("def ", "process", "workflow", "include", "params", "nextflow") + +_NF_PFX = "g" + + +def _nf_is_channel(rhs_list): + return any("Channel." in r for r in rhs_list) + + +_GA_LITERAL = re.compile(r"get_always\(\s*(['\"])((?:\\.|(?!\1)[^\\])*?)\1\s*\)") + + +def _nf_rw(text, scalars, channels, ex_s=None, ex_c=None): + """Rewrite bare references to global names: scalars -> params.gNAME, + channels -> NAME(). Interpolated forms handled as well. Literal + get_always('KEY')/get_always("KEY") lookups are preserved byte-exact; + dynamic get_always(KEY) and GString arguments are rewritten as usual.""" + s_active = set(scalars) - ({ex_s} if ex_s else set()) + c_active = set(channels) - ({ex_c} if ex_c else set()) + alln = s_active | c_active + if not alln: + return text + + ga = [] + + def _mask_ga(m): + if m.group(1) == '"' and "$" in m.group(2): + return m.group(0) + token = "\x00%d\x00" % len(ga) + ga.append(m.group(0)) + return token + + text = _GA_LITERAL.sub(_mask_ga, text) + + def repl(name, interp): + if name in c_active: + return ("${" + name + "()}") if interp else (name + "()") + return ( + ("${params." + _NF_PFX + name + "}") + if interp + else ("params." + _NF_PFX + name) + ) + + alt = "|".join(sorted(map(re.escape, alln), key=len, reverse=True)) + text = re.sub(r"\$\{(" + alt + r")\}", lambda m: repl(m.group(1), True), text) + text = re.sub(r"\$(" + alt + r")(?![\w(])", lambda m: repl(m.group(1), True), text) + text = re.sub( + r"(? 0: + buf.append(lines[i]) + depth += lines[i].count("{") - lines[i].count("}") + i += 1 + for k in range(start, i): + drop.add(k) + nm = None + for bl in buf: + mm = re.match(r"^\s*([A-Za-z_]\w*)\s*=(?!=)", bl) + if mm: + nm = mm.group(1) + break + if nm: + ifblocks.append((nm, buf)) + ifnames.append(nm) + continue + else: + drop.add(i) + i += 1 + + channels = set(nm for nm in order if _nf_is_channel(assigns[nm])) + scalars = (set(order) - channels) | set(ifnames) + + defs = [] + for name in order: + rhs_list = assigns[name] + if name in channels: + block_body = [] + for k, rhs in enumerate(rhs_list): + r = _nf_rw(rhs, scalars, channels, ex_c=name) + pre = " def %s = %s" if k == 0 else " %s = %s" + block_body.append(pre % (name, r)) + block_body.append(" return %s" % name) + defs.append("def %s(){\n%s\n}" % (name, "\n".join(block_body))) + elif len(rhs_list) == 1: + r = _nf_rw(rhs_list[0], scalars, channels, ex_s=name) + defs.append("params.%s%s = %s" % (_NF_PFX, name, r)) + else: + block_body = [] + for k, rhs in enumerate(rhs_list): + r = _nf_rw(rhs, scalars, channels, ex_s=name) + pre = "def %s = %s" if k == 0 else "%s = %s" + block_body.append(pre % (name, r)) + block_body.append("return %s" % name) + defs.append( + "params.%s%s = { %s }()" % (_NF_PFX, name, "; ".join(block_body)) + ) + + for nm, buf in ifblocks: + tb = [] + for bl in buf: + bl2 = re.sub(r"^(\s*)" + re.escape(nm) + r"\s*=\s*", r"\1return ", bl) + tb.append(bl2) + block = "\n".join(tb) + block = _nf_rw(block, scalars, channels, ex_s=nm) + defs.append("params.%s%s = {\n%s\n}()" % (_NF_PFX, nm, block)) + + kept = [ln for idx, ln in enumerate(lines) if idx not in drop] + body = "\n".join(kept) + + body = _nf_rw(body, scalars, channels - {"samples_ch"}) + if "samples_ch" in channels: + body = _nf_rewrite_samples_ch(body, scalars, channels) + + def_txt = "\n".join(defs) + kl = body.split("\n") + out_lines = [] + inserted = False + j = 0 + while j < len(kl): + out_lines.append(kl[j]) + if not inserted and kl[j].startswith("def get_always"): + k = j + while k < len(kl) and not kl[k].startswith("}"): + k += 1 + for x in range(j + 1, k + 1): + out_lines.append(kl[x]) + out_lines.append(def_txt) + j = k + 1 + inserted = True + continue + j += 1 + if not inserted: + out_lines = [def_txt] + kl + return "\n".join(out_lines) def _write_workflow(filepath, content): """Write workflow file with ORAS registry fix applied.""" - content = re.sub(oraspath, "oras://" + oras_registry + "/", content) + if str(filepath).endswith(".nf"): + content = nf_paramize(content) + content = _rewrite_oras(content) write_if_different(filepath, content) + +def _fill_template(path, loglevel, logfix, condapath, conda_repl, nfmode=False): + """Read a workflow template file, strip comments, and apply the standard + loglevel/conda/include substitutions. Returns the processed lines as a list.""" + marker = "include {" if nfmode else "include: " + out = list() + with open(path, "r") as fh: + for line in mu.comment_remover(fh.readlines()): + line = re.sub(logfix, "loglevel='" + loglevel + "'", line) + line = re.sub(condapath, conda_repl, line) + if marker in line: + if nfmode: + line = fixinclude( + line, loglevel, condapath, envpath, workflowpath, logfix, "nfmode" + ) + else: + line = fixinclude( + line, loglevel, condapath, envpath, workflowpath, logfix + ) + out.append(line) + return out + try: scriptname = ( os.path.basename(inspect.stack()[-1].filename) @@ -175,7 +437,7 @@ def get_combo(wfs, config, conditions): # Group all QC tools into a single combo position so pre-QC # (fastqc) and post-QC (rustqc) both appear in one combo name. if subwork == "QC" and len(tools) > 1: - qc_prio = {"fastqc": 0, "rustqc": 1} + qc_prio = {"fastqc": 0, "rastqc": 0, "rustqc": 1} tools = sorted( tools, key=lambda item: ( @@ -203,7 +465,7 @@ def get_processes(config): # Define workflow stages pre = ["QC", "FETCH", "BASECALL"] sub = ["TRIMMING", "MAPPING", "DEDUP", "QC"] - post = ["COUNTING", "TRACKS", "PEAKS", "DE", "DEU", "DAS", "DTU", "CIRCS"] + post = ["COUNTING", "TRACKS", "PEAKS", "DE", "DEU", "DAS", "DTU", "CIRCS", "FUSIONS"] wfs = [x.replace(" ", "") for x in config["WORKFLOWS"].split(",")] @@ -298,7 +560,7 @@ def create_subworkflow(config, subwork, conditions, envs=None, stage=None): for condition in list(set(conditions)): try: env = str(mu.sub_dict(config[subwork], condition)[stage + "ENV"]) - except: + except (KeyError, TypeError): if "TOOLS" not in config[subwork]: log.error( "No tool environment found for " @@ -310,7 +572,7 @@ def create_subworkflow(config, subwork, conditions, envs=None, stage=None): env = "" try: exe = str(mu.sub_dict(config[subwork], condition)[stage + "BIN"]) - except: + except (KeyError, TypeError): if "TOOLS" not in config[subwork]: log.error( "No tool binary found for " @@ -415,6 +677,7 @@ def create_subworkflow(config, subwork, conditions, envs=None, stage=None): "COUNTING", "TRACKS", "CIRCS", + "FUSIONS", ] ] ): @@ -454,6 +717,9 @@ def create_subworkflow(config, subwork, conditions, envs=None, stage=None): ) log.error("".join(tbe.format())) + if config.get("POSTDE", {}).get("enabled") and subwork == "DE": + tempconf["POSTDE"] = config["POSTDE"] + ( configs.append(tempconf) if tempconf.get("SETTINGS", False) @@ -494,15 +760,7 @@ def make_pre( add = list() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) for i in range(len(worklist)): log.debug( @@ -579,35 +837,12 @@ def make_pre( subname = toolenv + "_raw.smk" smkf = os.path.abspath(os.path.join(workflowpath, subname)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub( - logfix, "loglevel='" + loglevel + "'", line - ) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smko = os.path.abspath( os.path.join( @@ -645,16 +880,8 @@ def make_pre( for condition in conditions: add = list() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + add.append("\n\n") # Add variable for combination string add.append( @@ -726,28 +953,12 @@ def make_pre( ) # RuleThemAll for snakemake depending on chosen workflows smkf = os.path.abspath(os.path.join(workflowpath, subname)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smko = os.path.abspath( os.path.join( @@ -815,15 +1026,7 @@ def make_sub( add = list() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) for i in range(len(worklist)): log.debug( @@ -943,37 +1146,15 @@ def make_sub( subconf["PREDEDUP"] = "enabled" smkf = os.path.abspath(os.path.join(workflowpath, subname)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") - if "MAPPING" in works: + if "MAPPING" in works and envs[works.index("MAPPING")] not in [ + "piscem" + ]: smkf = os.path.abspath(os.path.join(workflowpath, "mapping.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") if "QC" in subworkflows: # Use rustqc-specific multiqc whenever rustqc is part of this combo qc_mqc = ( @@ -982,32 +1163,13 @@ def make_sub( else "multiqc.smk" ) smkf = os.path.abspath(os.path.join(workflowpath, qc_mqc)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") # Append footer and write out subsnake and subconf per condition smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smko = os.path.abspath( os.path.join( @@ -1035,16 +1197,8 @@ def make_sub( add = list() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + add.append("\n\n") # Add variable for combination string add.append( @@ -1136,31 +1290,17 @@ def make_sub( ) smkf = os.path.abspath(os.path.join(workflowpath, subname)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") if "MAPPING" in subworkflows: - smkf = os.path.abspath(os.path.join(workflowpath, "mapping.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) + _map_tools, _ = create_subworkflow(config, "MAPPING", [condition]) + _skip_postmap = _map_tools is not None and any( + str(t[0]) in ["piscem"] for t in _map_tools + ) + if not _skip_postmap: + smkf = os.path.abspath(os.path.join(workflowpath, "mapping.smk")) + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) subjobs.append("\n\n") if "QC" in subworkflows: # Use rustqc-specific multiqc whenever rustqc is part of this combo @@ -1172,32 +1312,12 @@ def make_sub( else "multiqc.smk" ) smkf = os.path.abspath(os.path.join(workflowpath, qc_mqc)) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smko = os.path.abspath( os.path.join(subdir, "_".join(["_".join(condition), "subsnake.smk"])) @@ -1230,6 +1350,16 @@ def make_post( log.debug(logid + f"STARTING POSTPROCESSING {postworkflow} FOR {conditions}") + if postworkflow == "DE": + postde_inputs = prepare_postde(config, subdir) + if postde_inputs: + config = dict(config) + config["POSTDE"] = { + "enabled": True, + "config": config["POSTDE"]["config"], + "inputs": postde_inputs, + } + jobs = list() condapath = re.compile(r'conda:\s+"') logfix = re.compile(r'loglevel="INFO"') @@ -1253,15 +1383,7 @@ def make_post( add = list() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) for i in range(len(envlist)): envs = envlist[i].split("-") @@ -1299,6 +1421,12 @@ def make_post( "CIRI2 needs BWA mapped files, will skip input produced otherwise" ) continue + if postworkflow == "FUSIONS": + if toolenv == "starfusion" and "star" not in envs: + log.warning( + "STAR-Fusion needs STAR chimeric output, will skip input produced otherwise" + ) + continue # Deep-merge postworkflow config from all conditions (preserves all condition paths) for cond in all_conditions: @@ -1365,38 +1493,13 @@ def make_post( os.path.abspath(os.path.join(workflowpath, toolenv)) + "_trim.smk" ) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) subjobs.append("\n\n") # Append footer and write out subsnake and subconf for all conditions smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "../', line) - if "include: " in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "../', nfmode=False)) + subjobs.append("\n\n") te = ( toolenv.split("_")[0] if "_" in toolenv else toolenv @@ -1442,15 +1545,7 @@ def make_post( add = list() subconf = mu.NestedDefaultDict() smkf = os.path.abspath(os.path.join(workflowpath, "header.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - add.append(line) + add.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) # Postprocessing runs across all conditions - collect configs from all conditions at once listoftools, listofconfigs = create_subworkflow( @@ -1517,27 +1612,13 @@ def make_post( smkf = ( os.path.abspath(os.path.join(workflowpath, toolenv)) + "_trim.smk" ) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") # Append footer and write out subsnake and subconf for all conditions smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") te = ( toolenv.split("_")[0] if "_" in toolenv else toolenv @@ -1623,16 +1704,8 @@ def make_summary(config, subdir, loglevel, combinations=None): subjobs.append("\n\n") smkf = os.path.abspath(os.path.join(workflowpath, "summary.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) + subjobs.append("\n\n") smkf = os.path.abspath(os.path.join(workflowpath, "footer.smk")) with open(smkf, "r") as smk: @@ -1665,10 +1738,10 @@ def rulethemall(subworkflows, config, loglevel, condapath, logfix, combo=""): else 'rule themall:\n\tinput:\texpand("MAPPED/{combo}/{file}_mapped_{type}_dedup.bam", combo=combo, file=samplecond(SAMPLES, config), type=["sorted", "sorted_unique"])' ) allqc = 'expand("QC/Multi/{combo}/{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' - allrawqc = 'rule themall:\n\tinput:\texpand("QC/Multi{combo}{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' - alltrimqc = 'rule themall:\n\tinput:\texpand("QC/Multi{combo}/{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' + allrawqc = 'rule themall:\n\tinput:\texpand("QC/Multi/{combo}/{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' + alltrimqc = 'rule themall:\n\tinput:\texpand("QC/Multi/{combo}/{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' alltrim = 'rule themall:\n\tinput: expand("TRIMMED_FASTQ/{combo}/{file}_{read}_trimmed.fastq.gz", combo=combo, file=samplecond(SAMPLES, config), read=["R1","R2"]) if paired == \'paired\' else expand("TRIMMED_FASTQ/{combo}/{file}_trimmed.fastq.gz", combo=combo, file=samplecond(SAMPLES, config))' - alldedupqc = 'rule themall:\n\tinput:\texpand("QC/Multi{combo}/{condition}/multiqc_report.html", combo=combo, condition=str.join(os.sep, conditiononly(SAMPLES[0], config)))' + alldedupqc = 'rule themall:\n\tinput:\texpand("QC/Multi/{combo}/{condition}/multiqc_report.html", combo=combo, condition=str.join(os.sep, conditiononly(SAMPLES[0], config)))' alldedup = 'rule themall:\n\tinput: expand("DEDUP_FASTQ/{combo}/{file}_{read}_dedup.fastq.gz", combo=combo, file=samplecond(SAMPLES, config), read=["R1","R2"]) if paired == \'paired\' else expand("DEDUP_FASTQ/{combo}/{file}_dedup.fastq.gz", combo=combo, file=samplecond(SAMPLES, config))' alltrimdedupqc = 'rule themall:\n\tinput:\texpand("QC/Multi/{combo}/{condition}/multiqc_report.html", condition=str.join(os.sep, conditiononly(SAMPLES[0], config)), combo=combo)' @@ -1696,15 +1769,7 @@ def rulethemall(subworkflows, config, loglevel, condapath, logfix, combo=""): log.debug(logid + "Simulated read trimming only!") mu.makeoutdir("TRIMMED_FASTQ") smkf = os.path.abspath(os.path.join(workflowpath, "simulatetrim.smk")) - with open(smkf, "r") as smk: - for line in mu.comment_remover(smk.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda: "' + envpath, line) - if "include: " in line: - line = fixinclude( - line, loglevel, condapath, envpath, workflowpath, logfix - ) - todos.append(line) + todos.extend(_fill_template(smkf, loglevel, logfix, condapath, 'conda: "' + envpath, nfmode=False)) todos.append("\n\n") if ( @@ -1746,7 +1811,7 @@ def fixinclude( toinclude = str.split(line)[-1].replace('"', "") toinclude = str.join(os.sep, [workflowpath, toinclude]) with open(toinclude, "r") as incl: - for line in incl.readlines(): + for line in mu.comment_remover(incl.readlines()): line = re.sub(logfix, "loglevel='" + loglevel + "'", line) line = re.sub(condapath, condaline + envpath, line) line = _fix_oras(line) @@ -2069,7 +2134,7 @@ def nf_fetch_params( if REF: REFERENCE = REF REFDIR = str(os.path.dirname(REFERENCE)) - if XENV in ["salmon", "kallisto"]: + if XENV in ["salmon", "kallisto", "simpleaf"]: IDX = XCONF.get("INDEX") if IDX: INDEX = IDX @@ -2145,7 +2210,7 @@ def nf_fetch_params( compstr = ",".join([i.split(":")[0] for i in comparison.split(",")]) retconf[x + "COMP"] = comparison retconf[x + "COMPS"] = compstr - pval = mp.get_cutoff_as_string(config, x, "pval") + pval = mp.get_cutoff_as_string(config, x, "padj") lfc = mp.get_cutoff_as_string(config, x, "lfc") retconf[x + "PVAL"] = pval retconf[x + "LFC"] = lfc @@ -2220,6 +2285,35 @@ def nf_fetch_params( retconf["CIRCSREFDIR"] = REFDIR retconf["CIRCSANNO"] = ANNOTATION + # FUSIONS Variables + if "FUSIONS" in config: + FUSCONF = mu.sub_dict(config["FUSIONS"], SETUP) + FUSBIN, FUSENV = mp.env_bin_from_config(config, "FUSIONS") + log.debug(logid + "FUSIONSCONFIG: " + str(SETUP) + "\t" + str(FUSCONF)) + REF = FUSCONF.get("REFERENCE") + if REF: + REFERENCE = REF + REFDIR = str(os.path.dirname(REFERENCE)) + FANNO = FUSCONF.get("ANNOTATION") + if FANNO: + ANNOTATION = FANNO + else: + ANNOTATION = ( + ANNO.get("GTF") if "GTF" in ANNO else ANNO.get("GFF") + ) # by default GTF format will be used + + retconf["FUSIONSREF"] = REFERENCE + retconf["FUSIONSREFDIR"] = REFDIR + retconf["FUSIONSANNO"] = ANNOTATION + retconf["FUSIONSLIB"] = FUSCONF.get(FUSENV, {}).get("OPTIONS", {}).get( + "INDEX", "" + ) or os.path.join(REFDIR, "CTAT", FUSENV) + + postde = config.get("POSTDE") + if postde and postde.get("enabled"): + retconf["POSTDE_ENABLED"] = True + retconf["POSTDE_INPUTS"] = postde.get("inputs", "") + retconf["REFERENCE"] = REFERENCE retconf["REFDIR"] = REFDIR retconf["IP"] = IP @@ -2353,94 +2447,7 @@ def nf_tool_params( @check_run def nf_get_processes(config): - logid = scriptname + ".Workflows_nf_get_processes: " - - preprocess = subworkflows = postprocess = [] - - # Define workflow stages - pre = ["QC", "FETCH", "BASECALL"] - sub = ["TRIMMING", "MAPPING", "QC", "DEDUP"] - post = ["COUNTING", "TRACKS", "PEAKS", "DE", "DEU", "DAS", "DTU", "CIRCS"] - - wfs = [x.replace(" ", "") for x in config["WORKFLOWS"].split(",")] - - if "WORKFLOWS" in config: - log.debug( - logid - + "CONFIG-WORKFLOWS: " - + str(wfs) - + "\t" - + str(pre) - + "\t" - + str(sub) - + "\t" - + str(post) - ) - subworkflows = [str(x) for x in wfs if x in sub] - log.debug(logid + "Sub: " + str(subworkflows)) - if len(subworkflows) == 0 or subworkflows[0] == "": - subworkflows = [] - preprocess = [x for x in wfs if x in pre] - if len(preprocess) == 0 or preprocess[0] == "": - preprocess = None - log.debug( - logid - + "Intermediate-WORKFLOWS: " - + str([preprocess, subworkflows, postprocess]) - ) - - if ( - subworkflows - and any(w in subworkflows for w in ["TRIMMING", "MAPPING", "DEDUP"]) - and preprocess - and "QC" in preprocess - ): - preprocess.remove("QC") - - if ( - preprocess - and "QC" in preprocess - and not any(w in subworkflows for w in ["TRIMMING", "MAPPING", "DEDUP"]) - ): - subworkflows.remove("QC") - - postprocess = [x for x in wfs if x in post] - if len(postprocess) == 0 or postprocess[0] == "": - postprocess = [] - else: - log.error("NO WORKFLOWS DEFINED, NOTHING TO DO!") - sys.exit() - - if preprocess: - try: - all([config[x] or x == "" for x in preprocess]) - except KeyError: - log.warning( - logid - + "Not all required preprocessing steps have configuration in the config file" - ) - - if subworkflows: - try: - all([config[x] or x == "TRIMMING" or x == "" for x in subworkflows]) - except KeyError: - log.warning( - logid - + "Not all required subworkflows have configuration in the config file" - ) - - if postprocess: - try: - all([config[x] or x == "" for x in postprocess]) - except KeyError: - log.warning( - logid - + "Not all required postprocessing steps have configuration in the config file" - ) - - log.debug(logid + "WORKFLOWS: " + str([preprocess, subworkflows, postprocess])) - - return [preprocess, subworkflows, postprocess] + return get_processes(config) @check_run @@ -2477,22 +2484,8 @@ def nf_make_pre( add = list() nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + add.append("\n\n") for i in range(len(worklist)): log.debug( @@ -2565,47 +2558,15 @@ def nf_make_pre( flowlist.append("MULTIQC") nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub( - logfix, "loglevel='" + loglevel + "'", line - ) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") if works[j] == "QC": nfi = os.path.abspath( os.path.join(workflowpath, "multiqc.nf") ) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub( - logfix, "loglevel='" + loglevel + "'", line - ) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") # workflow merger subjobs.append("\n\n" + "workflow {\n") @@ -2682,22 +2643,8 @@ def nf_make_pre( log.debug(logid + "PREPARING " + str(subwork) + " " + str(condition)) nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") listoftools, listofconfigs = create_subworkflow( config, subwork, [condition] @@ -2768,21 +2715,8 @@ def nf_make_pre( flowlist.append("BASECALL") nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") if "DEDUP" in works: flowlist.append("DEDUPBAM") @@ -2802,21 +2736,8 @@ def nf_make_pre( if subwork == "QC": flowlist.append("MULTIQC") nfi = os.path.abspath(os.path.join(workflowpath, "multiqc.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") # workflow merger log.debug("FLOWLIST: " + str(flowlist)) @@ -2899,22 +2820,8 @@ def nf_make_sub( add = list() nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + add.append("\n\n") for i in range(len(worklist)): log.debug( @@ -3045,21 +2952,8 @@ def nf_make_sub( subname = toolenv + "_dedup.nf" nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") tp.append( nf_tool_params( @@ -3088,40 +2982,15 @@ def nf_make_sub( "simulatetrim.nf", ) ) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") flowlist.append("MAPPING") + maprad = envs[works.index("MAPPING")] in ["piscem"] nfi = os.path.abspath(os.path.join(workflowpath, "mapping.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") if "DEDUP" in works: flowlist.append("DEDUPBAM") @@ -3137,21 +3006,8 @@ def nf_make_sub( else "multiqc.nf" ) nfi = os.path.abspath(os.path.join(workflowpath, _mqc_nf)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") # workflow merger log.debug("FLOWLIST: " + str(flowlist)) @@ -3197,9 +3053,10 @@ def nf_make_sub( subjobs.append(" " * 4 + w + "(TRIMMING.out.trimmed)\n") elif w == "MAPPING": subjobs.append(" " * 4 + w + "(TRIMMING.out.trimmed)\n") - subjobs.append( - " " * 4 + "POSTMAPPING(MAPPING.out.mapped)\n" - ) + if not maprad: + subjobs.append( + " " * 4 + "POSTMAPPING(MAPPING.out.mapped)\n" + ) elif w == "DEDUPBAM": if deduptool == "fgumi": subjobs.append( @@ -3423,39 +3280,13 @@ def nf_make_sub( flowlist.append("QC_DEDUP") subname = toolenv + ".nf" nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") subname = toolenv + "_dedup.nf" nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") tp.append( nf_tool_params( @@ -3471,45 +3302,23 @@ def nf_make_sub( ) if "MAPPING" in subworkflows: + _map_tools, _ = create_subworkflow(config, "MAPPING", [condition]) + maprad = _map_tools is not None and any( + str(t[0]) in ["piscem"] for t in _map_tools + ) if "QC" not in subworkflows: log.debug(logid + "Mapping without QC!") if "TRIMMING" not in subworkflows: log.debug(logid + "Simulated read trimming only!") flowlist.append("TRIMMING") nfi = os.path.abspath(os.path.join(workflowpath, "simulatetrim.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") flowlist.append("MAPPING") nfi = os.path.abspath(os.path.join(workflowpath, "mapping.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") if "QC" in subworkflows: flowlist.append("MULTIQC") @@ -3522,21 +3331,8 @@ def nf_make_sub( else "multiqc.nf" ) nfi = os.path.abspath(os.path.join(workflowpath, _mqc_nf)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") # workflow merger log.debug("FLOWLIST: " + str(flowlist)) @@ -3580,7 +3376,8 @@ def nf_make_sub( subjobs.append(" " * 4 + w + "(TRIMMING.out.trimmed)\n") elif w == "MAPPING": subjobs.append(" " * 4 + w + "(TRIMMING.out.trimmed)\n") - subjobs.append(" " * 4 + "POSTMAPPING(MAPPING.out.mapped)\n") + if not maprad: + subjobs.append(" " * 4 + "POSTMAPPING(MAPPING.out.mapped)\n") elif w == "DEDUPBAM": if deduptool == "fgumi": subjobs.append( @@ -3703,6 +3500,16 @@ def nf_make_post( log.debug(logid + f"STARTING POSTPROCESSING {postworkflow} FOR {conditions}") + if postworkflow == "DE": + postde_inputs = prepare_postde(config, subdir) + if postde_inputs: + config = dict(config) + config["POSTDE"] = { + "enabled": True, + "config": config["POSTDE"]["config"], + "inputs": postde_inputs, + } + jobs = list() condapath = re.compile(r'conda\s+"') includepath = re.compile(r'include:\s+"') @@ -3726,22 +3533,8 @@ def nf_make_post( add = list() nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + add.append("\n\n") for i in range(len(envlist)): envs = envlist[i].split("-") @@ -3777,6 +3570,12 @@ def nf_make_post( "CIRI2 needs BWA mapped files, will skip input produced otherwise" ) continue + if subwork == "FUSIONS": + if toolenv == "starfusion" and "star" not in envs: + log.warning( + "STAR-Fusion needs STAR chimeric output, will skip input produced otherwise" + ) + continue # Merge subwork config from all conditions for cond in all_conditions: @@ -3793,7 +3592,8 @@ def nf_make_post( # Append subwork suffix for all applicable postprocess workflows toolenv_orig = toolenv - toolenv = toolenv + "_" + subwork + if subwork in ["DE", "DEU", "DAS", "DTU"] and toolbin not in ["deseq", "diego"]: + toolenv = toolenv + "_" + subwork sconf[subwork + "ENV"] = toolenv sconf[subwork + "BIN"] = toolbin subsamples = mp.get_samples(sconf) @@ -3819,21 +3619,8 @@ def nf_make_post( subname = toolenv + ".nf" nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") tp.append( nf_tool_params( @@ -3909,22 +3696,8 @@ def nf_make_post( subwork = postworkflow add = list() nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - add.append(line) - add.append("\n\n") + add.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + add.append("\n\n") # Postprocessing runs across all conditions - collect configs from all conditions at once flowlist = list() @@ -3953,8 +3726,9 @@ def nf_make_post( subjobs = list() toolenv, toolbin = map(str, listoftools[a]) - # Append subwork suffix for all postprocessing workflows - toolenv = toolenv + "_" + subwork + # Append subwork suffix for all applicable postprocess workflows + if subwork in ["DE", "DEU", "DAS", "DTU"] and toolbin not in ["deseq", "diego"]: + toolenv = toolenv + "_" + subwork log.debug(logid + "toolenv: " + str(toolenv)) sconf[subwork + "ENV"] = toolenv @@ -3966,21 +3740,8 @@ def nf_make_post( subconf.update(sconf) subname = toolenv + ".nf" nfi = os.path.abspath(os.path.join(workflowpath, subname)) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") tp.append( nf_tool_params( @@ -4110,39 +3871,12 @@ def nf_make_summary(config, subdir, loglevel, combinations=None): subjobs = list() nfi = os.path.abspath(os.path.join(workflowpath, "header.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(logfix, "loglevel='" + loglevel + "'", line) - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") nfi = os.path.abspath(os.path.join(workflowpath, "summary.nf")) - with open(nfi, "r") as nf: - for line in mu.comment_remover(nf.readlines()): - line = re.sub(condapath, 'conda "' + envpath, line) - if "include {" in line: - line = fixinclude( - line, - loglevel, - condapath, - envpath, - workflowpath, - logfix, - "nfmode", - ) - subjobs.append(line) - subjobs.append("\n\n") + subjobs.extend(_fill_template(nfi, loglevel, logfix, condapath, 'conda "' + envpath, nfmode=True)) + subjobs.append("\n\n") nfo = os.path.abspath(os.path.join(subdir, "summary_subflow.nf")) _write_workflow(nfo, "".join(subjobs)) diff --git a/MONSDA/web_configurator.py b/MONSDA/web_configurator.py index ce964980..12759d14 100644 --- a/MONSDA/web_configurator.py +++ b/MONSDA/web_configurator.py @@ -12,11 +12,12 @@ from . import _version from .Params import samplesheet_to_settings +from .PostDE import load_postde_config __version__ = _version.get_versions()["version"] TEMPLATE_FILE = "template_base_commented.json" -NONE_WORKFLOW_KEYS = ["WORKFLOWS", "BINS", "MAXTHREADS", "SETTINGS", "VERSION"] +NONE_WORKFLOW_KEYS = ["WORKFLOWS", "BINS", "MAXTHREADS", "SETTINGS", "VERSION", "POSTDE"] app = FastAPI(title="MONSDA Configurator Web") @@ -41,6 +42,7 @@ class BuildConfigRequest(BaseModel): maxthreads: str = "16" settings: Optional[Dict[str, Any]] = None samplesheet_path: Optional[str] = None + postde: Optional[Dict[str, Any]] = None class ConditionFiles(BaseModel): @@ -243,6 +245,16 @@ def build_config(req: BuildConfigRequest) -> Dict[str, Any]: req.tools.get(wf, []), ) + if req.postde is None: + final_config["POSTDE"] = {"enabled": False, "config": ""} + else: + try: + final_config["POSTDE"] = load_postde_config( + req.postde, config=final_config + ) + except ValueError as err: + raise HTTPException(status_code=400, detail=str(err)) + return final_config @@ -355,9 +367,16 @@ def save_config(req: SaveConfigRequest) -> Dict[str, Any]: if not os.path.isdir(output_dir): raise HTTPException(status_code=400, detail="Output directory does not exist.") + config = copy.deepcopy(req.config) + if "POSTDE" in config: + try: + config["POSTDE"] = load_postde_config(config["POSTDE"], config=config) + except ValueError as err: + raise HTTPException(status_code=400, detail=str(err)) + path = os.path.join(output_dir, f"config_{config_name}.json") with open(path, "w") as fh: - json.dump(req.config, fh, indent=4) + json.dump(config, fh, indent=4) return {"message": "Config written.", "path": path} @@ -367,6 +386,15 @@ def create_project(req: ProjectRequest) -> Dict[str, Any]: base_dir = os.path.abspath(req.project_dir.strip()) project_name = req.project_name.strip() or "monsda" + # Validate POSTDE before any creation/writes + if req.config and "POSTDE" in req.config: + try: + req.config["POSTDE"] = load_postde_config( + req.config["POSTDE"], config=req.config + ) + except ValueError as err: + raise HTTPException(status_code=400, detail=str(err)) + # Project name becomes a subdirectory under the chosen path project_dir = os.path.join(base_dir, project_name) os.makedirs(project_dir, exist_ok=True) @@ -734,6 +762,16 @@ def root() -> str:
+
+ +
Runs enrichment/regulatory analysis on DE results. DESeq2/edgeR only, requires the DE workflow.
+
+ + +
+
@@ -946,7 +984,7 @@ def root() -> str: holder.innerHTML = ''; const workflows = Object.keys(data) - .filter(k => !['WORKFLOWS','BINS','MAXTHREADS','SETTINGS','VERSION'].includes(k)) + .filter(k => !['WORKFLOWS','BINS','MAXTHREADS','SETTINGS','VERSION','POSTDE'].includes(k)) .sort(); workflows.forEach(wf => { @@ -1256,7 +1294,11 @@ def root() -> str: tools: tools, maxthreads: document.getElementById('maxthreads').value.trim(), settings: settings, - samplesheet_path: document.getElementById('samplesheetPath').value.trim() || null + samplesheet_path: document.getElementById('samplesheetPath').value.trim() || null, + postde: { + enabled: document.getElementById('postdeEnabled').checked, + config: document.getElementById('postdeConfig').value.trim() + } }; const r = await fetch('/config/preview', { @@ -1436,23 +1478,23 @@ def root() -> str:
- +
- +
- +
- +
- +
- +
diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..869d272d --- /dev/null +++ b/Makefile @@ -0,0 +1,96 @@ +.PHONY: help install install-dev lint lint-check format test test-cov build clean clean-build clean-pyc pre-commit deploy sync-config-versions check-config-versions + +help: + @echo "MONSDA Development Commands:" + @echo "" + @echo "Setup:" + @echo " make install Install package in development mode" + @echo " make install-dev Install with development dependencies" + @echo "" + @echo "Code Quality:" + @echo " make lint Run isort and flake8 linters" + @echo " make lint-check Check linting without fixing (CI mode)" + @echo " make format Format code with isort" + @echo " make pre-commit Run pre-commit hooks" + @echo "" + @echo "Testing:" + @echo " make test Run unit tests" + @echo " make test-cov Run tests with coverage report" + @echo " make test-integration Run integration tests (requires conda)" + @echo "" + @echo "Versioning:" + @echo " make sync-config-versions Stamp config VERSION fields with the git-tag version" + @echo " make check-config-versions Verify config VERSION fields match (CI mode)" + @echo "" + @echo "Building:" + @echo " make build Build distribution packages" + @echo "" + @echo "Cleanup:" + @echo " make clean Remove all build/test artifacts" + @echo " make clean-build Remove build artifacts" + @echo " make clean-pyc Remove Python cache files" + @echo "" + +install: + python -m pip install --upgrade pip + python -m pip install -e . + +install-dev: install + python -m pip install pytest pytest-cov flake8 isort pre-commit build wheel twine + +lint: + isort MONSDA/ tests/ + flake8 MONSDA/ tests/ + +lint-check: + isort --check-only --diff MONSDA/ tests/ + flake8 MONSDA/ tests/ --count --statistics --show-source + +format: + isort MONSDA/ tests/ + +test: + pytest tests/test_Utils.py -v + +test-cov: + pytest tests/test_Utils.py -v --cov=MONSDA --cov-report=term-missing --cov-report=html + +test-integration: + cd tests && RUN_INTEGRATION_TESTS=1 bash cicd_test.sh + +sync-config-versions: + python scripts/update_config_versions.py configs/*.json tests/data/config_*.json + +check-config-versions: + python scripts/update_config_versions.py --check configs/*.json tests/data/config_*.json + +build: clean sync-config-versions + python -m pip install --upgrade pip + python -m pip install build wheel setuptools versioneer + python -m build + python -m pip install twine + twine check dist/* + +clean: clean-build clean-pyc + +clean-build: + rm -rf build/ + rm -rf dist/ + rm -rf *.egg-info + rm -rf .eggs/ + rm -rf *.egg + rm -rf .pytest_cache/ + rm -rf .coverage + rm -rf htmlcov/ + +clean-pyc: + find . -type f -name '*.py[cod]' -delete + find . -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null || true + find . -type d -name '*.egg-info' -exec rm -rf {} + 2>/dev/null || true + +pre-commit: + pre-commit install + pre-commit run --all-files + +deploy: build + python -m twine upload dist/* diff --git a/README.md b/README.md index 9271ea91..915dfe27 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,30 @@ monsda --nextflow -j ${THREADS} --configfile ${CONFIG}.json --directory ${PWD} ``` and add additional arguments for ```Nextflow``` as you see fit. +### Rerunning on an existing analysis while skipping specific steps + +When rerunning MONSDA on an already-existing analysis directory, keep the full, +original ```WORKFLOWS``` list in the config (e.g. keep ```QC```, ```TRIMMING```, +```MAPPING```, ```DEDUP``` listed alongside ```DE```/```DEU```/```DAS```/```DTU```) +even if you only care about one of the later steps. ```MONSDA``` derives the +```combo```/```scombo``` tool-chain names (used to locate existing output +directories, e.g. ```MAPPED/fastqc-bbduk-picard-bwa/...```) from which stages +are listed in ```WORKFLOWS```; dropping an earlier stage such as ```QC``` from +the list changes those names and makes ```Snakemake|Nextflow``` treat the +existing outputs as missing, forcing a full rerun from scratch. + +To still skip execution of a specific rule (e.g. avoid rerunning ```multiqc``` +against outputs generated by an older MONSDA version) without removing it from +```WORKFLOWS```, pass Snakemake's own ```--omit-from``` as an additional +argument: + +``` +monsda -j ${THREADS} --configfile ${CONFIG}.json --directory ${PWD} --omit-from multiqc +``` + +This prevents execution of the given rule (and anything downstream of it) +while leaving the rest of the DAG, and the already-up-to-date outputs it +depends on, untouched. ### Run on workload manager @@ -129,6 +153,39 @@ To also run the heavier pipeline smoke test from the helper script: RUN_INTEGRATION_TESTS=1 bash tests/cicd_test.sh ``` +## Troubleshooting + +### `gzip`/`zcat: No such file or directory` for a `.gz` reference when running in a container (apptainer/singularity) + +MONSDA links `.bgz` reference files (genome, GTF/GFF, decoy) into `GENOMES/` +with a `.gz` sibling symlink, so tools that expect the `.gz` extension keep +working (bgzip output is gzip compatible). This symlink chain typically looks +like: + +``` +GENOMES/hg38p14_softmasked.fa.gz -> hg38p14_softmasked.fa.bgz -> /path/to/original/hg38p14_softmasked.fa.bgz +``` + +The final target is an absolute path outside the run/project directory, +pointing at wherever the reference was originally located. When a rule runs +inside `apptainer`/`singularity`, only the run/project directory (and any +paths you explicitly bind) are visible inside the container. If the original +reference lives elsewhere on the host, resolving the symlink chain inside the +container fails with an error such as: + +``` +gzip: GENOMES/hg38p14_softmasked.fa.gz: No such file or directory +``` + +even though the file exists and is readable outside the container. + +**Fix:** bind-mount the directory that holds the original reference file(s) +into the container, in addition to the run directory, e.g. via +`--singularity-args`/`--use-apptainer` bind options passed to snakemake, or +the equivalent nextflow container options, so the symlink target is reachable +from inside the container. For example, if your reference lives under +`/scratch/arctic/GENOMES/human`, add that path (or its parent) as an +additional bind path for the container run. ## Contribute If you like this project, are missing features, want to contribute or @@ -144,4 +201,3 @@ workflows directory. Do not forget to also extend the template.json and add some documentation. PRs always welcome. - diff --git a/configs/postde_analysis.json b/configs/postde_analysis.json new file mode 100644 index 00000000..90f96c7a --- /dev/null +++ b/configs/postde_analysis.json @@ -0,0 +1,52 @@ +{ + "padj": 0.05, + "lfc": 1, + "seed": 42, + "min_size": 10, + "max_size": 500, + "gprofiler": { + "enabled": true, + "organism": "hsapiens", + "domain_scope": "custom", + "allow_network": true, + "sources": ["GO:BP", "GO:MF", "GO:CC", "REAC"], + "correction_method": "g_SCS", + "background": null + }, + "clusterprofiler": { + "enabled": false, + "go_expand": false, + "term2gene": null, + "term2name": null, + "background": null + }, + "gsva": { + "enabled": false, + "method": "gsva" + }, + "decoupler": { + "enabled": false, + "resource": "collectri", + "organism": "human", + "top": 500, + "min_size": 10, + "methods": ["ulm", "mlm"], + "contrast_activity": false, + "allow_network": false, + "network": null + }, + "dream": { + "enabled": false, + "formula": "~condition + (1|subject)", + "metadata": null + }, + "plots": { + "enabled": false + }, + "report": { + "enabled": false + }, + "shiny": { + "enabled": false + } +} diff --git a/configs/template.json b/configs/template.json index ad52f0ed..1ed68f3d 100644 --- a/configs/template.json +++ b/configs/template.json @@ -1,5 +1,5 @@ { - "WORKFLOWS": "", #which workflows do you want to run "MAPPING, QC, DEDUP, TRIMMING, COUNTING, TRACKS, PEAKS, DE, DEU, DAS, DTU, CIRCS" + "WORKFLOWS": "", #which workflows do you want to run "MAPPING, QC, DEDUP, TRIMMING, COUNTING, TRACKS, PEAKS, DE, DEU, DAS, DTU, CIRCS, FUSIONS" "BINS": "", #where to find customscripts used in the workflow !!!ADVANCED USAGE ONLY!!! "MAXTHREADS": "20", #maximum number of cores to use, make sure this fits your needs "VERSION": "1.4.0", #needs to be identical to the installed version for reproducibility reasons @@ -157,6 +157,7 @@ "MAPPING": { #options for mapping for each sample/condition "TOOLS": { "star": "STAR", + "rustar": "rustar", "segemehl3": "segemehl.x", "segemehl": "segemehl.x", "hisat2": "hisat2", @@ -179,6 +180,14 @@ "EXTENSION" : "" } }, + "rustar":{ + "OPTIONS": # drop-in STAR replacement (rustar-aligner), same options as star + { + "INDEX" : "--sjdbGTFfeatureExon exon --sjdbGTFtagExonParentTranscript Parent --genomeSAindexNbases 13", #indexing options + "MAP": "--sjdbGTFfeatureExon exon --sjdbGTFtagExonParentTranscript Parent --outSAMprimaryFlag AllBestScore --outSAMattributes NH HI NM MD AS nM jM jI XS", #mapping options + "EXTENSION" : "" + } + }, "segemehl3": { "OPTIONS": { @@ -242,7 +251,7 @@ "OPTIONS": { "INDEX": "", # salmon index options - "QUANT": "--validateMappings --seqBias --gcBias", # salmon quant options + "QUANT": "--seqBias --gcBias", # salmon quant options } } } @@ -440,7 +449,8 @@ "TOOLS" : { "dexseq" : "Analysis/DTU/DEXSEQ.R", - "drimseq" : "Analysis/DTU/DRIMSEQ.R" + "drimseq" : "Analysis/DTU/DRIMSEQ.R", + "spit" : "Analysis/DTU/SPIT.py" }, "COMPARABLE" : { @@ -463,7 +473,7 @@ "OPTIONS": { "INDEX": "", # salmon index options - "QUANT": "--validateMappings --seqBias --gcBias", # salmon quant options + "QUANT": "--seqBias --gcBias", # salmon quant options "DTU": "" # Options for Analysis } }, @@ -472,7 +482,16 @@ "OPTIONS": { "INDEX": "", # salmon index options - "QUANT": "--validateMappings --seqBias --gcBias", # salmon quant options + "QUANT": "--seqBias --gcBias", # salmon quant options + "DTU": "" # Options for Analysis + } + }, + "spit": + { + "OPTIONS": + { + "INDEX": "", # salmon index options + "QUANT": "--seqBias --gcBias", # salmon quant options "DTU": "" # Options for Analysis } } @@ -499,5 +518,27 @@ } } } + }, + "FUSIONS" : { + "TOOLS" : + { + "starfusion" : "STAR-Fusion" + }, + "id": { + "condition": { + "setting": { + "ENV" : "starfusion", + "BIN" : "STAR-Fusion", + "starfusion":{ + "OPTIONS": + { + "INDEX": "PATH_TO_CTAT_genome_lib_dir", # Path to an existing CTAT genome resource library, will be built from REFERENCE/ANNOTATION if not present + "BUILD": "", # Additional options for prep_genome_lib.pl when building the CTAT library + "FUSION": "" # Additional options for STAR-Fusion; set "FASTQ": "true" to run in FASTQ mode instead of consuming STAR chimeric junctions + } + } + } + } + } } } diff --git a/configs/template_base_commented.json b/configs/template_base_commented.json index 93928bb8..459376bb 100644 --- a/configs/template_base_commented.json +++ b/configs/template_base_commented.json @@ -1,26 +1,31 @@ { "comment":{ - "WORKFLOWS": "Select workflows you want to run, possible values: FETCH, MAPPING, QC, DEDUP, TRIMMING, COUNTING, TRACKS, PEAKS, DE, DEU, DAS, DTU, CIRCS", + "WORKFLOWS": "Select workflows you want to run, possible values: FETCH, MAPPING, QC, DEDUP, TRIMMING, COUNTING, TRACKS, PEAKS, DE, DEU, DAS, DTU, CIRCS, FUSIONS", "BINS": "Where to find the scripts used in the workflow, per default the scripts directory that comes with MONSDA. ONLY CHANGE IF YOU KNOW WHAT YOU ARE DOING!", "MAXTHREADS": "Maximum number of cores to use, make sure this fits your needs, acts as upper cap.", - "VERSION": "Version of MONSDA to use, needs to match installed version" + "VERSION": "Version of MONSDA to use, needs to match installed version", + "POSTDE": "Optional post-DE enrichment/regulatory analysis (GO/GSEA/GSVA/decoupleR/dream). Disabled by default. When enabled, 'config' must point to a PostDE analysis JSON (see configs/postde_analysis.json). Only supported with the DE workflow and the deseq2/edger DE tools." }, "WORKFLOWS": "", "BINS": "MONSDA/scripts", "MAXTHREADS": "", "VERSION": "", + "POSTDE": { + "enabled": false, + "config": "" + }, "SETTINGS": { "comment": { "GROUPS": "Please enter a comma separated list of the groups your samples belong to. This does not have to be in any relation with the condition-tree but is used solely to establish contrasts for differential analysis. Each sample needs a matching entry. Can be left blank to run all-vs-all comparison.", "TYPES": "Please enter a comma separated list for this setting matching your sample list, i.e. each sample should have a matching entry. Can be left blank.", "BATCHES": "Please enter a comma separated list for this setting matching your sample list, i.e. each sample should have a matching entry. Can be left blank.", "SEQUENCING": "single or paired or singlecell and stranded info [rf,fr,ff,unstranded (or empty)] comma separated", - "REFERENCE": "set path to reference, '.fa.gz' ending expected", + "REFERENCE": "set path to reference, '.fa', '.fa.gz' or '.fa.bgz' ending expected", "DECOY": "set path to decoy file if tool for mapping/counting needs decoys, like e.g. salmon", "INDEX": "set path to index, default index to use for mapping with this settings", "PREFIX": "if needed, prefix for mapping software can be set here", - "GTF": "set path to gtf, '.gtf.gz' ending expected, THIS WILL BE USED WHENEVER POSSIBLE OR NOT DEFINED OTHERWISE", - "GFF": "set path to gff, '.gff.gz' ending expected. THIS WILL BE USED WHEN GTF IS NOT AVAILABLE BUT ANNOTATION IS REQUIRED OR IF TOOL REQUIRES GFF FORMAT", + "GTF": "set path to gtf, '.gtf', '.gtf.gz' or '.gtf.bgz' ending expected, THIS WILL BE USED WHENEVER POSSIBLE OR NOT DEFINED OTHERWISE", + "GFF": "set path to gff, '.gff', '.gff.gz' or '.gff.bgz' ending expected. THIS WILL BE USED WHEN GTF IS NOT AVAILABLE BUT ANNOTATION IS REQUIRED OR IF TOOL REQUIRES GFF FORMAT", "IP": "OPTIONAL if PEAKS is run and files need specific processing, eg. for iCLIP protocol" }, "SAMPLES": [], @@ -207,7 +212,10 @@ "hisat2": "hisat2", "bwa": "bwa mem", "bwameth": "bwameth.py", - "minimap": "minimap2" + "minimap": "minimap2", + "salmonalign": "salmon", + "rammap": "rammap", + "piscem": "piscem" }, "ENV" : "", "BIN" : "", @@ -286,20 +294,53 @@ "minimap": { "comment": { "INDEX" : "Indexing options", - "MAP": "Mapping options", + "MAP": "Mapping options, for long reads set the appropriate preset, e.g. -x map-ont (Nanopore) or -x map-pb (PacBio)", "EXTENSION" : "Prefix for index name, can be left empty." }, "OPTIONS": { "INDEX" : "", - "MAP": "", + "MAP": "-x map-ont", "EXTENSION" : "" } }, + "salmonalign": { + "comment": { + "INDEX" : "salmon index options, uses the salmon environment/container", + "MAP": "salmon quant options; salmon is run as a selective-alignment mapper (--writeMappings) emitting SAM for downstream postprocessing" + }, + "OPTIONS": { + "INDEX" : "", + "MAP": "" + } + }, + "rammap": { + "comment": { + "INDEX" : "minimap2 index options (rammap wraps a minimap2 index)", + "MAP": "rammap mapping options for long-read alignment" + }, + "OPTIONS": { + "INDEX" : "", + "MAP": "" + } + }, + "piscem": { + "comment": { + "INDEX" : "piscem build options (single-cell only)", + "MAP": "piscem map-sc options, requires single-cell read setup (R1 = barcode+UMI, R2 = cDNA)" + }, + "OPTIONS": { + "INDEX" : "", + "MAP": "" + } + }, }, "COUNTING": { "TOOLS": { "countreads": "featureCounts", - "salmon": "salmon" + "salmon": "salmon", + "oarfish": "oarfish", + "alevinfry": "alevin-fry", + "simpleaf": "simpleaf" }, "ENV" : "", "BIN" : "", @@ -318,12 +359,45 @@ "salmon": { "comment": { "INDEX": "salmon index options", - "QUANT": "salmon quant options" + "QUANT": "salmon quant options", + "BAM": "Optional: quantify from an aligned BAM instead of FASTQ. Leave empty for standard FASTQ mode. Set to 'transcriptome' for transcriptome-aligned BAMs or 'genome' for genome-aligned BAMs (requires GTF annotation)." + }, + "OPTIONS": + { + "INDEX": "", + "QUANT": "--seqBias --gcBias", + "BAM": "" + } + }, + "oarfish": { + "comment": { + "COUNT": "oarfish options for long-read transcript quantification from a transcriptome-aligned BAM" + }, + "OPTIONS": + { + "COUNT": "" + } + }, + "alevinfry": { + "comment": { + "PERMIT": "alevin-fry generate-permit-list options (single-cell only, consumes piscem RAD output)", + "COUNT": "alevin-fry quant options" + }, + "OPTIONS": + { + "PERMIT": "-d fw --unfiltered-pl", + "COUNT": "-r cr-like --use-mtx" + } + }, + "simpleaf": { + "comment": { + "INDEX": "simpleaf index options (single-cell, self-contained index + quant from FASTQ)", + "COUNT": "simpleaf quant options" }, "OPTIONS": { "INDEX": "", - "QUANT": "--validateMappings --seqBias --gcBias" + "COUNT": "" } } }, @@ -404,7 +478,7 @@ ] }, "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "deseq2": { @@ -442,7 +516,7 @@ ] }, "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "dexseq": { @@ -480,7 +554,7 @@ ] }, "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "diego": { @@ -507,7 +581,8 @@ "DTU": { "TOOLS": { "dexseq": "Analysis/DTU/DEXSEQ.R", - "drimseq": "Analysis/DTU/DRIMSEQ.R" + "drimseq": "Analysis/DTU/DRIMSEQ.R", + "spit": "Analysis/DTU/SPIT.py" }, "ENV" : "", "BIN" : "", @@ -518,18 +593,19 @@ ] }, "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "drimseq": { "comment": { "INDEX": "salmon index options, e.g. path to decoy file", "QUANT": "salmon quant options", - "DTU": "Options for DTU Analysis, leave blank unless you know exactly what you are doing, can be used to set cutoffs for filtering, e.g. min_samps_feature_expr, min_gene_expr ..." + "DTU": "Options for DTU Analysis, leave blank unless you know exactly what you are doing, can be used to set cutoffs for filtering, e.g. min_samps_feature_expr, min_gene_expr ...", + "TERMINUS": "Optional: enable terminus transcript grouping prior to DTU. Leave blank/absent to disable (default). Set terminus group options here (e.g. -m 0.1) to enable." }, "OPTIONS": { "INDEX": "-d GENOMES/salmon_decoy", - "QUANT": "--validateMappings --seqBias --gcBias", + "QUANT": "--seqBias --gcBias", "DTU": "" } }, @@ -537,11 +613,25 @@ "comment": { "INDEX": "salmon index options, e.g. path to decoy file", "QUANT": "salmon quant options", - "DTU": "Options for DTU Analysis, leave blank unless you know exactly what you are doing, , can be used to set cutoffs for filtering, e.g. min_samps_feature_expr, min_gene_expr ..." + "DTU": "Options for DTU Analysis, leave blank unless you know exactly what you are doing, , can be used to set cutoffs for filtering, e.g. min_samps_feature_expr, min_gene_expr ...", + "TERMINUS": "Optional: enable terminus transcript grouping prior to DTU. Leave blank/absent to disable (default). Set terminus group options here (e.g. -m 0.1) to enable." + }, + "OPTIONS": { + "INDEX": "-d GENOMES/salmon_decoy", + "QUANT": "--seqBias --gcBias", + "DTU": "" + } + }, + "spit": { + "comment": { + "INDEX": "salmon index options, e.g. path to decoy file", + "QUANT": "salmon quant options", + "DTU": "Options for DTU Analysis, leave blank unless you know exactly what you are doing, can be used to set cutoffs for filtering, e.g. min_samps_feature_expr, min_gene_expr ...", + "TERMINUS": "Optional: enable terminus transcript grouping prior to DTU. Leave blank/absent to disable (default). Set terminus group options here (e.g. -m 0.1) to enable." }, "OPTIONS": { "INDEX": "-d GENOMES/salmon_decoy", - "QUANT": "--validateMappings --seqBias --gcBias", + "QUANT": "--seqBias --gcBias", "DTU": "" } } diff --git a/configs/template_clean.json b/configs/template_clean.json index 78ea39f1..698e2132 100644 --- a/configs/template_clean.json +++ b/configs/template_clean.json @@ -140,7 +140,7 @@ } } }, - "MAP": { + "MAPPING": { "TOOLS": { "star": "STAR", "segemehl3": "segemehl.x", @@ -233,7 +233,7 @@ "OPTIONS": { "INDEX": "", - "QUANT": "--validateMappings --seqBias --gcBias" + "QUANT": "--seqBias --gcBias" } } } @@ -306,7 +306,7 @@ }, "EXCLUDE": [], "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "id": { @@ -345,7 +345,7 @@ }, "EXCLUDE": [], "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "id": { @@ -386,7 +386,7 @@ }, "EXCLUDE": [], "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "id": { @@ -413,7 +413,8 @@ "DTU": { "TOOLS": { "dexseq": "Analysis/DTU/DEXSEQ.R", - "drimseq": "Analysis/DTU/DRIMSEQ.R" + "drimseq": "Analysis/DTU/DRIMSEQ.R", + "spit": "Analysis/DTU/SPIT.py" }, "COMPARABLE": { "WTvsKO": [ @@ -427,7 +428,7 @@ }, "EXCLUDE": [], "CUTOFFS": { - "pvalue": "0.05", + "padj": "0.05", "lfc": "1.5" }, "id": { @@ -437,7 +438,7 @@ "OPTIONS": { "INDEX": "", - "QUANT": "--validateMappings --seqBias --gcBias", + "QUANT": "--seqBias --gcBias", "DTU": "" } }, @@ -446,7 +447,16 @@ "OPTIONS": { "INDEX": "", - "QUANT": "--validateMappings --seqBias --gcBias", + "QUANT": "--seqBias --gcBias", + "DTU": "" + } + }, + "spit": + { + "OPTIONS": + { + "INDEX": "", + "QUANT": "--seqBias --gcBias", "DTU": "" } } @@ -472,4 +482,27 @@ } } } + }, + "FUSIONS" : { + "TOOLS" : + { + "starfusion" : "STAR-Fusion" + }, + "id": { + "condition": { + "setting": { + "ENV" : "starfusion", + "BIN" : "STAR-Fusion", + "starfusion":{ + "OPTIONS": + { + "INDEX": "PATH_TO_CTAT_genome_lib_dir", + "BUILD": "", + "FUSION": "" + } + } + } + } + } + } } diff --git a/configs/template_umicollapse.json b/configs/template_umicollapse.json index cff24128..e264259b 100644 --- a/configs/template_umicollapse.json +++ b/configs/template_umicollapse.json @@ -109,7 +109,7 @@ "salmon": { "OPTIONS": { "INDEX": "-k 31", - "COUNTING": "-l U --validateMappings" + "COUNTING": "-l U" } } } diff --git a/configs/tutorial_exhaustive.json b/configs/tutorial_exhaustive.json index da2cecac..8ad239ac 100644 --- a/configs/tutorial_exhaustive.json +++ b/configs/tutorial_exhaustive.json @@ -504,7 +504,8 @@ "TOOLS" : { "dexseq" : "Analysis/DTU/DEXSEQ.R", - "drimseq" : "Analysis/DTU/DRIMSEQ.R" + "drimseq" : "Analysis/DTU/DRIMSEQ.R", + "spit" : "Analysis/DTU/SPIT.py" }, "COMPARABLE" : { @@ -535,6 +536,17 @@ "QUANT": "", # Options for read counting, independent of COUNTS workflows "DTU": "min_samps_feature_expr = 1, min_feature_expr = .1, min_samps_gene_expr = 1, min_gene_expr = 1" # Options for DE Analysis } + }, + "spit": { + "REFERENCE": "GENOMES/Ecoli/Ecoli_trans.fa.gz", + "ANNOTATION": "GENOMES/Ecoli/Ecoli_trans_fix.gtf.gz", + "DECOY": "GENOMES/Ecoli/salmon_decoy", + "OPTIONS": + { + "INDEX": "", + "QUANT": "", # Options for read counting, independent of COUNTS workflows + "DTU": "" # Options for DE Analysis + } } } }, @@ -560,6 +572,17 @@ "QUANT": "", # Options for read counting, independent of COUNTS workflows "DTU": "min_samps_feature_expr = 1, min_feature_expr = .1, min_samps_gene_expr = 1, min_gene_expr = 1" # Options for DE Analysis } + }, + "spit": { + "REFERENCE": "GENOMES/Ecoli/Ecoli_trans.fa.gz", + "ANNOTATION": "GENOMES/Ecoli/Ecoli_trans_fix.gtf.gz", + "DECOY": "GENOMES/Ecoli/salmon_decoy", + "OPTIONS": + { + "INDEX": "", + "QUANT": "", # Options for read counting, independent of COUNTS workflows + "DTU": "" # Options for DE Analysis + } } } } diff --git a/containers/apptainer/Container_skeleton.def b/containers/apptainer/Container_skeleton.def index 31db54ad..f3016ffb 100644 --- a/containers/apptainer/Container_skeleton.def +++ b/containers/apptainer/Container_skeleton.def @@ -1,5 +1,5 @@ Bootstrap: docker -From: continuumio/miniconda3 +From: continuumio/miniconda3:latest %files ##environment.yaml## @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x ##environment_short.yaml## diff --git a/containers/apptainer/Convert_yaml_to_def.sh b/containers/apptainer/Convert_yaml_to_def.sh old mode 100644 new mode 100755 index 9704f151..eb6bffa5 --- a/containers/apptainer/Convert_yaml_to_def.sh +++ b/containers/apptainer/Convert_yaml_to_def.sh @@ -1,10 +1,24 @@ -# zsh -i Convert_yaml_to_def.sh +#!/usr/bin/env bash +set -uo pipefail -for i in ${HOME}/MONSDA/envs/*.yaml;do a=${i%.yaml};b=${a##*/};cat ${HOME}/MONSDA/containers/apptainer/Container_skeleton.def |sed -e "s,##environment.yaml##,${i} /opt/envs/,g" -e "s,##environment_short.yaml##,/opt/envs/${b}.yaml,g" -e "s/##ENV_NAME##/${b}/g" > ${b}.def;done -sed -i 's,=base,=monsda_base,g' base.def -conda activate apptainer -mkdir -p TMP_BUILDDIR -rm -f build.log;for i in *.def;do a=${i%.def};APPTAINER_TMPDIR="TMP_BUILDDIR"; apptainer build --force --fakeroot ${a}.sif ${i} &>> build.log;done -rm -rf TMP_BUILDDIR +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -# for i in bbduk.sif bbmap.sif bwa2.sif bwameth.sif bwa.sif ciri2.sif cutadapt.sif deseq2_DE.sif dexseq_DEU.sif diego_DAS.sif dorado.sif drimseq_DTU.sif edger_DE.sif fastp.sif fastqc.sif guppy.sif hisat2.sif index.sif kallisto.sif macs.sif minimap.sif perl.sif picard.sif piranha.sif qc.sif salmon.sif samtools.sif scyphy.sif segemehl3.sif segemehl.sif sra.sif star.sif summary.sif test.sif trimgalore.sif trimmomatic.sif trnascan.sif ucsc.sif umitools.sif zip.sif ;do a=${i%.sif};apptainer push ${i} oras://docker.io/jfallmann/monsda:${a};done +for i in ${HOME}/MONSDA/envs/*.yaml; do + a=${i%.yaml} + b=${a##*/} + sed -e "s,##environment.yaml##,${i} /opt/envs/,g" -e "s,##environment_short.yaml##,/opt/envs/${b}.yaml,g" -e "s/##ENV_NAME##/${b}/g" "$script_dir/Container_skeleton.def" > "$script_dir/${b}.def" +done +sed -i 's,=base,=monsda_base,g' "$script_dir/base.def" +mkdir -p "$script_dir/TMP_BUILDDIR" +rm -f "$script_dir/build.log" +for i in "$script_dir"/*.def; do + a=${i%.def} + sif=${a}.sif + if APPTAINER_TMPDIR="$script_dir/TMP_BUILDDIR" conda run -n apptainer apptainer build --force --fakeroot "$sif" "$i" >> "$script_dir/build.log" 2>&1 && "$script_dir/smoke_test.sh" "$sif" "$script_dir/build.log"; then + printf '%s\tsmoke-passed\n' "${a##*/}" | tee -a "$script_dir/build.log" + else + rm -f "$sif" + printf '%s\tsmoke-failed\n' "${a##*/}" | tee -a "$script_dir/build.log" + fi +done +rm -rf "$script_dir/TMP_BUILDDIR" diff --git a/containers/apptainer/LoveSonesonPatro.def b/containers/apptainer/LoveSonesonPatro.def index e2e66522..b73001c6 100644 --- a/containers/apptainer/LoveSonesonPatro.def +++ b/containers/apptainer/LoveSonesonPatro.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/LoveSonesonPatro.yaml diff --git a/containers/apptainer/base.def b/containers/apptainer/base.def index 21493b0a..1b27c8f2 100644 --- a/containers/apptainer/base.def +++ b/containers/apptainer/base.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/base.yaml diff --git a/containers/apptainer/bbduk.def b/containers/apptainer/bbduk.def index 5d4b3d84..c6a48816 100644 --- a/containers/apptainer/bbduk.def +++ b/containers/apptainer/bbduk.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bbduk.yaml diff --git a/containers/apptainer/bbmap.def b/containers/apptainer/bbmap.def index da26b296..c2c72ac9 100644 --- a/containers/apptainer/bbmap.def +++ b/containers/apptainer/bbmap.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bbmap.yaml diff --git a/containers/apptainer/bedtools.def b/containers/apptainer/bedtools.def index 24d8827f..c50fbcd5 100644 --- a/containers/apptainer/bedtools.def +++ b/containers/apptainer/bedtools.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bedtools.yaml diff --git a/containers/apptainer/build_all_local.sh b/containers/apptainer/build_all_local.sh new file mode 100755 index 00000000..6375390f --- /dev/null +++ b/containers/apptainer/build_all_local.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +set -uo pipefail + +repo=${1:-${HOME}/Gits/Monsda} +out=${2:-${HOME}/Work2/Container/MONSDA} +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +version=$(cd "$repo" && python scripts/container_matrix.py --version | cut -d= -f2) +revision=$(cd "$repo" && git rev-parse HEAD) +mapfile -t environments < <(cd "$repo" && python scripts/container_matrix.py | python -c 'import json,sys; print("\n".join(json.load(sys.stdin)["environment"]))') +mkdir -p "$out/logs" "$out/tmp" +printf 'environment\tstatus\timage\n' > "$out/results.tsv" + +for environment in "${environments[@]}"; do + sif="$out/${environment}-${version}.sif" + log="$out/logs/${environment}-${version}.log" + image="local/monsda:${environment}-${version}" + available=$(df --output=avail -B1 "$out" | tail -1) + if (( available < 16106127360 )); then + printf '%s\t%s\t%s\n' "$environment" "stopped-low-disk" "$sif" | tee -a "$out/results.tsv" + break + fi + if [[ -s "$sif" ]] && "$script_dir/smoke_test.sh" "$sif" "$log"; then + printf '%s\t%s\t%s\n' "$environment" "existing-smoke-passed" "$sif" | tee -a "$out/results.tsv" + continue + fi + rm -f "$sif" + printf '%s\t%s\n' "$environment" "building" | tee -a "$out/build.log" + if ! docker build --pull --build-arg "ENVIRONMENT=$environment" --build-arg "MONSDA_VERSION=$version" --build-arg "VCS_REF=$revision" -t "$image" -f "$repo/docker/Dockerfile" "$repo" >>"$log" 2>&1; then + printf '%s\t%s\t%s\n' "$environment" "docker-build-failed" "$sif" | tee -a "$out/results.tsv" + continue + fi + if APPTAINER_TMPDIR="$out/tmp" conda run -n apptainer apptainer build --force --fakeroot "$sif" "docker-daemon://$image" >>"$log" 2>&1 && "$script_dir/smoke_test.sh" "$sif" "$log"; then + printf '%s\t%s\t%s\n' "$environment" "smoke-passed" "$sif" | tee -a "$out/results.tsv" + else + rm -f "$sif" + printf '%s\t%s\t%s\n' "$environment" "smoke-failed" "$sif" | tee -a "$out/results.tsv" + fi + docker image rm "$image" >>"$log" 2>&1 || true +done + +rm -rf "$out/tmp" +printf 'finished\n' | tee -a "$out/build.log" diff --git a/containers/apptainer/bwa.def b/containers/apptainer/bwa.def index 23342c56..b4dbb047 100644 --- a/containers/apptainer/bwa.def +++ b/containers/apptainer/bwa.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bwa.yaml diff --git a/containers/apptainer/bwa2.def b/containers/apptainer/bwa2.def index 573db0a6..7461bb89 100644 --- a/containers/apptainer/bwa2.def +++ b/containers/apptainer/bwa2.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bwa2.yaml diff --git a/containers/apptainer/bwameth.def b/containers/apptainer/bwameth.def index 220bdcdf..920807fe 100644 --- a/containers/apptainer/bwameth.def +++ b/containers/apptainer/bwameth.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/bwameth.yaml diff --git a/containers/apptainer/ciri2.def b/containers/apptainer/ciri2.def index 9cb7b263..7a1173e3 100644 --- a/containers/apptainer/ciri2.def +++ b/containers/apptainer/ciri2.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/ciri2.yaml diff --git a/containers/apptainer/countreads.def b/containers/apptainer/countreads.def index 2bed3b8f..540a9786 100644 --- a/containers/apptainer/countreads.def +++ b/containers/apptainer/countreads.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/countreads.yaml diff --git a/containers/apptainer/countreads_de.def b/containers/apptainer/countreads_de.def index 33ce6440..00620c4f 100644 --- a/containers/apptainer/countreads_de.def +++ b/containers/apptainer/countreads_de.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/countreads_de.yaml diff --git a/containers/apptainer/cutadapt.def b/containers/apptainer/cutadapt.def index 734f32bf..2121f754 100644 --- a/containers/apptainer/cutadapt.def +++ b/containers/apptainer/cutadapt.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/cutadapt.yaml diff --git a/containers/apptainer/deseq2_DE.def b/containers/apptainer/deseq2_DE.def index 3650a52a..2f82ce09 100644 --- a/containers/apptainer/deseq2_DE.def +++ b/containers/apptainer/deseq2_DE.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/deseq2_DE.yaml diff --git a/containers/apptainer/dexseq_DEU.def b/containers/apptainer/dexseq_DEU.def index ad9c7558..9b18be79 100644 --- a/containers/apptainer/dexseq_DEU.def +++ b/containers/apptainer/dexseq_DEU.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/dexseq_DEU.yaml diff --git a/containers/apptainer/dexseq_DTU.def b/containers/apptainer/dexseq_DTU.def index 18e01858..59e10b8e 100644 --- a/containers/apptainer/dexseq_DTU.def +++ b/containers/apptainer/dexseq_DTU.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/dexseq_DTU.yaml diff --git a/containers/apptainer/diego_DAS.def b/containers/apptainer/diego_DAS.def index ac1ff58f..1d18b300 100644 --- a/containers/apptainer/diego_DAS.def +++ b/containers/apptainer/diego_DAS.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/diego_DAS.yaml diff --git a/containers/apptainer/dorado.def b/containers/apptainer/dorado.def index c674e18d..6d02bdfe 100644 --- a/containers/apptainer/dorado.def +++ b/containers/apptainer/dorado.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/dorado.yaml diff --git a/containers/apptainer/drimseq_DTU.def b/containers/apptainer/drimseq_DTU.def index ffbe4847..9c5b2273 100644 --- a/containers/apptainer/drimseq_DTU.def +++ b/containers/apptainer/drimseq_DTU.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/drimseq_DTU.yaml diff --git a/containers/apptainer/edger_DAS.def b/containers/apptainer/edger_DAS.def index 66edc324..93b365ae 100644 --- a/containers/apptainer/edger_DAS.def +++ b/containers/apptainer/edger_DAS.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/edger_DAS.yaml diff --git a/containers/apptainer/edger_DE.def b/containers/apptainer/edger_DE.def index 2b76b250..c9f5a3b9 100644 --- a/containers/apptainer/edger_DE.def +++ b/containers/apptainer/edger_DE.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/edger_DE.yaml diff --git a/containers/apptainer/edger_DEU.def b/containers/apptainer/edger_DEU.def index 758ff2d2..ec66e905 100644 --- a/containers/apptainer/edger_DEU.def +++ b/containers/apptainer/edger_DEU.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/edger_DEU.yaml diff --git a/containers/apptainer/fastp.def b/containers/apptainer/fastp.def index 9f9af87e..5dca52c8 100644 --- a/containers/apptainer/fastp.def +++ b/containers/apptainer/fastp.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/fastp.yaml diff --git a/containers/apptainer/fastqc.def b/containers/apptainer/fastqc.def index e50e632c..51095502 100644 --- a/containers/apptainer/fastqc.def +++ b/containers/apptainer/fastqc.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/fastqc.yaml diff --git a/containers/apptainer/fgumi.def b/containers/apptainer/fgumi.def index 4d305696..4e8add5e 100644 --- a/containers/apptainer/fgumi.def +++ b/containers/apptainer/fgumi.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/fgumi.yaml diff --git a/containers/apptainer/guppy.def b/containers/apptainer/guppy.def index e8a2c7d7..3472fee8 100644 --- a/containers/apptainer/guppy.def +++ b/containers/apptainer/guppy.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/guppy.yaml diff --git a/containers/apptainer/hisat2.def b/containers/apptainer/hisat2.def index c6c8dc82..00ab1bb6 100644 --- a/containers/apptainer/hisat2.def +++ b/containers/apptainer/hisat2.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/hisat2.yaml diff --git a/containers/apptainer/index.def b/containers/apptainer/index.def index f1b004ae..8705ea85 100644 --- a/containers/apptainer/index.def +++ b/containers/apptainer/index.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/index.yaml diff --git a/containers/apptainer/isoformswitchanalyzer.def b/containers/apptainer/isoformswitchanalyzer.def index e9a453b9..8d337291 100644 --- a/containers/apptainer/isoformswitchanalyzer.def +++ b/containers/apptainer/isoformswitchanalyzer.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/isoformswitchanalyzer.yaml diff --git a/containers/apptainer/kallisto.def b/containers/apptainer/kallisto.def index 6cb6db1c..90cd3ba5 100644 --- a/containers/apptainer/kallisto.def +++ b/containers/apptainer/kallisto.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/kallisto.yaml diff --git a/containers/apptainer/macs.def b/containers/apptainer/macs.def index 09361528..795b2946 100644 --- a/containers/apptainer/macs.def +++ b/containers/apptainer/macs.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/macs.yaml diff --git a/containers/apptainer/minimap.def b/containers/apptainer/minimap.def index 3d78d6c9..fdb79d96 100644 --- a/containers/apptainer/minimap.def +++ b/containers/apptainer/minimap.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/minimap.yaml diff --git a/containers/apptainer/monsda.def b/containers/apptainer/monsda.def index 1142e27c..c4ee84be 100644 --- a/containers/apptainer/monsda.def +++ b/containers/apptainer/monsda.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/monsda.yaml diff --git a/containers/apptainer/peaks.def b/containers/apptainer/peaks.def index 7ad1097e..cc2fd306 100644 --- a/containers/apptainer/peaks.def +++ b/containers/apptainer/peaks.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/peaks.yaml diff --git a/containers/apptainer/perl.def b/containers/apptainer/perl.def index a39ecf40..1cb93b79 100644 --- a/containers/apptainer/perl.def +++ b/containers/apptainer/perl.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/perl.yaml diff --git a/containers/apptainer/picard.def b/containers/apptainer/picard.def index cda7650b..778718af 100644 --- a/containers/apptainer/picard.def +++ b/containers/apptainer/picard.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/picard.yaml diff --git a/containers/apptainer/piranha.def b/containers/apptainer/piranha.def index 53ce797f..cf2eda39 100644 --- a/containers/apptainer/piranha.def +++ b/containers/apptainer/piranha.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/piranha.yaml diff --git a/containers/apptainer/postde.def b/containers/apptainer/postde.def new file mode 100644 index 00000000..d84f9d70 --- /dev/null +++ b/containers/apptainer/postde.def @@ -0,0 +1,24 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + envs/postde.yaml /opt/envs/ + scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/postde.yaml + + ENV_NAME=postde + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/postde.yaml -p /opt/conda/envs/$ENV_NAME + conda clean --all + +%runscript + exec "$@" diff --git a/containers/apptainer/push_to_oras.sh b/containers/apptainer/push_to_oras.sh new file mode 100755 index 00000000..7da06238 --- /dev/null +++ b/containers/apptainer/push_to_oras.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +set -uo pipefail +shopt -s nullglob + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +repo_root=${HOME}/Gits/Monsda + +if [[ ! -f "$repo_root/scripts/container_matrix.py" ]]; then + printf 'Could not locate MONSDA repo root from %s\n' "$script_dir" >&2 + exit 1 +fi + +version=$(python "$repo_root/scripts/container_matrix.py" --version | cut -d= -f2) +search_dir=${1:-$PWD} +log_file="$search_dir/push-ghcr.log" +results_file="$search_dir/push-ghcr-results.tsv" +: > "$results_file" + +matches=("$search_dir"/*-${version}.sif) +if (( ${#matches[@]} == 0 )); then + printf 'No SIF files matching %s/*-%s.sif found in %s\n' "$search_dir" "$version" "$search_dir" | tee "$log_file" >&2 + exit 1 +fi + +for sif in "${matches[@]}"; do + printf 'Preparing %s\n' "$sif" + base_name=${sif##*/} + environment=${base_name%-${version}.sif} + target="oras://ghcr.io/jfallmann/monsda:${environment}-${version}" + + if ! "$script_dir/smoke_test.sh" "$sif" "$log_file"; then + printf '%s\tsmoke-failed\t%s\n' "$environment" "$target" >> "$results_file" + continue + fi + printf 'Pushing %s\n' "$target" + if conda run -n apptainer apptainer push --allow-unsigned "$sif" "$target"; then + printf '%s\tpassed\t%s\n' "$environment" "$target" >> "$results_file" + else + printf '%s\tfailed\t%s\n' "$environment" "$target" >> "$results_file" + fi +done 2>&1 | tee "$log_file" diff --git a/containers/apptainer/qc.def b/containers/apptainer/qc.def index 4f8ce9b5..2797dfec 100644 --- a/containers/apptainer/qc.def +++ b/containers/apptainer/qc.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/qc.yaml diff --git a/containers/apptainer/rammap.def b/containers/apptainer/rammap.def new file mode 100644 index 00000000..f23a56b5 --- /dev/null +++ b/containers/apptainer/rammap.def @@ -0,0 +1,24 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + /home/fall/Gits/Monsda/envs/rammap.yaml /opt/envs/ + /home/fall/Gits/Monsda/scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/rammap.yaml + + ENV_NAME=rammap + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/rammap.yaml -p /opt/conda/envs/$ENV_NAME + conda clean --all + +%runscript + exec "$@" diff --git a/containers/apptainer/rastqc.def b/containers/apptainer/rastqc.def new file mode 100644 index 00000000..065107b8 --- /dev/null +++ b/containers/apptainer/rastqc.def @@ -0,0 +1,24 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + /home/fall/MONSDA/envs/rastqc.yaml /opt/envs/ + ${HOME}/MONSDA/scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/rastqc.yaml + + ENV_NAME=rastqc + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/rastqc.yaml -p /opt/conda/envs/$ENV_NAME + conda clean --all + +%runscript + exec "$@" diff --git a/containers/apptainer/rustar.def b/containers/apptainer/rustar.def new file mode 100644 index 00000000..2a7d1afd --- /dev/null +++ b/containers/apptainer/rustar.def @@ -0,0 +1,32 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + /home/fall/MONSDA/envs/rustar.yaml /opt/envs/ + ${HOME}/MONSDA/scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/rustar.yaml + + ENV_NAME=rustar + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/rustar.yaml -p /opt/conda/envs/$ENV_NAME + conda activate $ENV_NAME + + # rustar-aligner is not distributed via conda/crates.io, so build it from source + # using the Rust toolchain provided by the conda environment. + cargo install --git https://github.com/scverse/rustar-aligner --locked --root /opt/conda/envs/$ENV_NAME + # MONSDA expects the mapper binary to be called "rustar" + ln -fs /opt/conda/envs/$ENV_NAME/bin/rustar-aligner /opt/conda/envs/$ENV_NAME/bin/rustar + + conda clean --all + +%runscript + exec "$@" diff --git a/containers/apptainer/rustqc.def b/containers/apptainer/rustqc.def index 2d9f57b8..841fc35c 100644 --- a/containers/apptainer/rustqc.def +++ b/containers/apptainer/rustqc.def @@ -1,5 +1,5 @@ Bootstrap: docker -From: continuumio/miniconda3 +From: continuumio/miniconda3:latest %files /home/fall/MONSDA/envs/rustqc.yaml /opt/envs/ @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/rustqc.yaml diff --git a/containers/apptainer/salmon.def b/containers/apptainer/salmon.def index 251eeb4a..26cc15f4 100644 --- a/containers/apptainer/salmon.def +++ b/containers/apptainer/salmon.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/salmon.yaml diff --git a/containers/apptainer/samtools.def b/containers/apptainer/samtools.def index b54ed33c..9182a7e5 100644 --- a/containers/apptainer/samtools.def +++ b/containers/apptainer/samtools.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/samtools.yaml diff --git a/containers/apptainer/scyphy.def b/containers/apptainer/scyphy.def index c56d6933..4d960b53 100644 --- a/containers/apptainer/scyphy.def +++ b/containers/apptainer/scyphy.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/scyphy.yaml diff --git a/containers/apptainer/segemehl.def b/containers/apptainer/segemehl.def index c74ccff3..306399ee 100644 --- a/containers/apptainer/segemehl.def +++ b/containers/apptainer/segemehl.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/segemehl.yaml diff --git a/containers/apptainer/segemehl3.def b/containers/apptainer/segemehl3.def index ea94815b..ac6b63fc 100644 --- a/containers/apptainer/segemehl3.def +++ b/containers/apptainer/segemehl3.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/segemehl3.yaml diff --git a/containers/apptainer/smoke_test.sh b/containers/apptainer/smoke_test.sh new file mode 100755 index 00000000..e5fb5b86 --- /dev/null +++ b/containers/apptainer/smoke_test.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +set -uo pipefail + +sif=${1:?Usage: smoke_test.sh IMAGE.sif [LOG]} +log=${2:-/dev/stderr} + +if command -v apptainer >/dev/null 2>&1; then + runner=(apptainer) +else + runner=(conda run -n apptainer apptainer) +fi + +"${runner[@]}" exec --cleanenv "$sif" sh -c ' + report=$(mktemp) + test -x /opt/MONSDA/scripts/Analysis/build_count_table_simple.py || exit 1 + found=0 + for bindir in /opt/conda/envs/*/bin; do + test -d "$bindir" || continue + found=1 + find "$bindir" -type f -perm /111 -exec sh -c '\'' + for executable do + magic=$(od -An -tx1 -N4 "$executable" 2>/dev/null | tr -d " \n") + test "$magic" = 7f454c46 || continue + output=$(ldd "$executable" 2>&1 || true) + if printf "%s\n" "$output" | grep -Eq "version .+ not found"; then + printf "%s\n%s\n" "$executable" "$output" + fi + done + '\'' sh {} + >> "$report" + done + if test "$found" -eq 0 || test -s "$report"; then + cat "$report" >&2 + rm -f "$report" + exit 1 + fi + rm -f "$report" +' >>"$log" 2>&1 diff --git a/containers/apptainer/spit_DTU.def b/containers/apptainer/spit_DTU.def new file mode 100644 index 00000000..81b329cb --- /dev/null +++ b/containers/apptainer/spit_DTU.def @@ -0,0 +1,24 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + /home/fall/MONSDA/envs/spit_DTU.yaml /opt/envs/ + ${HOME}/MONSDA/scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/spit_DTU.yaml + + ENV_NAME=spit_DTU + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/spit_DTU.yaml -p /opt/conda/envs/$ENV_NAME + conda clean --all + +%runscript + exec "$@" diff --git a/containers/apptainer/sra.def b/containers/apptainer/sra.def index 9b7d93ae..17bc2896 100644 --- a/containers/apptainer/sra.def +++ b/containers/apptainer/sra.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/sra.yaml diff --git a/containers/apptainer/star.def b/containers/apptainer/star.def index 7fecd634..daeac763 100644 --- a/containers/apptainer/star.def +++ b/containers/apptainer/star.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/star.yaml diff --git a/containers/apptainer/starfusion.def b/containers/apptainer/starfusion.def new file mode 100644 index 00000000..013cf5d3 --- /dev/null +++ b/containers/apptainer/starfusion.def @@ -0,0 +1,24 @@ +Bootstrap: docker +From: continuumio/miniconda3 + +%files + /home/fall/MONSDA/envs/starfusion.yaml /opt/envs/ + ${HOME}/MONSDA/scripts /opt/MONSDA/ + +%environment + +%post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + ls -alrt /opt/envs + chmod -R +x /opt/envs/starfusion.yaml + + ENV_NAME=starfusion + echo ". /opt/conda/etc/profile.d/conda.sh" >> $APPTAINER_ENVIRONMENT + echo "conda activate $ENV_NAME" >> $APPTAINER_ENVIRONMENT + + . /opt/conda/etc/profile.d/conda.sh + conda env create -f /opt/envs/starfusion.yaml -p /opt/conda/envs/$ENV_NAME + conda clean --all + +%runscript + exec "$@" \ No newline at end of file diff --git a/containers/apptainer/summary.def b/containers/apptainer/summary.def index 327080a3..fb144944 100644 --- a/containers/apptainer/summary.def +++ b/containers/apptainer/summary.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/summary.yaml diff --git a/containers/apptainer/trimgalore.def b/containers/apptainer/trimgalore.def index 993bcc2a..78090536 100644 --- a/containers/apptainer/trimgalore.def +++ b/containers/apptainer/trimgalore.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/trimgalore.yaml diff --git a/containers/apptainer/trimm.def b/containers/apptainer/trimm.def index 5dce7490..466646bd 100644 --- a/containers/apptainer/trimm.def +++ b/containers/apptainer/trimm.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/trimm.yaml diff --git a/containers/apptainer/trimmomatic.def b/containers/apptainer/trimmomatic.def index 4f2bff98..965b59dc 100644 --- a/containers/apptainer/trimmomatic.def +++ b/containers/apptainer/trimmomatic.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/trimmomatic.yaml diff --git a/containers/apptainer/trnascan.def b/containers/apptainer/trnascan.def index e0b150b2..53e73e1c 100644 --- a/containers/apptainer/trnascan.def +++ b/containers/apptainer/trnascan.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/trnascan.yaml diff --git a/containers/apptainer/ucsc.def b/containers/apptainer/ucsc.def index 5946a849..67b98789 100644 --- a/containers/apptainer/ucsc.def +++ b/containers/apptainer/ucsc.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/ucsc.yaml diff --git a/containers/apptainer/umitools.def b/containers/apptainer/umitools.def index 53721e5f..c3844b78 100644 --- a/containers/apptainer/umitools.def +++ b/containers/apptainer/umitools.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/umitools.yaml diff --git a/containers/apptainer/zip.def b/containers/apptainer/zip.def index f6b341d5..c6602523 100644 --- a/containers/apptainer/zip.def +++ b/containers/apptainer/zip.def @@ -8,6 +8,7 @@ From: continuumio/miniconda3 %environment %post + find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + ls -alrt /opt/envs chmod -R +x /opt/envs/zip.yaml diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 00000000..3c146a40 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,23 @@ +FROM mambaorg/micromamba:latest + +ARG ENVIRONMENT +ARG MONSDA_VERSION +ARG VCS_REF + +LABEL org.opencontainers.image.source="https://github.com/jfallmann/MONSDA" +LABEL org.opencontainers.image.revision="${VCS_REF}" +LABEL org.opencontainers.image.version="${MONSDA_VERSION}" +LABEL org.opencontainers.image.title="MONSDA ${ENVIRONMENT} environment" + +COPY --chown=$MAMBA_USER:$MAMBA_USER envs/${ENVIRONMENT}.yaml /tmp/environment.yaml +COPY --chown=$MAMBA_USER:$MAMBA_USER scripts /opt/MONSDA/scripts +RUN find /opt/MONSDA/scripts -type f \( -name '*.py' -o -name '*.R' -o -name '*.sh' -o -name '*.pl' \) -exec chmod a+x {} + + +RUN micromamba create --yes --prefix /opt/conda/envs/monsda --file /tmp/environment.yaml \ + && micromamba clean --all --yes + +ENV PATH="/opt/conda/envs/monsda/bin:${PATH}" +ENV MONSDA_VERSION="${MONSDA_VERSION}" + +ENTRYPOINT ["/usr/local/bin/_entrypoint.sh"] +CMD ["/bin/bash"] diff --git a/docs/source/config.rst b/docs/source/config.rst index 022fe09d..b1c2cd18 100644 --- a/docs/source/config.rst +++ b/docs/source/config.rst @@ -34,6 +34,15 @@ You can always define differing ENV/BIN keys for each condition-tree leaf separa The next key-level is the *OPTIONS* key which is where you can define additional parameters for each tool. It is not needed to define anything related to *single-/paired-* end or *singlecell* sequencing, this is done automatically. To add parameters simply add the *OPTION* key which defines a dict where you can set parameters for each defined subworkflow-step. Parameters are here defined as key/value pairs corresponding to the subworkflow-step, e.g. 'INDEX' to generate an index file for mapping and all settings similar to a command line call as values. This should become clear having a look at the different processing steps in the template json. If there are no options just leave the 'OPTIONS' dict empty. +.. note:: + + A few tool-specific options are worth highlighting: + + * **salmon COUNTING from BAM**: in addition to the default FASTQ quantification, salmon can quantify directly from an alignment BAM by setting the ``BAM`` option to ``transcriptome`` (alignments against a transcriptome FASTA) or ``genome`` (alignments against the genome, requires the annotation GTF). Leave it empty/unset for the default FASTQ mode. Salmon indices are automatically rebuilt when the environment changes (salmon >= 2.3, ``--validateMappings`` is no longer needed). + * **long-read quantification**: use ``minimap`` with ``-x map-ont``/``-x map-pb`` (or ``rammap``) for mapping and ``oarfish`` for transcript quantification from the resulting transcriptome BAM. + * **single-cell**: use ``piscem`` as MAPPING tool (emits a RAD file, single-cell only, generic post-mapping is skipped) together with ``alevinfry`` as COUNTING tool, or use the self-contained ``simpleaf`` COUNTING tool. + * **DTU pre-grouping**: any DTU tool (drimseq/dexseq/spit) accepts an optional ``TERMINUS`` option to pre-group transcripts with terminus before analysis. It is disabled by default. + .. literalinclude:: ../../configs/template.json :language: json @@ -103,3 +112,122 @@ Another special case is 'MACS', a peak caller for ChIP-Seq data, which needs to For everything else please refer to the :ref:`tutorials`, the :ref:`condition-tree` and the :ref:`Workflow Overview`. Keep in mind that every workflow step needs a corresponding entry in the config file or **MONSDA.py** will throw an error. + +PostDE (GO/GSEA/GSVA/decoupleR/dream) +===================================== + +PostDE is an optional post-processing step that runs enrichment (GO/GSEA via gProfiler or clusterProfiler), gene-set variation analysis (GSVA) and regulatory inference (decoupleR ULM/MLM, variancePartition dream) on the results of a DE analysis. It is **disabled by default** and only supported together with the **DE** workflow and the **deseq2**/**edger** DE tools (DEU/DTU/DAS are not supported). + +Enable it in the top-level of the config file, next to ``WORKFLOWS``, ``DE`` and the DE ``TOOLS`` (which stay unchanged): + +.. code-block:: json + + { + "WORKFLOWS": "DE", + "DE": { + "TOOLS": { + "deseq2": "Analysis/DE/DESEQ2.R", + "edger": "Analysis/DE/EDGER.R" + } + }, + "POSTDE": { + "enabled": true, + "config": "/abs/path/to/postde_analysis.json" + } + } + +The ``POSTDE.config`` path is resolved relative to the **run working directory** (the directory you launch ``monsda`` from). Resource files referenced inside the analysis JSON (``term2gene``, ``background``, ``network``, ``metadata``) are resolved relative to the directory of the analysis JSON itself. The analysis JSON is validated at startup and again when the DE sub-workflow is prepared; an invalid config aborts the run with a clear error. + +Analysis JSON schema +-------------------- + +The analysis JSON supports the following top-level keys (all optional unless noted): + +.. code-block:: json + + { + "padj": 0.05, + "lfc": 1, + "seed": 1, + "min_size": 10, + "max_size": 500, + "gprofiler": { "enabled": false }, + "clusterprofiler": { "enabled": false }, + "gsva": { "enabled": false }, + "decoupler": { "enabled": false }, + "dream": { "enabled": false }, + "plots": { "enabled": false }, + "report": { "enabled": false }, + "shiny": { "enabled": false } + } + +Defaults (read from the actual R implementation): ``padj=0.05``, ``lfc=1``, ``seed=1``, ``min_size=10``, ``max_size=500``. ``padj``/``lfc`` are the PostDE significance cutoffs used to define the significant gene sets for ORA; they are **independent of** the ``DE.CUTOFFS`` in the DE section. GSEA always consumes the **full ranked gene list** and is never filtered by ``padj``/``lfc``. + +Per-method keys: + +* ``gprofiler``: ``enabled``, ``organism`` (required when enabled, e.g. ``hsapiens``), ``domain_scope`` (``annotated``, ``known``, ``custom``, ``custom_annotated``; default ``custom``), ``allow_network`` (required ``true``, gost is an online call), ``sources`` (list, e.g. ``["GO:BP", "GO:MF", "GO:CC", "REAC"]``; default all), ``correction_method`` (default ``g_SCS``), ``background`` (optional file, one gene ID per row, no header). +* ``clusterprofiler``: ``enabled``, ``term2gene`` (required when clusterprofiler or gsva is enabled), ``term2name`` (optional), ``go_expand`` (optional boolean, builds a GO map via ``buildGOmap``), ``background`` (optional file, one gene ID per row, no header). +* ``gsva``: ``enabled``, ``method`` (``gsva`` or ``ssgsea``; default ``gsva``), ``min_size``/``max_size`` (default to the top-level values). +* ``decoupler``: ``enabled``, ``network`` (offline network file) **or** ``allow_network`` (retrieve from OmniPath), ``source_col``/``target_col``/``mor_col`` (default ``source``/``target``/``mor``), ``resource`` (``collectri`` or ``progeny``; default ``collectri``), ``organism`` (default ``human``), ``top`` (PROGENy top pathways, default ``500``), ``min_size`` (default top-level), ``methods`` (``ulm`` and/or ``mlm``; default both), ``contrast_activity`` (optional boolean). +* ``dream``: ``enabled``, ``metadata`` (required when enabled; TSV with a ``sample`` column, **no** ``condition`` column), ``formula`` (required when enabled, must include a random effect, e.g. ``~condition + (1|subject)``), ``contrasts`` (optional). +* ``plots``/``report``/``shiny``: only ``enabled``; setting ``enabled=true`` is rejected because the HTML reporting layer is not provided yet. + +A ready-to-use online example (human) is shipped as ``configs/postde_analysis.json`` in the repository. It enables gProfiler with ``organism=hsapiens``, ``domain_scope=custom``, ``allow_network=true`` and the sources ``GO:BP``, ``GO:MF``, ``GO:CC``, ``REAC``; all other methods are disabled and no resource files are required. Change the organism for non-human data — the example is not universal. + +Offline example (clusterProfiler + GSVA) +---------------------------------------- + +For offline runs (no gProfiler network access) use clusterProfiler/GSVA with a custom ``term2gene`` file. The file must have a header with exactly the columns ``term`` and ``gene``: + +.. code-block:: text + + term gene + GO:0008150 GENE1 + GO:0008150 GENE2 + GO:0009987 GENE1 + +An optional ``term2name`` file has the header ``term`` and ``name``; an optional ``background`` file has one gene ID per row and no header. A decoupler ``network`` file has the columns ``source``, ``target``, ``mor``; a dream ``metadata`` file has a ``sample`` column and no ``condition`` column. + +.. code-block:: json + + { + "padj": 0.05, + "lfc": 1, + "seed": 42, + "min_size": 10, + "max_size": 500, + "clusterprofiler": { + "enabled": true, + "term2gene": "term2gene.tsv", + "term2name": "term2name.tsv", + "background": "background.txt" + }, + "gsva": { + "enabled": true, + "method": "gsva" + }, + "gprofiler": { "enabled": false }, + "decoupler": { "enabled": false }, + "dream": { "enabled": false }, + "plots": { "enabled": false }, + "report": { "enabled": false }, + "shiny": { "enabled": false } + } + +Running PostDE +-------------- + +From a populated project root (raw BAMs and a completed DE run are prerequisites, see the DE section of the :ref:`Workflow Overview`), with the root ``POSTDE`` path enabled and the config ``VERSION`` matching the installed MONSDA version: + +.. code-block:: bash + + monsda -c config_monsda.json -d . -j 8 + +Add ``--nextflow`` to run the Nextflow engine or ``--save`` to keep intermediate files. The CLI flags ``-d``/``-j`` are supported by the existing runner (covered by the test suite). + +Environment +----------- + +The PostDE R code needs the ``postde`` conda environment (``envs/postde.yaml`` in the repository). A container recipe exists (``containers/apptainer/postde.def``) and builds from that environment file, but no image is published yet — prefer the conda route until a local/cluster container build is available. + +The interactive Configurator (``monsda_configure``) can enable PostDE when creating or modifying a config: it asks for the analysis JSON path when the DE workflow is active, and the modify menu offers a dedicated *configure PostDE* option. The web Configurator exposes the same option as a checkbox next to the workflow selection. diff --git a/docs/source/workflows.rst b/docs/source/workflows.rst index 008756bc..2453561c 100644 --- a/docs/source/workflows.rst +++ b/docs/source/workflows.rst @@ -32,6 +32,8 @@ DTU -> Differential Transcript Usage Analysis CIRCS -> Circular RNA identification +FUSIONS -> Fusion gene detection + PREPROCESSING ============= @@ -61,13 +63,13 @@ Creates FASTQ files from ONT FAST5, Guppy needs to be installed locally :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+---------------------+------------------------------------------------------------------------+--------+---------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +=======+============================================================================================================================================================================================================================================================================+========+=====================+========================================================================+========+=========+ - | Guppy | Data processing toolkit that contains Oxford Nanopore’s basecalling algorithms, and several bioinformatic post-processing features, such as barcoding/demultiplexing, adapter trimming, and alignment. Needs to be installed locally as no **conda** version is available | guppy | $PATH_TO_LOCAL_BIN | `guppy `_ | FAST5 | FASTQ | - +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+---------------------+------------------------------------------------------------------------+--------+---------+ - | Dorado | Dorado is a high-performance, easy-to-use, open source basecaller for Oxford Nanopore reads. Needs to be installed locally as no **conda** version is available | dorado | $PATH_TO_LOCAL_BIN | `dorado `_ | POD5 | FASTQ | - +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+---------------------+------------------------------------------------------------------------+--------+---------+ + +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+--------------------+-----------------------------------------------------------------------+-------+--------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +========+===========================================================================================================================================================================================================================================================================+========+====================+=======================================================================+=======+========+ + | Guppy | Data processing toolkit that contains Oxford Nanopore’s basecalling algorithms, and several bioinformatic post-processing features, such as barcoding/demultiplexing, adapter trimming, and alignment. Needs to be installed locally as no **conda** version is available | guppy | $PATH_TO_LOCAL_BIN | `guppy `_ | FAST5 | FASTQ | + +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+--------------------+-----------------------------------------------------------------------+-------+--------+ + | Dorado | Dorado is a high-performance, easy-to-use, open source basecaller for Oxford Nanopore reads. Needs to be installed locally as no **conda** version is available | dorado | $PATH_TO_LOCAL_BIN | `dorado `_ | POD5 | FASTQ | + +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+--------------------+-----------------------------------------------------------------------+-------+--------+ QUALITY CONTROL I @@ -79,13 +81,13 @@ This workflow step can be run as preprocessing step if none of the processing wo :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +============================+============================================================+=========+=========+=========================================================================+============+===========+ - | FASTQC (includes MULTIQC) | A quality control tool for high throughput sequence data. | fastqc | fastqc | `fastqc `_ | FASTQ/BAM | ZIP/HTML | - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ - | RustQC (includes MULTIQC) | High-performance RNA-seq QC suite with MultiQC-compatible outputs. | rustqc | rustqc | `rustqc `_ | BAM | TEXT/TSV/LOG | - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +===========================+====================================================================+========+========+========================================================================+===========+==============+ + | FASTQC (includes MULTIQC) | A quality control tool for high throughput sequence data. | fastqc | fastqc | `fastqc `_ | FASTQ/BAM | ZIP/HTML | + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ + | RustQC (includes MULTIQC) | High-performance RNA-seq QC suite with MultiQC-compatible outputs. | rustqc | rustqc | `rustqc `_ | BAM | TEXT/TSV/LOG | + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ PROCESSING @@ -100,13 +102,39 @@ If any of the below listed processing steps is defined in the config.json, quali :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +============================+============================================================+=========+=========+=========================================================================+============+===========+ - | FASTQC (includes MULTIQC) | A quality control tool for high throughput sequence data. | fastqc | fastqc | `fastqc `_ | FASTQ/BAM | ZIP/HTML | - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ - | RustQC (includes MULTIQC) | High-performance RNA-seq QC suite with MultiQC-compatible outputs. | rustqc | rustqc | `rustqc `_ | BAM | TEXT/TSV/LOG | - +----------------------------+------------------------------------------------------------+---------+---------+-------------------------------------------------------------------------+------------+-----------+ + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +===========================+====================================================================+========+========+========================================================================+===========+==============+ + | FASTQC (includes MULTIQC) | A quality control tool for high throughput sequence data. | fastqc | fastqc | `fastqc `_ | FASTQ/BAM | ZIP/HTML | + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ + | RustQC (includes MULTIQC) | High-performance RNA-seq QC suite with MultiQC-compatible outputs. | rustqc | rustqc | `rustqc `_ | BAM | TEXT/TSV/LOG | + +---------------------------+--------------------------------------------------------------------+--------+--------+------------------------------------------------------------------------+-----------+--------------+ + +RustQC runs on mapped BAM files and includes a strandedness guard: after the +RustQC command finishes, the inferred strand assignment from its +``infer_experiment`` report is compared against the ``SEQUENCING`` setting +(``paired,fr``, ``paired,rf`` or ``single,fr``/``single,rf``) configured for +the sample. A confident mismatch aborts the QC job with an error instead of +silently accepting a conflicting configuration. No extra config switch is +needed when RustQC is selected. ``fr`` means forward, ``rf`` means reverse, +and omitting either strand suffix means unstranded. + +The check requires at least 50% of reads to have a strand assignment. Among +assigned reads, at least 80% forward or reverse support identifies that strand; +40--60% forward support identifies an unstranded library. Intermediate ratios +or insufficient assignments produce a warning (``NOT validated``), not a +mismatch. These are fraction-based heuristics, not a statistical confidence +test or a minimum-read-count check. Missing or malformed inference reports +fail the task rather than silently bypassing validation. + +The verdict is written to ``strandedness_check.json`` inside the RustQC output +directory. On failure, Snakemake may remove failed outputs; the error remains +in ``LOGS///QC/rustqc/``. Nextflow retains the error and verdict +in the failed task's work directory. MONSDA exits nonzero and does not proceed +to subsequent analysis workflows. Already-running independent tasks may have +produced outputs before the QC failure is detected; this is not a pre-mapping +check. Verify the library protocol and annotation, correct ``SEQUENCING`` if +needed, and regenerate the workflows before resuming. Trimming @@ -118,17 +146,17 @@ Trims adaptor left-overs from fastq files :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+--------------+----------------------------------------------------------------------------------+--------+----------------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +==============+==========================================================================================================================================================================================================================+=============+==============+==================================================================================+========+================+ - | trim_galore | A wrapper tool around Cutadapt and FastQC to consistently apply quality and adapter trimming to FastQ files, with some extra functionality for MspI-digested RRBS-type (Reduced Representation Bisufite-Seq) libraries. | trimgalore | trim_galore | `trimgalore `_ | FASTQ | TRIMMED_FASTQ | - +--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+--------------+----------------------------------------------------------------------------------+--------+----------------+ - | cutadapt | Cutadapt finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads. | cutadapt | cutadapt | `cutadapt `_ | FASTQ | TRIMMED_FASTQ | - +--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+--------------+----------------------------------------------------------------------------------+--------+----------------+ - | bbduk | “Duk” stands for Decontamination Using Kmers. BBDuk was developed to combine most common data-quality-related trimming, filtering, and masking operations into a single high-performance tool. | bbduk | bbduk | `bbduk `_ | FASTQ | TRIMMED_FASTQ | - +--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+--------------+----------------------------------------------------------------------------------+--------+----------------+ - | fastp | A tool designed to provide fast all-in-one preprocessing for FastQ files. This tool is developed in C++ with multithreading supported to afford high performance. | fastp | fastp | `fastp `_ | FASTQ | TRIMMED_FASTQ | - +--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+--------------+----------------------------------------------------------------------------------+--------+----------------+ + +-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-------------------------------------------------------------------------------------------------------+-------+---------------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +=============+=========================================================================================================================================================================================================================+============+=============+=======================================================================================================+=======+===============+ + | trim_galore | A wrapper tool around Cutadapt and FastQC to consistently apply quality and adapter trimming to FastQ files, with some extra functionality for MspI-digested RRBS-type (Reduced Representation Bisufite-Seq) libraries. | trimgalore | trim_galore | `trimgalore `_ | FASTQ | TRIMMED_FASTQ | + +-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-------------------------------------------------------------------------------------------------------+-------+---------------+ + | cutadapt | Cutadapt finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads. | cutadapt | cutadapt | `cutadapt `_ | FASTQ | TRIMMED_FASTQ | + +-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-------------------------------------------------------------------------------------------------------+-------+---------------+ + | bbduk | “Duk” stands for Decontamination Using Kmers. BBDuk was developed to combine most common data-quality-related trimming, filtering, and masking operations into a single high-performance tool. | bbduk | bbduk | `bbduk `_ | FASTQ | TRIMMED_FASTQ | + +-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-------------------------------------------------------------------------------------------------------+-------+---------------+ + | fastp | A tool designed to provide fast all-in-one preprocessing for FastQ files. This tool is developed in C++ with multithreading supported to afford high performance. | fastp | fastp | `fastp `_ | FASTQ | TRIMMED_FASTQ | + +-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-------------------------------------------------------------------------------------------------------+-------+---------------+ Mapping @@ -140,27 +168,35 @@ Maps sequences to reference genomes or transcriptomes :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +==============+============================================================================================================================+======================+=============+=====================================================================+======================+=============+ - | HISAT2 | HISAT2 is a fast and sensitive alignment program | hisat2 | hisat2 | `hisat2 `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | STAR | Spliced Transcripts Alignment to a Reference | star | STAR | `star `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | STARsolo | STARsolo: mapping, demultiplexing and quantification for single cell RNA-seq | star | STAR | `STARsolo `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | Segemehl2|3 | Segemehl is a software to map short sequencer reads to reference genomes. | segemehl2/segemehl3 | segemehl.x | `segemehl `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | Segemehl2|3 bisulfite | Segemehl is a software to map short sequencer reads to reference genomes. This is the bisulfite mapping mode | segemehl2bisulfite/segemehl3bisulfite | segemehl.x | `segemehl `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | BWA | BWA is a software package for mapping low-divergent sequences against a large reference genome | bwa | bwa mem | `bwa `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | BWA2 | BWA is a software package for mapping low-divergent sequences against a large reference genome | bwa2 | bwa2-mem | `bwa2 `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | BWA-Meth | BWA-meth, Fast and accurante alignment of BS-Seq reads. | bwameth | bwameth.py | `bwa-meth `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ - | Minimap2 | Minimap2 is a versatile sequence alignment program that aligns DNA or mRNA sequences against a large reference database. | minimap | minimap2 | `minimap `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | - +--------------+----------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+---------------------------------------------------------------------+----------------------+-------------+ + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +=======================+====================================================================================================================================================================================================+=======================================+============+==============================================================================+=====================+============+ + | HISAT2 | HISAT2 is a fast and sensitive alignment program | hisat2 | hisat2 | `hisat2 `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | STAR | Spliced Transcripts Alignment to a Reference | star | STAR | `star `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | rustar | Pure-Rust reimplementation of STAR, drop-in replacement with STAR-compatible CLI and index | rustar | rustar | `rustar-aligner `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | STARsolo | STARsolo: mapping, demultiplexing and quantification for single cell RNA-seq | star | STAR | `STARsolo `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | Segemehl2|3 | Segemehl is a software to map short sequencer reads to reference genomes. | segemehl2/segemehl3 | segemehl.x | `segemehl `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | Segemehl2|3 bisulfite | Segemehl is a software to map short sequencer reads to reference genomes. This is the bisulfite mapping mode | segemehl2bisulfite/segemehl3bisulfite | segemehl.x | `segemehl `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | BWA | BWA is a software package for mapping low-divergent sequences against a large reference genome | bwa | bwa mem | `bwa `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | BWA2 | BWA is a software package for mapping low-divergent sequences against a large reference genome | bwa2 | bwa2-mem | `bwa2 `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | BWA-Meth | BWA-meth, Fast and accurante alignment of BS-Seq reads. | bwameth | bwameth.py | `bwa-meth `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | Minimap2 | Minimap2 is a versatile sequence alignment program that aligns DNA or mRNA sequences against a large reference database. Use ``-x map-ont``/``-x map-pb`` for long-read (Nanopore/PacBio) mapping. | minimap | minimap2 | `minimap `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | salmonalign | Salmon selective-alignment used as a mapper (``salmon quant --writeMappings``). Reuses the salmon environment/container. | salmonalign | salmon | `salmon `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | rammap | Rust-accelerated long-read (Nanopore/PacBio) mapper (cargo-built, not on bioconda). | rammap | rammap | `rammap `_ | FASTQ/TRIMMED_FASTQ | SAM.gz/BAM | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ + | piscem | Fast single-cell/bulk mapper producing a RAD file for downstream alevin-fry quantification (single-cell only). | piscem | piscem | `piscem `_ | FASTQ/TRIMMED_FASTQ | RAD | + +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+------------+------------------------------------------------------------------------------+---------------------+------------+ DEDUP @@ -195,15 +231,21 @@ Count (unique) mapped reads and how often they map to defined features :widths: 10, 40, 10, 10, 10, 10, 10 :class: tight-table - +----------------+-----------------------------------------------------------------------------------------------------------------------+-------------+----------------+------------------------------------------------------------------+----------------------+---------+ - | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | - +================+=======================================================================================================================+=============+================+==================================================================+======================+=========+ - | FeatureCounts | A software program developed for counting reads to genomic features such as genes, exons, promoters and genomic bins | countreads | featureCounts | `featurecounts `_ | BAM/FASTQ | TEXT | - +----------------+-----------------------------------------------------------------------------------------------------------------------+-------------+----------------+------------------------------------------------------------------+----------------------+---------+ - | Salmon | Salmon is a tool for wicked-fast transcript quantification from RNA-seq data. | salmon | salmon | `salmon `_ | FASTQ/TRIMMED_FASTQ | TEXT | - +----------------+-----------------------------------------------------------------------------------------------------------------------+-------------+----------------+------------------------------------------------------------------+----------------------+---------+ - | Kallisto | Kallisto is a program for quantifying abundances of transcripts from bulk and single-cell RNA-Seq data, or more generally of target sequences using high-throughput sequencing reads. | kallisto | kallisto | `kallisto `_ | FASTQ/TRIMMED_FASTQ | TEXT | - +----------------+-----------------------------------------------------------------------------------------------------------------------+-------------+----------------+------------------------------------------------------------------+----------------------+---------+ + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +===============+=====================================================================================================================================================================================================+============+===============+=================================================================+=========================+========+ + | FeatureCounts | A software program developed for counting reads to genomic features such as genes, exons, promoters and genomic bins | countreads | featureCounts | `featurecounts `_ | BAM/FASTQ | TEXT | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | Salmon | Salmon is a tool for wicked-fast transcript quantification from RNA-seq data. Also supports quantification directly from an alignment BAM via the ``BAM`` option (``transcriptome`` or ``genome``). | salmon | salmon | `salmon `_ | FASTQ/TRIMMED_FASTQ/BAM | TEXT | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | Kallisto | Kallisto is a program for quantifying abundances of transcripts from bulk and single-cell RNA-Seq data, or more generally of target sequences using high-throughput sequencing reads. | kallisto | kallisto | `kallisto `_ | FASTQ/TRIMMED_FASTQ | TEXT | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | oarfish | Long-read (Nanopore/PacBio) transcript quantification from a transcriptome-aligned BAM. | oarfish | oarfish | `oarfish `_ | BAM | TEXT | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | alevin-fry | Single-cell quantification consuming a piscem RAD file to produce a count matrix. | alevinfry | alevin-fry | `alevin-fry `_ | RAD | MATRIX | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ + | simpleaf | Self-contained single-cell quantification wrapper (index + map + quant) around salmon/alevin-fry. | simpleaf | simpleaf | `simpleaf `_ | FASTQ/TRIMMED_FASTQ | MATRIX | + +---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------------+-----------------------------------------------------------------+-------------------------+--------+ Differential Analyses @@ -214,26 +256,50 @@ Includes DE, DEU, DAS and DTU .. table:: :class: tight-table - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | Tool | Analysis | Filtering | Normalization | Distribution | Testing | Significance | Results Table | further | SigTables | Clustering | further | Rmd | - +===========+=====================================+==================+=================+================+=================================+================+======================================================+=========================================+=========================================+===================+===================================================================+=======+ - | edgeR | Differential Gene Expression | filterByExpr() | TMM | NB | Fisher’s exact test | pValue, LFC | results, sorted-results | normalized | Sig, SigUP, SigDOWN | MDS-plot | BCV, QLDisp, MD(per comparison) | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | edgeR | Differential Exon Usage | filterByExpr() | TMM | NB | Fisher’s exact test | pValue, LFC | results | normalized | | MDS-plot | BCV, QLDisp, MD(per comparison) | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | edgeR | Differential Alternative Splicing | filterByExpr() | TMM | NB | Simes, gene-level, exon-level | pValue, LFC | results(diffSpliceExonTest, Simes-Test, Gene-Test) | | Sig, SigUP, SigDOWN | MDS-plot | BCV, QLDisp, MD(per comparison), topSpliceSimes-plots(per Gene) | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | DESeq2 | Differential Gene Expression | RowSums >= 10 | RLE | NB | Wald test | pValue, LFC | results | rld, vsd, results(per comparison) | Sig, SigUP, SigDOWN | PCA | Heatmaps, MA(per comparison), VST-and-log2 | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | DEXSeq | Differential Exon Usage | RowSums >= 10 | RLE | Cox-Reid | likelihood ratio test | | | | | | | | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | DEXSeq | Differential Transcript Usage | dmFilter() | RLE | Cox-Reid | likelihood ratio test | pValue | results | | | | | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | DIEGO | Differential Alternative Splicing | | | | Mann-Whitney U test | pValue | results | | Sig | Dendrogram-plot | | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - | DRIMSeq | Differential Transcript Usage | dmFilter() | | DM | | pValue, LFC | results(transcript, genes) | Proportions-table, genewise precision | Sig, SigUP, SigDOWN (transcipt, gene) | | FeatPerGene, precision, Pvalues (per comparison) | ✓ | - +-----------+-------------------------------------+------------------+-----------------+----------------+---------------------------------+----------------+------------------------------------------------------+-----------------------------------------+-----------------------------------------+-------------------+-------------------------------------------------------------------+-------+ - + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | Tool | Analysis | Filtering | Normalization | Distribution | Testing | Significance | Results Table | further | SigTables | Clustering | further | Rmd | + +=========+===================================+================+===============+==============+===============================+==============+====================================================+=======================================+=======================================+=================+=================================================================+=====+ + | edgeR | Differential Gene Expression | filterByExpr() | TMM | NB | Fisher’s exact test | pValue, LFC | results, sorted-results | normalized | Sig, SigUP, SigDOWN | MDS-plot | BCV, QLDisp, MD(per comparison) | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | edgeR | Differential Exon Usage | filterByExpr() | TMM | NB | Fisher’s exact test | pValue, LFC | results | normalized | | MDS-plot | BCV, QLDisp, MD(per comparison) | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | edgeR | Differential Alternative Splicing | filterByExpr() | TMM | NB | Simes, gene-level, exon-level | pValue, LFC | results(diffSpliceExonTest, Simes-Test, Gene-Test) | | Sig, SigUP, SigDOWN | MDS-plot | BCV, QLDisp, MD(per comparison), topSpliceSimes-plots(per Gene) | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | DESeq2 | Differential Gene Expression | RowSums >= 10 | RLE | NB | Wald test | pValue, LFC | results | rld, vsd, results(per comparison) | Sig, SigUP, SigDOWN | PCA | Heatmaps, MA(per comparison), VST-and-log2 | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | DEXSeq | Differential Exon Usage | RowSums >= 10 | RLE | Cox-Reid | likelihood ratio test | | | | | | | | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | DEXSeq | Differential Transcript Usage | dmFilter() | RLE | Cox-Reid | likelihood ratio test | pValue | results | | | | | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | DIEGO | Differential Alternative Splicing | | | | Mann-Whitney U test | pValue | results | | Sig | Dendrogram-plot | | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | DRIMSeq | Differential Transcript Usage | dmFilter() | | DM | | pValue, LFC | results(transcript, genes) | Proportions-table, genewise precision | Sig, SigUP, SigDOWN (transcipt, gene) | | FeatPerGene, precision, Pvalues (per comparison) | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + | SPIT | Differential Transcript Usage | IF-dominance | | KDE | Mann-Whitney U | pValue, LFC | results(transcript, genes, ifs) | IF-table | Sig, SigUP, SigDOWN (transcipt, gene) | | PValues, Wasserstein (per comparison) | ✓ | + +---------+-----------------------------------+----------------+---------------+--------------+-------------------------------+--------------+----------------------------------------------------+---------------------------------------+---------------------------------------+-----------------+-----------------------------------------------------------------+-----+ + +.. note:: + + For DTU analyses, transcripts can optionally be pre-grouped with `terminus `_ to collapse transcripts that cannot be confidently distinguished. This is disabled by default and enabled per DTU tool via the ``TERMINUS`` option. + +PostDE (GO/GSEA/GSVA/decoupleR/dream) +##################################### + +PostDE runs enrichment and regulatory analysis on the results of a DE analysis (DESeq2/edgeR only; DEU/DTU/DAS are not supported). It is configured via the top-level ``POSTDE`` section of the config file, see :ref:`config-file` for the analysis JSON schema and examples. + +Outputs are written under ``POSTDE//`` for each DE tool combo: + +* ``manifest.json`` — machine-readable summary of all PostDE outputs. +* ``report_data.rds`` — the full R report data (config, bundle and outputs) for downstream reporting. +* one subdirectory per contrast with the per-contrast result TSVs (enrichment tables, GSVA scores/differential, decoupler activities/differential, dream results). + +GSEA consumes the **full ranked gene list** from the DE analysis — it is never restricted to the significant tables. ORA (gProfiler/clusterProfiler enricher) uses the significant genes defined by the PostDE ``padj``/``lfc`` cutoffs, which are independent of the ``DE.CUTOFFS``; a custom ``background`` file (one gene ID per row, no header) is supported and tested for ORA. + +GSVA runs per contrast on the cohort and design of that contrast, using the expression matrix captured by the DE analysis (optionally spike-in adjusted when the DE run used spike-in normalization). dream runs on the raw counts with the provided metadata/formula (no condition column in the metadata; the condition comes from the DE analysis). Online gProfiler runs require network consent (``allow_network=true``); the saved queries and provenance are stored alongside the results. + +HTML reporting (``plots``/``report``/``shiny``) is not implemented yet — enabling those keys is rejected at validation time. + + TRACKS ############### @@ -286,3 +352,32 @@ Find circular RNAs in mapping data, CIRI2 needs to be installed locally. +=======+=========================================================================================================================================================================================================+========+====================+=======================================================================+========+==================+ | CIRI2 | CIRI (circRNA identifier) is a novel chiastic clipping signal based algorithm,which can unbiasedly and accurately detect circRNAs from transcriptome data by employing multiple filtration strategies. | ciri2 | $Path_to_CIRI2.pl | `ciri2 `_ | BAM | BED/BEDG/BIGWIG | +-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+--------------------+-----------------------------------------------------------------------+--------+------------------+ + + +FUSIONS +############### + +Detect fusion genes with STAR-Fusion. FUSIONS requires STAR in the MAPPING step and, by +default, consumes the chimeric junctions produced by STAR. STAR does not emit chimeric output +unless configured to do so, therefore add the chimeric options to the STAR ``MAP`` options in +the MAPPING config, e.g. ``--chimSegmentMin 12 --chimOutType Junctions``. MONSDA then finds the +resulting ``MAPPED//.Chimeric.out.junction`` files and only runs STAR-Fusion when +they exist. Set ``"FASTQ": "true"`` in the FUSIONS ``OPTIONS`` to run STAR-Fusion directly on the +trimmed FASTQ files instead of the chimeric junctions. + +STAR-Fusion needs a CTAT genome resource library. Point ``INDEX`` to an existing +``genome_lib_dir``; if it does not exist, it is built from ``REFERENCE`` and ``ANNOTATION`` with +``prep_genome_lib.pl`` (extra build options can be passed via ``BUILD``). Building a CTAT library +is resource intensive, so the bundled cluster profile sets a long runtime for the +``generate_ctat_lib`` rule. + +.. table:: + :widths: 10, 40, 10, 10, 10, 10, 10 + :class: tight-table + + +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-----------------------------------------------------------------+-------------------------------+--------+ + | TOOL | DESCRIPTION | ENV | BIN | LINK | INPUT | OUTPUT | + +=============+=====================================================================================================================================================+============+=============+=================================================================+===============================+========+ + | STAR-Fusion | STAR-Fusion identifies candidate fusion transcripts from the chimeric alignments produced by the STAR aligner using a CTAT genome resource library. | starfusion | STAR-Fusion | `starfusion `_ | Chimeric.out.junction / FASTQ | TSV | + +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+-----------------------------------------------------------------+-------------------------------+--------+ + diff --git a/envs/LoveSonesonPatro.yaml b/envs/LoveSonesonPatro.yaml index 2cac941a..4ad7bdda 100644 --- a/envs/LoveSonesonPatro.yaml +++ b/envs/LoveSonesonPatro.yaml @@ -4,132 +4,17 @@ channels: - bioconda - nodefaults dependencies: - - bioconductor-annotate =1.68.0 =r40_0 - - bioconductor-annotationdbi =1.52.0 =r40_0 - - bioconductor-biobase =2.50.0 =r40h037d062_0 - - bioconductor-biocfilecache =1.14.0 =r40_0 - - bioconductor-biocgenerics =0.36.0 =r40_0 - - bioconductor-biocparallel =1.24.0 =r40h5f743cb_0 - - bioconductor-biomart =2.46.0 =r40_0 - - bioconductor-biostrings =2.58.0 =r40h037d062_0 - - bioconductor-delayedarray =0.16.0 =r40h037d062_0 - - bioconductor-deseq2 =1.39.0 =r40h5f743cb_0 - - bioconductor-dexseq =1.36.0 =r40_0 - - bioconductor-drimseq =1.18.0 =r40_0 - - bioconductor-edger =4.4.0 =r40h5f743cb_0 - - bioconductor-genefilter =1.72.0 =r40hc6cf775_0 - - bioconductor-geneplotter =1.68.0 =r40_0 - - bioconductor-genomeinfodb =1.26.0 =r40_0 - - bioconductor-genomeinfodbdata =1.2.4 =r40_0 - - bioconductor-genomicalignments =1.26.0 =r40h037d062_0 - - bioconductor-genomicfeatures =1.42.0 =r40_0 - - bioconductor-genomicranges =1.42.0 =r40h037d062_0 - - bioconductor-iranges =2.24.0 =r40h037d062_0 - - bioconductor-limma =3.46.0 =r40h037d062_0 - - bioconductor-matrixgenerics =1.2.0 =r40_0 - - bioconductor-rhtslib =1.22.0 =r40h037d062_0 - - bioconductor-rsamtools =2.6.0 =r40h5f743cb_0 - - bioconductor-rtracklayer =1.50.0 =r40h9bb0e53_1 - - bioconductor-s4vectors =0.28.0 =r40h037d062_0 - - bioconductor-stager =1.12.0 =r40_0 - - bioconductor-summarizedexperiment =1.20.0 =r40_0 - - bioconductor-tximport =1.18.0 =r40_0 - - bioconductor-xvector =0.30.0 =r40h037d062_0 - - bioconductor-zlibbioc =1.36.0 =r40h037d062_0 - - r-askpass =1.1 =r40hcdcec82_2 - - r-assertthat =0.2.1 =r40h6115d3f_2 - - r-backports =1.2.0 =r40h9e2df91_0 - - r-base =4.0.3 =hd272fe0_2 - - r-bh =1.72.0_3 =r40h6115d3f_1 - - r-bit =4.0.4 =r40hcdcec82_0 - - r-bit64 =4.0.5 =r40hcdcec82_0 - - r-bitops =1.0_6 =r40hcdcec82_1004 - - r-blob =1.2.1 =r40h6115d3f_1 - - r-brio =1.1.0 =r40h9e2df91_1 - - r-callr =3.5.1 =r40h142f84f_0 - - r-cli =2.1.0 =r40h142f84f_0 - - r-colorspace =2.0_0 =r40h9e2df91_0 - - r-crayon =1.3.4 =r40h6115d3f_1003 - - r-curl =4.3 =r40hcdcec82_1 - - r-dbi =1.1.0 =r40h6115d3f_1 - - r-dbplyr =2.0.0 =r40hc72bb7e_0 - - r-desc =1.2.0 =r40h6115d3f_1003 - - r-diffobj =0.3.2 =r40h9e2df91_1 - - r-digest =0.6.27 =r40h1b71b39_0 - - r-dplyr =1.0.2 =r40h0357c0b_0 - - r-ellipsis =0.3.1 =r40hcdcec82_0 - - r-evaluate =0.14 =r40h6115d3f_2 - - r-fansi =0.4.1 =r40hcdcec82_1 - - r-farver =2.0.3 =r40h0357c0b_1 - - r-formatr =1.7 =r40h6115d3f_2 - - r-futile.logger =1.4.3 =r40h6115d3f_1003 - - r-futile.options =1.0.1 =r40h6115d3f_1002 - - r-generics =0.1.0 =r40hc72bb7e_0 - - r-ggplot2 =3.3.2 =r40h6115d3f_0 - - r-glue =1.4.2 =r40hcdcec82_0 - - r-gtable =0.3.0 =r40h6115d3f_3 - - r-hms =0.5.3 =r40h6115d3f_1 - - r-httr =1.4.2 =r40h6115d3f_0 - - r-hwriter =1.3.2 =r40h6115d3f_1003 - - r-isoband =0.2.2 =r40h0357c0b_0 - - r-jsonlite =1.7.1 =r40hcdcec82_0 - - r-labeling =0.4.2 =r40h142f84f_0 - - r-lambda.r =1.2.4 =r40h6115d3f_1 - - r-lattice =0.20_41 =r40hcdcec82_2 - - r-lifecycle =0.2.0 =r40h6115d3f_1 - - r-locfit =1.5_9.4 =r40hcdcec82_1 - - r-magrittr =2.0.1 =r40hc72bb7e_0 - - r-mass =7.3_53 =r40hcdcec82_0 - - r-matrix =1.2_18 =r40h7fa42b6_3 - - r-matrixstats =0.57.0 =r40hcdcec82_0 - - r-memoise =1.1.0 =r40h6115d3f_1004 - - r-mgcv =1.8_33 =r40h7fa42b6_0 - - r-mime =0.9 =r40hcdcec82_1 - - r-munsell =0.5.0 =r40h6115d3f_1003 - - r-nlme =3.1_150 =r40h31ca83e_0 - - r-openssl =1.4.3 =r40he5c4762_0 - - r-pillar =1.4.6 =r40h6115d3f_0 - - r-pkgbuild =1.1.0 =r40h6115d3f_0 - - r-pkgconfig =2.0.3 =r40h6115d3f_1 - - r-pkgload =1.1.0 =r40h0357c0b_0 - - r-plogr =0.2.0 =r40h6115d3f_1003 - - r-plyr =1.8.6 =r40h0357c0b_1 - - r-praise =1.0.0 =r40h6115d3f_1004 - - r-prettyunits =1.1.1 =r40h6115d3f_1 - - r-processx =3.4.4 =r40hcdcec82_0 - - r-progress =1.2.2 =r40h6115d3f_2 - - r-ps =1.4.0 =r40h0eb13af_0 - - r-purrr =0.3.4 =r40hcdcec82_1 - - r-r6 =2.5.0 =r40hc72bb7e_0 - - r-rappdirs =0.3.1 =r40hcdcec82_1004 - - r-rcolorbrewer =1.1_2 =r40h6115d3f_1003 - - r-rcpp =1.0.5 =r40he524a50_0 - - r-rcpparmadillo =0.10.1.2.0 =r40hfbb317b_0 - - r-rcurl =1.98_1.2 =r40hcdcec82_1 - - r-rematch2 =2.1.2 =r40h6115d3f_1 - - r-reshape2 =1.4.4 =r40h0357c0b_1 - - r-rlang =0.4.8 =r40h9e2df91_0 - - r-rprojroot =2.0.2 =r40hc72bb7e_0 - - r-rsqlite =2.2.1 =r40h1b71b39_0 - - r-rstudioapi =0.13 =r40hc72bb7e_0 - - r-scales =1.1.1 =r40h6115d3f_0 - - r-snow =0.4_3 =r40h6115d3f_1002 - - r-statmod =1.4.35 =r40h6e990d7_1 - - r-stringi =1.5.3 =r40h604b29c_0 - - r-stringr =1.4.0 =r40h6115d3f_2 - - r-survival =3.2_7 =r40hcdcec82_0 - - r-sys =3.4 =r40hcdcec82_0 - - r-testthat =3.0.0 =r40he524a50_0 - - r-tibble =3.0.4 =r40h0eb13af_0 - - r-tidyselect =1.1.0 =r40h6115d3f_0 - - r-utf8 =1.1.4 =r40hcdcec82_1003 - - r-vctrs =0.3.5 =r40h9e2df91_0 - - r-viridislite =0.3.0 =r40h6115d3f_1003 - - r-waldo =0.2.3 =r40hc72bb7e_0 - - r-withr =2.3.0 =r40h6115d3f_0 - - r-xml =3.99_0.3 =r40hcdcec82_1 - - r-xml2 =1.3.2 =r40h0357c0b_1 - - r-xtable =1.8_4 =r40h6115d3f_3 - - r-zeallot =0.1.0 =r40h6115d3f_1002 - - readline =8.2 - - sed =4.8 =he412f7d_0 \ No newline at end of file + - bioconductor-annotationdbi =1.72.0 + - bioconductor-biomart =2.66.1 + - bioconductor-deseq2 =1.50.2 + - bioconductor-dexseq =1.56.0 + - bioconductor-drimseq =1.38.0 + - bioconductor-edger =4.8.2 + - bioconductor-genomicfeatures =1.62.0 + - bioconductor-genomicranges =1.62.1 + - bioconductor-rtracklayer =1.70.1 + - bioconductor-summarizedexperiment =1.40.0 + - bioconductor-tximport =1.38.2 + - r-base =4.5.3 + - readline =8.3 + - sed =4.10 diff --git a/envs/alevinfry.yaml b/envs/alevinfry.yaml new file mode 100644 index 00000000..e2f79c76 --- /dev/null +++ b/envs/alevinfry.yaml @@ -0,0 +1,7 @@ +name: alevinfry +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - alevin-fry =0.18.1 diff --git a/envs/dexseq_DEU.yaml b/envs/dexseq_DEU.yaml index f1effda5..4c699a3a 100644 --- a/envs/dexseq_DEU.yaml +++ b/envs/dexseq_DEU.yaml @@ -3,28 +3,24 @@ channels: - conda-forge - bioconda - nodefaults - - r dependencies: - - bcftools =1.16 - - bioconductor-apeglm=1.28.0 - - bioconductor-biomart=2.62.0 - - bioconductor-deseq2=1.46.0 - - bioconductor-dexseq-1.52.0 - - bioconductor-edger=4.4.0 - - bioconductor-enhancedvolcano=1.24.0 - - bioconductor-rsamtools=2.22.0 - - bioconductor-ruvseq=1.40.0 - - htseq =2.0.2 - - htslib =1.16 - - matplotlib =3.7.0 - - matplotlib-base =3.7.0 - - numpy =1.24.2 + - bcftools =1.24 + - bioconductor-apeglm =1.28.0 + - bioconductor-biomart =2.62.0 + - bioconductor-deseq2 =1.46.0 + - bioconductor-dexseq =1.52.0 + - bioconductor-edger =4.4.0 + - bioconductor-enhancedvolcano =1.24.0 + - bioconductor-rsamtools =2.22.0 + - bioconductor-ruvseq =1.40.0 + - htseq =2.0.9 + - matplotlib =3.3.2 + - numpy =1.26.4 - perl =5.32.1 - - python =3.10.8 - - python-dateutil =2.8.2 - - python_abi =3.10 - - pyyaml =6.0 - - readline =8.2 - - samtools =1.21 - - sed =4.8 - - yaml =0.2.5 \ No newline at end of file + - python =3.9.23 + - python-dateutil =2.9.0.post0 + - pyyaml =6.0.2 + - readline =8.3 + - samtools =1.24 + - sed =4.10 + - yaml =0.2.5 diff --git a/envs/dexseq_DTU.yaml b/envs/dexseq_DTU.yaml index ba31fff5..4c699a3a 100644 --- a/envs/dexseq_DTU.yaml +++ b/envs/dexseq_DTU.yaml @@ -4,26 +4,23 @@ channels: - bioconda - nodefaults dependencies: - - bcftools =1.16 - - bioconductor-apeglm=1.28.0 - - bioconductor-biomart=2.62.0 - - bioconductor-deseq2=1.46.0 - - bioconductor-dexseq-1.52.0 - - bioconductor-edger=4.4.0 - - bioconductor-enhancedvolcano=1.24.0 - - bioconductor-rsamtools=2.22.0 - - bioconductor-ruvseq=1.40.0 - - htseq =2.0.2 - - htslib =1.16 - - matplotlib =3.7.0 - - matplotlib-base =3.7.0 - - numpy =1.24.2 + - bcftools =1.24 + - bioconductor-apeglm =1.28.0 + - bioconductor-biomart =2.62.0 + - bioconductor-deseq2 =1.46.0 + - bioconductor-dexseq =1.52.0 + - bioconductor-edger =4.4.0 + - bioconductor-enhancedvolcano =1.24.0 + - bioconductor-rsamtools =2.22.0 + - bioconductor-ruvseq =1.40.0 + - htseq =2.0.9 + - matplotlib =3.3.2 + - numpy =1.26.4 - perl =5.32.1 - - python =3.10.8 - - python-dateutil =2.8.2 - - python_abi =3.10 - - pyyaml =6.0 - - readline =8.2 - - samtools =1.21 - - sed =4.8 - - yaml =0.2.5 \ No newline at end of file + - python =3.9.23 + - python-dateutil =2.9.0.post0 + - pyyaml =6.0.2 + - readline =8.3 + - samtools =1.24 + - sed =4.10 + - yaml =0.2.5 diff --git a/envs/diego_DAS.yaml b/envs/diego_DAS.yaml index e0ec8522..28a98355 100644 --- a/envs/diego_DAS.yaml +++ b/envs/diego_DAS.yaml @@ -4,16 +4,16 @@ channels: - bioconda - nodefaults dependencies: - - bedtools =2.30.0 + - bedtools =2.31.1 - diego =0.1.2 - - graphviz =2.49.1 - - imagemagick =7.1.0_10 - - matplotlib-base =3.5.1 - - numpy =1.22.3 + - graphviz =14.1.2 + - imagemagick =7.1.2_27 + - matplotlib-base =3.11.1 + - numpy =2.5.2 - perl =5.32.1 - - python-dateutil =2.8.2 - - pysam =0.23.0 - - python =3.9 - - readline =8.2 - - samtools =1.21 - - scipy =1.8.0 \ No newline at end of file + - python-dateutil =2.9.0.post0 + - pysam =0.24.0 + - python =3.14.6 + - readline =8.3 + - samtools =1.24 + - scipy =1.18.0 diff --git a/envs/fastqc.yaml b/envs/fastqc.yaml index 8a22aa3d..7359edef 100644 --- a/envs/fastqc.yaml +++ b/envs/fastqc.yaml @@ -3,5 +3,5 @@ channels: - bioconda - conda-forge dependencies: -- multiqc =1.21 +- multiqc =1.35 - fastqc =0.12.1 diff --git a/envs/index.yaml b/envs/index.yaml index 19a4fd8b..64153a8c 100644 --- a/envs/index.yaml +++ b/envs/index.yaml @@ -4,7 +4,7 @@ channels: - bioconda - nodefaults dependencies: - - samtools =1.21 - - curl =7.87.0 + - curl =8.21.0 + - pigz =2.8 + - samtools =1.24 - segemehl =0.2.0 - - pigz =2.6 \ No newline at end of file diff --git a/envs/isoformswitchanalyzer.yaml b/envs/isoformswitchanalyzer.yaml index 4990833a..c215763e 100644 --- a/envs/isoformswitchanalyzer.yaml +++ b/envs/isoformswitchanalyzer.yaml @@ -4,130 +4,13 @@ channels: - bioconda - nodefaults dependencies: - - bioconductor-annotate =1.66.0 - - bioconductor-annotationdbi =1.50.0 - - bioconductor-biobase =2.48.0 - - bioconductor-biocfilecache =1.12.0 - - bioconductor-biocgenerics =0.34.0 - - bioconductor-biocparallel =1.22.0 - - bioconductor-biomart =2.44.0 - - bioconductor-biostrings =2.56.0 - - bioconductor-bsgenome =1.56.0 - - bioconductor-delayedarray =0.14.0 - - bioconductor-deseq2 =1.39.0 - - bioconductor-dexseq =1.34.0 - - bioconductor-drimseq =1.16.0 - - bioconductor-edger =4.4.0 - - bioconductor-genefilter =1.70.0 - - bioconductor-geneplotter =1.66.0 - - bioconductor-genomeinfodb =1.24.0 - - bioconductor-genomeinfodbdata =1.2.3 - - bioconductor-genomicalignments =1.24.0 - - bioconductor-genomicranges =1.40.0 - - bioconductor-iranges =2.22.1 - - bioconductor-isoformswitchanalyzer =1.10.0 - - bioconductor-limma =3.44.1 - - bioconductor-rhtslib =1.20.0 - - bioconductor-rsamtools =2.4.0 - - bioconductor-rtracklayer =1.48.0 - - bioconductor-s4vectors =0.26.0 - - bioconductor-summarizedexperiment =1.18.1 - - bioconductor-tximport =1.16.0 - - bioconductor-xvector =0.28.0 - - bioconductor-zlibbioc =1.34.0 - - r-askpass =1.1 - - r-assertthat =0.2.1 - - r-backports =1.1.8 - - r-base =4.0.2 - - r-bh =1.72.0_3 - - r-bit =4.0.4 - - r-bit64 =4.0.2 - - r-bitops =1.0_6 - - r-blob =1.2.1 - - r-callr =3.4.3 - - r-cli =2.0.2 - - r-clipr =0.7.0 - - r-colorspace =1.4_1 - - r-crayon =1.3.4 - - r-curl =4.3 - - r-dbi =1.1.0 - - r-dbplyr =1.4.4 - - r-desc =1.2.0 - - r-digest =0.6.25 - - r-dplyr =1.0.2 - - r-ellipsis =0.3.1 - - r-evaluate =0.14 - - r-fansi =0.4.1 - - r-farver =2.0.3 - - r-formatr =1.7 - - r-futile.logger =1.4.3 - - r-futile.options =1.0.1 - - r-generics =0.0.2 - - r-ggplot2 =3.3.2 - - r-glue =1.4.1 - - r-gridextra =2.3 - - r-gtable =0.3.0 - - r-hms =0.5.3 - - r-httr =1.4.2 - - r-hwriter =1.3.2 - - r-isoband =0.2.2 - - r-jsonlite =1.7.0 - - r-labeling =0.3 - - r-lambda.r =1.2.4 - - r-lattice =0.20_41 - - r-lifecycle =0.2.0 - - r-locfit =1.5_9.4 - - r-magrittr =1.5 - - r-mass =7.3_52 - - r-matrix =1.2_18 - - r-matrixstats =0.56.0 - - r-memoise =1.1.0 - - r-mgcv =1.8_32 - - r-mime =0.9 - - r-munsell =0.5.0 - - r-nlme =3.1_149 - - r-openssl =1.4.2 - - r-pillar =1.4.6 - - r-pkgbuild =1.1.0 - - r-pkgconfig =2.0.3 - - r-pkgload =1.1.0 - - r-plogr =0.2.0 - - r-plyr =1.8.6 - - r-praise =1.0.0 - - r-prettyunits =1.1.1 - - r-processx =3.4.3 - - r-progress =1.2.2 - - r-ps =1.3.4 - - r-purrr =0.3.4 - - r-r6 =2.4.1 - - r-rappdirs =0.3.1 - - r-rcolorbrewer =1.1_2 - - r-rcpp =1.0.4.6 - - r-rcpparmadillo =0.9.900.2.0 - - r-rcurl =1.98_1.2 - - r-readr =1.3.1 - - r-reshape2 =1.4.4 - - r-rlang =0.4.7 - - r-rprojroot =1.3_2 - - r-rsqlite =2.2.0 - - r-rstudioapi =0.11 - - r-scales =1.1.1 - - r-snow =0.4_3 - - r-statmod =1.4.34 - - r-stringi =1.4.6 - - r-stringr =1.4.0 - - r-survival =3.2_3 - - r-sys =3.4 - - r-testthat =2.3.2 - - r-tibble =3.0.3 - - r-tidyselect =1.1.0 - - r-utf8 =1.1.4 - - r-vctrs =0.3.2 - - r-venndiagram =1.6.20 - - r-viridislite =0.3.0 - - r-withr =2.2.0 - - r-xml =3.99_0.3 - - r-xtable =1.8_4 - - r-zeallot =0.1.0 - - readline =8.2 - - sed =4.8 \ No newline at end of file + - bioconductor-deseq2 =1.50.2 + - bioconductor-dexseq =1.56.0 + - bioconductor-edger =4.8.2 + - bioconductor-genomicranges =1.62.1 + - bioconductor-isoformswitchanalyzer =2.10.0 + - bioconductor-summarizedexperiment =1.40.0 + - bioconductor-tximport =1.38.2 + - r-base =4.5.3 + - readline =8.3 + - sed =4.10 diff --git a/envs/macs2.yaml b/envs/macs2.yaml index 499f93a7..e4c07c47 100644 --- a/envs/macs2.yaml +++ b/envs/macs2.yaml @@ -4,7 +4,9 @@ channels: - bioconda - nodefaults dependencies: + - bc =1.07.1 - macs2 =2.2.7.1 + - perl =5.32.1 - readline =8.2 - samtools =1.21 - sed =4.8 \ No newline at end of file diff --git a/envs/macs3.yaml b/envs/macs3.yaml index 2333cde2..16489c1b 100644 --- a/envs/macs3.yaml +++ b/envs/macs3.yaml @@ -4,7 +4,9 @@ channels: - bioconda - nodefaults dependencies: + - bc =1.07.1 - macs3 =3.0.2 + - perl =5.32.1 - readline =8.2 - samtools =1.21 - sed =4.8 \ No newline at end of file diff --git a/envs/minimap.yaml b/envs/minimap.yaml index 76c64bc4..4afe0606 100644 --- a/envs/minimap.yaml +++ b/envs/minimap.yaml @@ -4,6 +4,6 @@ channels: - bioconda - nodefaults dependencies: - - minimap2 =2.24 + - minimap2 =2.31 - pigz =2.6 - samtools =1.21 \ No newline at end of file diff --git a/envs/monsda.yaml b/envs/monsda.yaml index 697dc095..32a77197 100644 --- a/envs/monsda.yaml +++ b/envs/monsda.yaml @@ -7,8 +7,8 @@ dependencies: - biopython =1.87 - fastapi =0.136.1 - grep >=3.4 - - isort =8.0.1 - - monsda =1.4.0 + - isort =8.0.1 + - monsda =1.5.0 - natsort =8.4.0 - nextflow =26.04.0 - numpy =2.4.3 @@ -26,7 +26,7 @@ dependencies: - snakemake-interface-report-plugins =1.3.0 - snakemake-interface-storage-plugins =4.4.1 - snakemake-storage-plugin-s3 =0.3.6 - - snakemake-storage-plugin-ftp =0.1.3 + - snakemake-storage-plugin-ftp =0.1.3 - snakemake-storage-plugin-http =0.3.1 - snakemake-storage-plugin-zenodo =0.1.5 - - uvicorn =0.46.0 \ No newline at end of file + - uvicorn =0.46.0 diff --git a/envs/oarfish.yaml b/envs/oarfish.yaml new file mode 100644 index 00000000..f8e417e6 --- /dev/null +++ b/envs/oarfish.yaml @@ -0,0 +1,8 @@ +name: oarfish +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - oarfish =0.10.3 + - samtools =1.21 diff --git a/envs/peaks.yaml b/envs/peaks.yaml index 7a028381..d4df03a5 100644 --- a/envs/peaks.yaml +++ b/envs/peaks.yaml @@ -4,224 +4,23 @@ channels: - bioconda - nodefaults dependencies: + - bc =1.07.1 - clustalw =2.1 - curl =7.87.0 - - grep =3.4 - - perl =5.26.2 - - perl-aceperl =1.92 - - perl-algorithm-diff =1.1903 - - perl-algorithm-munkres =0.08 - - perl-apache-test =1.40 - - perl-app-cpanminus =1.7044 - - perl-appconfig =1.71 - - perl-array-compare =3.0.1 - - perl-autoloader =5.74 - - perl-base =2.23 - - perl-bio-asn1-entrezgene =1.73 - - perl-bio-coordinate =1.007001 - - perl-bio-featureio =1.6.905 - - perl-bio-phylo =0.58 - - perl-bio-samtools =1.43 - - perl-bio-tools-phylo-paml =1.7.3 - - perl-bio-tools-run-alignment-clustalw =1.7.4 - - perl-bio-tools-run-alignment-tcoffee =1.7.4 - - perl-bioperl =1.7.2 - - perl-bioperl-core =1.007002 - - perl-bioperl-run =1.007002 - - perl-business-isbn =3.004 - - perl-business-isbn-data =20140910.003 - - perl-cache-cache =1.08 - - perl-capture-tiny =0.48 - - perl-carp =1.38 - - perl-cgi =4.44 - - perl-class-data-inheritable =0.08 - - perl-class-inspector =1.34 - - perl-class-load =0.25 - - perl-class-load-xs =0.10 - - perl-class-method-modifiers =2.12 - - perl-clone =0.42 - - perl-common-sense =3.74 - - perl-compress-raw-zlib =2.087 - - perl-constant =1.33 - - perl-convert-binary-c =0.78 - - perl-convert-binhex =1.125 - - perl-crypt-rc4 =2.02 - - perl-data-dumper =2.173 - - perl-data-optlist =0.110 - - perl-data-stag =0.14 - - perl-date-format =2.30 - - perl-db-file =1.855 - - perl-dbd-sqlite =1.64 - - perl-dbi =1.642 - - perl-devel-globaldestruction =0.14 - - perl-devel-overloadinfo =0.005 - - perl-devel-stacktrace =2.04 - - perl-digest-hmac =1.03 - - perl-digest-md5 =2.55 - - perl-digest-perl-md5 =1.9 - - perl-digest-sha1 =2.13 - - perl-dist-checkconflicts =0.11 - - perl-dynaloader =1.25 - - perl-email-date-format =1.005 - - perl-encode =2.88 - - perl-encode-locale =1.05 - - perl-error =0.17027 - - perl-eval-closure =0.14 - - perl-exception-class =1.44 - - perl-exporter =5.72 - - perl-exporter-tiny =1.002001 - - perl-extutils-makemaker =7.36 - - perl-file-listing =6.04 - - perl-file-path =2.16 - - perl-file-slurp =9999.27 - - perl-file-slurp-tiny =0.004 - - perl-file-sort =1.01 - - perl-file-temp =0.2304 - - perl-file-which =1.23 - - perl-findbin-real =1.05 - - perl-font-afm =1.20 - - perl-font-ttf =1.06 - - perl-gd =2.68 - - perl-getopt-long =2.50 - - perl-graph =0.9704 - - perl-graphviz =2.24 - - perl-html-element-extended =1.18 - - perl-html-entities-numbered =0.04 - - perl-html-formatter =2.16 - - perl-html-parser =3.72 - - perl-html-tableextract =2.13 - - perl-html-tagset =3.20 - - perl-html-tidy =1.60 - - perl-html-tree =5.07 - - perl-html-treebuilder-xpath =0.14 - - perl-http-cookies =6.04 - - perl-http-daemon =6.01 - - perl-http-date =6.02 - - perl-http-message =6.18 - - perl-http-negotiate =6.01 - - perl-image-info =1.38 - - perl-image-size =3.300 - - perl-io-html =1.001 - - perl-io-sessiondata =1.03 - - perl-io-socket-ssl =2.066 - - perl-io-string =1.08 - - perl-io-stringy =2.111 - - perl-io-tty =1.12 - - perl-ipc-run =20180523.0 - - perl-ipc-sharelite =0.17 - - perl-jcode =2.07 - - perl-json =4.02 - - perl-json-xs =2.34 - - perl-lib =0.63 - - perl-libwww-perl =6.39 - - perl-libxml-perl =0.08 - - perl-list-moreutils =0.428 - - perl-list-moreutils-xs =0.428 - - perl-list-util =1.38 - - perl-lwp-mediatypes =6.04 - - perl-lwp-protocol-https =6.07 - - perl-lwp-simple =6.15 - - perl-mailtools =2.21 - - perl-math-cdf =0.1 - - perl-math-derivative =1.01 - - perl-math-random =0.72 - - perl-math-round =0.07 - - perl-math-spline =0.02 - - perl-mime-base64 =3.15 - - perl-mime-lite =3.030 - - perl-mime-tools =5.508 - - perl-mime-types =2.17 - - perl-mldbm =2.05 - - perl-module-implementation =0.09 - - perl-module-load =0.32 - - perl-module-runtime =0.016 - - perl-module-runtime-conflicts =0.003 - - perl-moo =2.003004 - - perl-moose =2.2011 - - perl-mozilla-ca =20180117 - - perl-mro-compat =0.13 - - perl-net-http =6.19 - - perl-net-ssleay =1.88 - - perl-ntlm =1.09 - - perl-ole-storage_lite =0.19 - - perl-package-deprecationmanager =0.17 - - perl-package-stash =0.38 - - perl-package-stash-xs =0.28 - - perl-params-util =1.07 - - perl-parent =0.236 - - perl-parse-recdescent =1.967015 - - perl-pathtools =3.75 - - perl-pdf-api2 =2.035 - - perl-perlio-gzip =0.20 - - perl-pod-escapes =1.07 - - perl-pod-usage =1.69 - - perl-postscript =0.06 - - perl-role-tiny =2.000008 - - perl-scalar-list-utils =1.52 - - perl-set-scalar =1.29 - - perl-soap-lite =1.19 - - perl-socket =2.027 - - perl-sort-naturally =1.03 - - perl-spreadsheet-parseexcel =0.65 - - perl-spreadsheet-writeexcel =2.40 - - perl-statistics-descriptive =3.0702 - - perl-storable =3.15 - - perl-sub-exporter =0.987 - - perl-sub-exporter-progressive =0.001013 - - perl-sub-identify =0.14 - - perl-sub-install =0.928 - - perl-sub-name =0.21 - - perl-sub-quote =2.006003 - - perl-sub-uplevel =0.2800 - - perl-svg =2.84 - - perl-svg-graph =0.02 - - perl-task-weaken =1.06 - - perl-template-toolkit =2.26 - - perl-test =1.26 - - perl-test-deep =1.128 - - perl-test-differences =0.67 - - perl-test-exception =0.43 - - perl-test-harness =3.42 - - perl-test-leaktrace =0.16 - - perl-test-most =0.35 - - perl-test-requiresinternet =0.05 - - perl-test-warn =0.36 - - perl-text-diff =1.45 - - perl-tie-hash-indexed =0.05 - - perl-tie-ixhash =1.23 - - perl-time-hires =1.9760 - - perl-time-local =1.28 - - perl-timedate =2.30 - - perl-tree-dag_node =1.31 - - perl-try-tiny =0.30 - - perl-type-tiny =1.004004 - - perl-types-serialiser =1.0 - - perl-unicode-map =0.112 - - perl-uri =1.76 - - perl-www-robotrules =6.02 - - perl-xml-dom =1.46 - - perl-xml-dom-xpath =0.14 - - perl-xml-filter-buffertext =1.01 - - perl-xml-libxml =2.0132 - - perl-xml-libxslt =1.94 - - perl-xml-namespacesupport =1.12 - - perl-xml-parser =2.44_01 - - perl-xml-regexp =0.04 - - perl-xml-sax =1.02 - - perl-xml-sax-base =1.09 - - perl-xml-sax-expat =0.51 - - perl-xml-sax-writer =0.57 - - perl-xml-simple =2.25 - - perl-xml-twig =3.52 - - perl-xml-writer =0.625 - - perl-xml-xpath =1.44 - - perl-xml-xpathengine =0.14 - - perl-xsloader =0.24 - - perl-yaml =1.29 - - pigz =2.6 - - pip =22.3.1 - - python =3.7.16 - - readline =8.2 - - setuptools =66.1.0 - - sqlite =3.40.1 - - t_coffee =11.0.8 \ No newline at end of file + - grep =3.12 + - perl =5.22.0.1 + - perl-bioperl =1.6.924 + - perl-bioperl-core =1.6.924 + - perl-bioperl-run =1.006900 + - perl-gd =2.56 + - perl-graphviz =2.20 + - perl-html-tree =5.03 + - perl-libwww-perl =6.15 + - perl-xml-twig =3.49 + - pigz =2.8 + - pip =21.3.1 + - python =3.6.15 + - readline =8.3 + - setuptools =58.0.4 + - sqlite =3.53.4 + - t_coffee =11.0.8 diff --git a/envs/piranha.yaml b/envs/piranha.yaml index 9750ad4d..2f6cb8f9 100644 --- a/envs/piranha.yaml +++ b/envs/piranha.yaml @@ -5,5 +5,7 @@ channels: - nodefaults dependencies: - bamtools =2.5.1 + - bc =1.07.1 + - perl =5.32.1 - piranha =1.2.1 - readline =8.2 \ No newline at end of file diff --git a/envs/piscem.yaml b/envs/piscem.yaml new file mode 100644 index 00000000..77a086d9 --- /dev/null +++ b/envs/piscem.yaml @@ -0,0 +1,8 @@ +name: piscem +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - piscem =0.23.0 + - samtools =1.21 diff --git a/envs/postde.yaml b/envs/postde.yaml new file mode 100644 index 00000000..f3e1408d --- /dev/null +++ b/envs/postde.yaml @@ -0,0 +1,28 @@ +name: postde +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - r-base=4.5.3 + - r-jsonlite=2.0.0 + - r-gprofiler2=0.2.4 + - bioconductor-clusterprofiler=4.18.4 + - bioconductor-deseq2=1.50.2 + - bioconductor-decoupler=2.16.0 + - bioconductor-edger=4.8.2 + - bioconductor-fgsea=1.36.2 + - bioconductor-glimma=2.20.0 + - bioconductor-go.db=3.22.0 + - bioconductor-gsva=2.4.9 + - bioconductor-limma=3.66.0 + - bioconductor-variancepartition=1.40.2 + - bioconductor-biocparallel=1.44.0 + - r-plotly=4.12.1 + - r-htmlwidgets=1.6.4 + - r-shiny=1.14.0 + - r-dt=0.34.0 + - r-rmarkdown=2.32 + - r-knitr=1.52 + - r-lme4=2.0_6 + - quarto=1.9.38 diff --git a/envs/rammap.yaml b/envs/rammap.yaml new file mode 100644 index 00000000..2e613e27 --- /dev/null +++ b/envs/rammap.yaml @@ -0,0 +1,10 @@ +name: rammap +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - rammap =1.1.2 + - zlib + - pigz =2.6 + - samtools =1.21 diff --git a/envs/rastqc.yaml b/envs/rastqc.yaml new file mode 100644 index 00000000..9c97adde --- /dev/null +++ b/envs/rastqc.yaml @@ -0,0 +1,7 @@ +name: qc +channels: +- bioconda +- conda-forge +dependencies: +- multiqc =1.35 +- rastqc =0.1.0 diff --git a/envs/rustar.yaml b/envs/rustar.yaml new file mode 100644 index 00000000..2004ea2a --- /dev/null +++ b/envs/rustar.yaml @@ -0,0 +1,11 @@ +name: rustar +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - samtools =1.21 + - rust >=1.85 + - git + - openssl + - pkg-config diff --git a/envs/rustqc.yaml b/envs/rustqc.yaml index 5f268a0f..f47198a7 100644 --- a/envs/rustqc.yaml +++ b/envs/rustqc.yaml @@ -4,4 +4,4 @@ channels: - conda-forge dependencies: - rustqc =0.2.1 -- multiqc =1.21 +- multiqc =1.35 diff --git a/envs/salmon.yaml b/envs/salmon.yaml index cfff7361..9c9d254d 100644 --- a/envs/salmon.yaml +++ b/envs/salmon.yaml @@ -2,5 +2,8 @@ name: salmon channels: - conda-forge - bioconda + - nodefaults dependencies: - - salmon =1.11.4 \ No newline at end of file + - salmon =2.7.0 + - samtools =1.21 + - pigz =2.6 \ No newline at end of file diff --git a/envs/scyphy.yaml b/envs/scyphy.yaml index d71a067d..a01aa7ae 100644 --- a/envs/scyphy.yaml +++ b/envs/scyphy.yaml @@ -5,7 +5,9 @@ channels: - nodefaults dependencies: - bamtools =2.5.1 + - bc =1.07.1 - bzip2 =1.0.8 + - perl =5.32.1 - piranha =1.2.1 - pyfaidx =0.7.0 - pysam =0.20.0 diff --git a/envs/simpleaf.yaml b/envs/simpleaf.yaml new file mode 100644 index 00000000..1c96f56b --- /dev/null +++ b/envs/simpleaf.yaml @@ -0,0 +1,10 @@ +name: simpleaf +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - simpleaf =0.30.0 + - piscem =0.23.0 + - alevin-fry =0.18.1 + - pyroe diff --git a/envs/spit_DTU.yaml b/envs/spit_DTU.yaml new file mode 100644 index 00000000..e6a7c3ec --- /dev/null +++ b/envs/spit_DTU.yaml @@ -0,0 +1,19 @@ +name: spit_DTU +channels: + - bioconda + - conda-forge + - nodefaults +dependencies: + - python >=3.9 + - pandas >=1.3.4 + - numpy >=1.20.3 + - scipy >=1.11.1 + - scikit-learn >=1.1.1 + - matplotlib-base >=3.4.3 + - matplotlib-venn >=0.11.6 + - seaborn >=0.11.2 + - tqdm >=4.62.3 + - sed=4.8 + - pip + - pip: + - spit diff --git a/envs/starfusion.yaml b/envs/starfusion.yaml new file mode 100644 index 00000000..4f431f93 --- /dev/null +++ b/envs/starfusion.yaml @@ -0,0 +1,7 @@ +name: starfusion +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - star-fusion =1.15.1 diff --git a/envs/terminus.yaml b/envs/terminus.yaml new file mode 100644 index 00000000..2398124b --- /dev/null +++ b/envs/terminus.yaml @@ -0,0 +1,7 @@ +name: terminus +channels: + - conda-forge + - bioconda + - nodefaults +dependencies: + - terminus diff --git a/envs/trimm.yaml b/envs/trimm.yaml index 5c6aa9fb..155e1cbc 100644 --- a/envs/trimm.yaml +++ b/envs/trimm.yaml @@ -4,12 +4,12 @@ channels: - bioconda - nodefaults dependencies: - - bbmap =38.22 - - cutadapt =1.18 - - fastqc =0.11.8 - - trim-galore =0.5.0 - - perl =5.26.2 - - pigz =2.6 - - pip =18.1 - - python =3.9 + - bbmap =40.02 + - cutadapt =5.2 + - fastqc =0.12.1 + - trim-galore =2.3.0 + - perl =5.32.1 + - pigz =2.8 + - pip =26.2.1 + - python =3.13.15 - rename =1.601 diff --git a/envs/ucsc.yaml b/envs/ucsc.yaml index 3426fa6c..642312d4 100644 --- a/envs/ucsc.yaml +++ b/envs/ucsc.yaml @@ -4,7 +4,9 @@ channels: - bioconda - nodefaults dependencies: + - bc =1.07.1 - grep =2.14 + - perl =5.32.1 - ucsc-bedgraphtobigwig =377 - ucsc-beditemoverlapcount =377 - ucsc-fatotwobit =377 diff --git a/profile_nextflow/nextflow.config b/profile_nextflow/nextflow.config index 6d15e289..6079f069 100644 --- a/profile_nextflow/nextflow.config +++ b/profile_nextflow/nextflow.config @@ -16,10 +16,10 @@ profiles { // Non-default overrides only – rules that need more than the global defaults. process { - withName: 'bwa_idx|bwa2_idx|bwameth_idx|hisat2_idx|minimap_idx|segemehl_idx|segemehl3_idx|star_idx' { + withName: 'bwa_idx|bwa2_idx|bwameth_idx|hisat2_idx|minimap_idx|segemehl_idx|segemehl3_idx|star_idx|generate_ctat_lib' { memory = { 80.GB * (1 << ((task.attempt ?: 1) - 1)) } } - withName: 'bwa_mapping|bwa2_mapping|bwameth_mapping|hisat2_mapping|minimap_mapping|segemehl_mapping|segemehl3_mapping|star_mapping' { + withName: 'bwa_mapping|bwa2_mapping|bwameth_mapping|hisat2_mapping|minimap_mapping|segemehl_mapping|segemehl3_mapping|star_mapping|starfusion' { memory = { 80.GB * (1 << ((task.attempt ?: 1) - 1)) } } withName: 'salmon_idx|kallisto_idx' { @@ -33,7 +33,13 @@ profiles { queue = 'medium' time = { 18.h * (1 << ((task.attempt ?: 1) - 1)) } } - withName: 'run_drimseq|run_dexseq' { + withName: 'postde' { + memory = { 24.GB * (1 << ((task.attempt ?: 1) - 1)) } + queue = 'medium' + time = { 6.h * (1 << ((task.attempt ?: 1) - 1)) } + maxRetries = 3 + } + withName: 'run_drimseq|run_dexseq|run_spit' { memory = { 40.GB * (1 << ((task.attempt ?: 1) - 1)) } queue = 'long' time = { 120.h * (1 << ((task.attempt ?: 1) - 1)) } diff --git a/profile_snakemake/cluster_config.yaml b/profile_snakemake/cluster_config.yaml deleted file mode 100644 index f9f7b122..00000000 --- a/profile_snakemake/cluster_config.yaml +++ /dev/null @@ -1,18 +0,0 @@ -__default__: - account: ${USER} # your account name - partition: c # the partition to use - time: 1500 # default time (minutes) - nodes: 1 - output: "/SLURMLOG/{rule}.{wildcards}.out" - error: "/SLURMLOG/{rule}.{wildcards}.err" - #ntasks: 1 - #mem: 14GB # default memory - -generate_index: - mem: 200GB - -mapping: - mem: 200GB - -multiqc: - mem: 30GB diff --git a/profile_snakemake/config.v8+.yaml b/profile_snakemake/config.v8+.yaml index a984a4d2..7922f675 100755 --- a/profile_snakemake/config.v8+.yaml +++ b/profile_snakemake/config.v8+.yaml @@ -5,7 +5,9 @@ default-resources: slurm_partition: "${SLURM_PARTITION}" # the partition to use nodes: 1 mem_mb: "20000 * (2 ** (attempt - 1))" - runtime: "480 * (2 ** (attempt - 1))" + runtime: "480 * (2 ** (attempt - 1))" + partition: "('c_long' if min(480 * (2 ** (attempt - 1)), 960) > 2880 else 'c_medium' if min(480 * (2 ** (attempt - 1)), 960) > 1440 else 'c_short')" + qos: "('long' if min(480 * (2 ** (attempt - 1)), 960) > 2880 else 'medium' if min(480 * (2 ** (attempt - 1)), 960) > 1440 else 'short')" set-resources: generate_index: mem_mb: "80000 * (2 ** (attempt - 1))" @@ -25,6 +27,10 @@ set-resources: mem_mb: "40000 * (2 ** (attempt - 1))" slurm_qos: "medium" runtime: "18h" + postde: + mem_mb: "24000 * (2 ** (attempt - 1))" + slurm_qos: "medium" + runtime: "6h" salmon_index: mem_mb: "40000 * (2 ** (attempt - 1))" sortsam: @@ -41,6 +47,12 @@ set-resources: gridss_setupreference: mem_mb: "40000 * (2 ** (attempt - 1))" threads: 8 + generate_ctat_lib: + mem_mb: "80000 * (2 ** (attempt - 1))" + runtime: "24h" + starfusion: + mem_mb: "100000 * (2 ** (attempt - 1))" + runtime: "24h" jobs: 1000 keep-going: True rerun-incomplete: True diff --git a/profile_snakemake/config.yaml b/profile_snakemake/config.yaml deleted file mode 100644 index a4e4edb5..00000000 --- a/profile_snakemake/config.yaml +++ /dev/null @@ -1,12 +0,0 @@ -restart-times: 3 -jobscript: "slurm-jobscript.sh" -cluster-generic-submit-cmd: "slurm-submit.py" -#cluster-status: "slurm-status.py" -max-jobs-per-second: 1 -max-status-checks-per-second: 3 -local-cores: 1 -latency-wait: 600 -#use-conda: True -keep-going: True -rerun-incomplete: True -#printshellcmds: True diff --git a/profile_snakemake/slurm-jobscript.sh b/profile_snakemake/slurm-jobscript.sh deleted file mode 100755 index 391741ef..00000000 --- a/profile_snakemake/slurm-jobscript.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash -# properties = {properties} -{exec_job} diff --git a/profile_snakemake/slurm-status.py b/profile_snakemake/slurm-status.py deleted file mode 100755 index 688c2276..00000000 --- a/profile_snakemake/slurm-status.py +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env python3 -import re -import subprocess as sp -import shlex -import sys -import time -import logging - -logger = logging.getLogger("__name__") - -STATUS_ATTEMPTS = 20 - -jobid = sys.argv[1] - - # noqa: E999,E225 -cluster = "--cluster=cluster" - # noqa: E225 - -for i in range(STATUS_ATTEMPTS): - try: - sacct_res = sp.check_output(shlex.split(f"sacct {cluster} -P -b -j {jobid} -n")) - res = { - x.split("|")[0]: x.split("|")[1] - for x in sacct_res.decode().strip().split("\n") - } - break - except sp.CalledProcessError as e: - logger.error("sacct process error") - logger.error(e) - except IndexError as e: - pass - # Try getting job with scontrol instead in case sacct is misconfigured - try: - sctrl_res = sp.check_output( - shlex.split(f"scontrol {cluster} -o show job {jobid}") - ) - m = re.search(r"JobState=(\w+)", sctrl_res.decode()) - res = {jobid: m.group(1)} - break - except sp.CalledProcessError as e: - logger.error("scontrol process error") - logger.error(e) - if i >= STATUS_ATTEMPTS - 1: - print("failed") - exit(0) - else: - time.sleep(1) - -status = res[jobid] - -if status == "BOOT_FAIL": - print("failed") -elif status == "OUT_OF_MEMORY": - print("failed") -elif status.startswith("CANCELLED"): - print("failed") -elif status == "COMPLETED": - print("success") -elif status == "DEADLINE": - print("failed") -elif status == "FAILED": - print("failed") -elif status == "NODE_FAIL": - print("failed") -elif status == "PREEMPTED": - print("failed") -elif status == "TIMEOUT": - print("failed") -# Unclear whether SUSPENDED should be treated as running or failed -elif status == "SUSPENDED": - print("failed") -else: - print("running") diff --git a/profile_snakemake/slurm-submit-advanced.py b/profile_snakemake/slurm-submit-advanced.py deleted file mode 100755 index bfe9509f..00000000 --- a/profile_snakemake/slurm-submit-advanced.py +++ /dev/null @@ -1,292 +0,0 @@ -#!/usr/bin/env python3 -import sys -import os -import re -import math -import argparse -import subprocess -from snakemake.utils import read_job_properties - - -############################## -# Helper functions -############################## -def _get_default_partition(): - """Retrieve default partition for cluster""" - if "": - return "" - cmd = "sinfo -O \"partition\"" - res = subprocess.run(cmd, check=True, shell=True, - stdout=subprocess.PIPE) - m = re.search("(?P\S+)\*", res.stdout.decode(), re.M) - partition = m.group("partition") - return partition - - -def _get_cluster_configuration(partition): - """Retrieve cluster configuration for a partition.""" - # Retrieve partition info; we tacitly assume we only get one response - cmd = " ".join( - ["sinfo -e -O \"partition,cpus,memory,time,size,maxcpuspernode\"", - "-h -p {}".format(partition)]) - res = subprocess.run(cmd, check=True, shell=True, stdout=subprocess.PIPE) - m = re.search("(?P\S+)\s+(?P\d+)\s+(?P\S+)\s+((?P\d+)-)?(?P\d+):(?P\d+):(?P\d+)\s+(?P\S+)\s+(?P\S+)", - res.stdout.decode()) - if m is None: - m = re.search("(?P\S+)\s+(?P\d+)\s+(?P\S+)\s+(?P