Clarify config.yaml and samples.csv requirements in cluster submission instructions

CLUSTER_INTERACTIVE_INSTRUCTIONS.md

@@ -0,0 +1,118 @@
+# Dundee cluster interactive instructions (no shared filesystem)
+
+These notes describe how to run the **myRna-seq** Snakemake workflow on the Dundee
+cluster **inside a single interactive job**. Because the cluster has **no shared
+filesystem**, you must stage the project, references, and FASTQs on the compute
+node and write outputs to node-local storage. At the end, sync the results back
+to your persistent storage.
+
+## 1) Start an interactive session
+
+Follow the same interactive workflow used for other Dundee pipelines, then
+request resources with the SGE flags used by this project.
+
+```bash
+# 1. Connect to the cluster
+ssh compute.dundee.ac.uk
+
+# 2. Start a screen session (protects against connection loss)
+screen -S snakemake_job
+
+# If connection drops, reconnect with:
+# ssh compute.dundee.ac.uk
+# screen -r snakemake_job
+
+# 3. Request an interactive session with the SGE settings used by this pipeline
+qrsh -pe smp 40 \
+  -adds l_hard local_free 200G \
+  -mods l_hard m_mem_free 20G \
+  -adds l_hard avx 1
+```
+
+Once the session starts you will be on a compute node.
+
+## 2) Stage the project and inputs on local scratch
+
+Navigate to your lab folder, clone the repository (first time only), and set up
+your environment (first time only). Then copy data to node-local scratch for
+the run.
+
+```bash
+# 4. Navigate to your lab folder
+cd /cluster/majf_lab/mtinti  # Replace with your lab folder path
+
+# 5. Clone the repository (this step is done only once)
+git clone https://github.com/mtinti/myRna-seq.git
+
+# 6. Create and activate conda environment (this step is done only once)
+conda create -n snakemake snakemake
+# 6b. (This step is every time)
+conda activate snakemake
+
+# 7. Navigate to the repository
+cd myRna-seq
+# 7b. git pull (this step is every time)
+```
+
+Pick a run directory on the node (`$TMPDIR` is node-local) and copy the
+pipeline repository plus your input data there.
+
+```bash
+RUN_DIR="$TMPDIR/myRna-seq-$USER-$(date +%Y%m%d_%H%M%S)"
+mkdir -p "$RUN_DIR"
+
+# Copy the pipeline repository (exclude git metadata to keep it small)
+rsync -a --exclude '.git' /cluster/majf_lab/mtinti/myRna-seq/ "$RUN_DIR/myRna-seq/"
+
+# Copy input FASTQs and references (examples)
+rsync -a /path/to/fastqs/ "$RUN_DIR/fastqs/"
+rsync -a /path/to/reference/ "$RUN_DIR/reference/"
+```
+
+> **Tip:** If your login node storage is not visible from the compute node,
+> replace the `rsync` steps with whatever transfer method you use on Dundee
+> (e.g., `scp` from a data transfer node).
+
+## 3) Configure the workflow for node-local paths
+
+Set the paths in `config.yaml` to match where you staged the data. You can either
+edit the file directly or override values at runtime.
+
+Example override at runtime:
+
+```bash
+cd "$RUN_DIR/myRna-seq"
+
+snakemake --cores 40 \
+  --config \
+  processing_dir="$RUN_DIR/processing" \
+  results_dir="$RUN_DIR/results" \
+  benchmark_dir="$RUN_DIR/benchmarks" \
+  reference_fasta="$RUN_DIR/reference/genome.fa" \
+  gtf_file="$RUN_DIR/reference/annotation.gtf" \
+  samples_csv="$RUN_DIR/samples.csv"
+```
+
+## 4) Sync results back to persistent storage
+
+When the workflow finishes, copy outputs back to your long-term storage:
+
+```bash
+rsync -a "$RUN_DIR/results/" /path/to/output/results/
+rsync -a "$RUN_DIR/benchmarks/" /path/to/output/benchmarks/
+```
+
+You can also archive logs if needed:
+
+```bash
+rsync -a "$RUN_DIR/.snakemake/log/" /path/to/output/snakemake-logs/
+```
+
+---
+
+### Notes
+
+- This approach runs the full workflow on **one node**; avoid Snakemake cluster
+  submission (`--cluster`) because compute nodes do not share a filesystem.
+- Set `max_cores` and the `cores_*` values in `config.yaml` to match your job
+  allocation if you override these defaults.

CLUSTER_SUBMISSION_INSTRUCTIONS.md

@@ -0,0 +1,49 @@
+# Dundee cluster batch submission instructions (no shared filesystem)
+
+These instructions submit **myRna-seq** as a single batch job using `qsub` and a
+wrapper script. The wrapper stages the workflow into node-local scratch, runs
+Snakemake, and syncs results back at the end. The wrapper is configured for the
+Dundee cluster’s SGE directives shown at the top of the script.
+
+## 1) Review the wrapper script
+
+Open `submit_snakemake_cluster.sh` and update resource requests (cores, memory,
+walltime) and any path overrides.
+
+Before submitting, make sure your pipeline configuration is complete in
+`config.yaml` and that it references a properly paired `samples.csv` file. The
+required `samples.csv` format and configuration instructions are documented in
+the GitHub README for this repository.
+
+Key variables you can customize inside the script:
+
+- `CORES`: Snakemake cores to use (defaults to 40 to match `-pe smp 40`).
+- `CONFIG_OVERRIDES`: Runtime overrides for `processing_dir`, `results_dir`,
+  `benchmark_dir`, and any reference/sample paths.
+
+## 2) Submit the job
+
+From the directory containing the pipeline repository:
+
+```bash
+qsub submit_snakemake_cluster.sh
+```
+
+Check the job queue with `qstat` and review the job’s stdout/stderr log once it
+finishes.
+
+## 3) Retrieve results
+
+The wrapper script syncs `results/`, `benchmarks/`, and Snakemake logs back to
+your submission directory by default. Adjust the `RESULTS_DEST` variable in the
+wrapper if you want outputs elsewhere.
+
+---
+
+### Notes
+
+- Because the cluster has **no shared filesystem**, the workflow runs on a
+  **single node**. Do not use Snakemake cluster submission.
+- If the compute node cannot access your submission directory, update the
+  staging and sync commands in the wrapper to use your preferred transfer
+  method (e.g., `scp` from a data transfer node).

submit_snakemake_cluster.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+#$ -adds l_hard local_free 200G
+#$ -mods l_hard m_mem_free 20G
+#$ -adds l_hard avx 1
+#$ -cwd
+#$ -V
+#$ -j y
+#$ -N snakemake_oligo
+#$ -o snakemake_oligo_errors_$JOB_ID
+#$ -pe smp 40
+
+# Exit on error and undefined variables
+set -e
+set -u
+
+WORKDIR="${SGE_O_WORKDIR:-$PWD}"
+JOB_ID="${JOB_ID:-manual}"
+RUN_ROOT="${TMPDIR:-/tmp}"
+RUN_DIR="$RUN_ROOT/myRna-seq-${USER}-${JOB_ID}"
+RESULTS_DEST="${RESULTS_DEST:-$WORKDIR}"
+
+CORES="${CORES:-40}"
+CONFIG_OVERRIDES=(
+  "processing_dir=$RUN_DIR/processing"
+  "results_dir=$RUN_DIR/results"
+  "benchmark_dir=$RUN_DIR/benchmarks"
+)
+
+mkdir -p "$RUN_DIR"
+
+rsync -a \
+  --exclude '.git' \
+  --exclude 'processing' \
+  --exclude 'results' \
+  --exclude 'benchmarks' \
+  "$WORKDIR/" "$RUN_DIR/myRna-seq/"
+
+cd "$RUN_DIR/myRna-seq"
+
+SNK_ARGS=(--cores "$CORES" --config "${CONFIG_OVERRIDES[@]}")
+
+snakemake "${SNK_ARGS[@]}"
+
+mkdir -p "$RESULTS_DEST/results" "$RESULTS_DEST/benchmarks" "$RESULTS_DEST/snakemake-logs"
+
+rsync -a "$RUN_DIR/results/" "$RESULTS_DEST/results/"
+rsync -a "$RUN_DIR/benchmarks/" "$RESULTS_DEST/benchmarks/"
+
+if [[ -d "$RUN_DIR/.snakemake/log" ]]; then
+  rsync -a "$RUN_DIR/.snakemake/log/" "$RESULTS_DEST/snakemake-logs/"
+fi
+
+echo "Run directory: $RUN_DIR"
+echo "Results synced to: $RESULTS_DEST"