From a7f463076104b8f9f221e604243a41be6118adb8 Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 15:29:24 +0200 Subject: [PATCH 1/9] =?UTF-8?q?docs:System=20context,=20service=20inventor?= =?UTF-8?q?y,=20request=20flows=20(embedding=E2=86=92GO,=20sequence?= =?UTF-8?q?=E2=86=92GO,=20async=20jobs,=20training),=20data/artifact=20flo?= =?UTF-8?q?w,=20Compose=20topology,=20security=20(NGINX=20tiers),=20integr?= =?UTF-8?q?ation=20points,=20design=20decisions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/architecture.md | 217 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..7d852d6 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,217 @@ +# Architecture + +This document describes how ProSeqGO is structured: services, data flows, security boundaries, and key design decisions. For operational runbooks, see [monitoring.md](monitoring.md) and [troubleshooting.md](troubleshooting.md). + +## System context + +ProSeqGO is an end-to-end MLOps platform for **multi-label Gene Ontology (GO) prediction** from protein sequences. It targets: + +| Audience | Primary entry point | +|----------|---------------------| +| Product / lab users | Streamlit UI (`/ui/`) or GO prediction API (`/api/predict/`) | +| ML engineers | CLI scripts (`scripts/`), Training API (`/api/train/`) | +| Platform / ops | Docker Compose, NGINX gateway, Prometheus/Grafana | +| Auditors | MLflow tracking UI (`/mlflow/`) | + +**In scope:** preprocessing, embedding generation, model training, registry-based serving, secured gateway routing, async job queues, observability. + +**External dependencies:** + +- [Kaggle CAFA 5/6 training dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) +- Hugging Face protein language models (ESM2, ProtBERT, ProtT5) +- Local or containerized infrastructure (Postgres, Redis, MinIO) + +## Logical architecture + +```text +User / Client + | + v +NGINX (TLS + Basic Auth + Rate Limit + Routing) + |-----------------------> /ui/ -----------------------> Streamlit UI + |-----------------------> /api/v1/* ------------------> Embedding API + | |-> Go Prediction API (/predict) + |-----------------------> /api/predict/* -------------> Go Prediction API + |-----------------------> /api/train* ----------------> Training API (profile: training) + |-----------------------> /mlflow/* ------------------> MLflow UI / Registry + +Prometheus <---------------- /metrics from embedding/go/training workers +Grafana <------------------- Prometheus datasource +``` + +### Service inventory + +| Service | Role | Profile | +|---------|------|---------| +| `nginx` | Single public ingress (ports 80/443) | default | +| `embedding-api` | FastAPI: async embedding jobs, sequence→GO orchestration | default | +| `embedding-worker` | RQ worker: runs embedding jobs, exposes worker metrics | default | +| `go-prediction-api` | FastAPI: embedding→GO inference from registry `@champion` | default | +| `streamlit-ui` | Interactive UI over gateway | default | +| `mlflow` | Experiment tracking and model registry | default | +| `postgres` | MLflow backend store + `proseqgo_jobs` job history DB | default | +| `redis` | RQ job dispatch (embedding + training) | default | +| `minio` | S3-compatible artifact store for MLflow | default | +| `trainer-api` / `trainer-worker` | Async retrain jobs | `training` | +| `prometheus` / `grafana` / `redis-exporter` | Observability | `monitoring` | +| `postgres-backup` / `backup-offload` | Daily Postgres dumps, MinIO offload | default (skipped in CI overlay) | + +## Request flows + +### 1. Embedding → GO (direct inference) + +```text +Client → NGINX (/api/predict/predict) → go-prediction-api + ↓ + MLflow registry (@champion) + ↓ + GO term predictions (top_k) +``` + +The GO prediction API loads `models:/@champion` (default: `cafa-go-model@champion`) and validates embedding dimension before inference. + +### 2. Sequence → GO (orchestrated) + +```text +Client → NGINX (/api/v1/predict-go-from-sequences|fasta) + → embedding-api + → create embedding job (Postgres + Redis/RQ) + → embedding-worker processes job + → load test_embeddings.npy + → for each sequence: go-prediction-api /predict + → aggregated PredictGoResponse +``` + +Sync wrappers poll the embedding job with configurable `timeout_seconds` (default 1800 s) and `poll_interval_seconds`. + +### 3. Async embedding job + +```text +Client → embedding-api POST /api/v1/jobs + → Postgres (proseqgo_jobs.embedding_jobs, status=queued) + → Redis/RQ enqueue → embedding-worker + → artifacts under outputs/service_artifacts/{job_id}/ +Client polls GET /api/v1/jobs/{job_id} +Client downloads GET /api/v1/jobs/{job_id}/artifacts/{name} +``` + +Postgres is the source of truth for job status; Redis holds transient dispatch state. + +### 4. Training / retraining (optional profile) + +```text +Client → NGINX (/api/train/train) → trainer-api + → Postgres + Redis/RQ → trainer-worker + → scripts/retrain_pipeline.py (train → eval → promote) + → MLflow runs, model registration, optional champion promotion +``` + +## Data and artifact flow + +```text +Kaggle dataset (FASTA + terms) + ↓ +scripts/preprocess.py → label matrix (outputs/) +scripts/split_train_holdout.py → deterministic splits (outputs/splits/) +scripts/embed_sequences.py → embeddings (data/embeddings/) + ↓ +scripts/train.py → checkpoints + MLflow run + model version +scripts/evaluate_holdout.py → holdout metrics +scripts/promote_model.py → champion alias (if metric ≥ threshold) + ↓ +go-prediction-api serves @champion +``` + +See [data.md](data.md) and [training.md](training.md) for detail. + +## Deployment topology + +Compose uses a **portable base** plus optional overlays: + +| File | Purpose | +|------|---------| +| `docker-compose.yml` | All services; CPU-safe base (no `gpus:`) | +| `docker-compose.gpu.yml` | Adds `gpus: all` for inference/training workers | +| `docker-compose.ci.yml` | CPU smoke: `CAFA_DEVICE=cpu`, skips backup sidecars | + +`make up` auto-adds the GPU overlay when `nvidia-smi` is available. + +**Networks:** all services share the `proseqgo` bridge network. Only NGINX (80/443), Prometheus (9090), Grafana (3000), and MinIO console (9000/9001) bind to the host; APIs are internal and reached via NGINX. + +**Volumes:** + +| Volume / mount | Contents | +|----------------|----------| +| `postgres_data` | MLflow backend + job DB | +| `minio_data` | MLflow artifacts, DB backups | +| `redis_data` | RQ persistence (AOF) | +| `./data` | CAFA raw data, embeddings, HF cache | +| `./outputs` | Splits, checkpoints, service artifacts | +| `./backups/postgres` | Local Postgres dumps | + +## Security architecture + +NGINX is the single public ingress. See [nginx/README.md](../nginx/README.md) for gateway specifics. + +| Control | Implementation | +|---------|----------------| +| TLS | HTTP→HTTPS redirect on port 80 | +| Authentication | Basic auth: admin tier (`.htpasswd-admin`) vs user tier (`.htpasswd-user`) | +| Rate limiting | Admin zone 15 r/s; predict zone 30 r/s | +| Body size limits | Per-route caps (512 MB embedding jobs, 5 MB FASTA, 8 MB predict, etc.) | +| Timeouts | 600 s read/send for long jobs | +| Trace headers | `X-Trace-Id`, auth tier/user forwarded upstream | + +**Secrets:** copy [`.env.example`](../.env.example) to `.env`; generate htpasswd with `make gateway-auth`. Never commit real credentials. + +**Internal metrics:** `/metrics` endpoints are scraped on the Docker network and are not exposed through NGINX. + +## Integration points + +| System | Internal URI | Notes | +|--------|--------------|-------| +| MLflow tracking | `http://mlflow:5000` | Gateway: `https://localhost/mlflow/` | +| MLflow artifacts | `s3://mlflow-artifacts/` via MinIO | Requires S3 env vars in clients | +| Job history DB | `postgresql://…/proseqgo_jobs` | Init: `docker/postgres/init-proseqgo-jobs.sh` | +| Redis / RQ | `redis://redis:6379/0` | Queues: `embedding-jobs`, training queue | +| GO prediction (internal) | `http://go-prediction-api:8000` | Used by embedding-api orchestration | + +## Design decisions + +### Async jobs for embedding and training + +Long-running GPU work is offloaded to RQ workers. The API returns immediately with a job ID; Postgres records durable state for polling and audit. + +### Champion alias for serving + +Production inference always resolves `models:/@champion`. Promotion is gated on holdout `holdout_f1_micro ≥ PROMOTION_THRESHOLD` (default 0.35). This decouples experiment versions from the live model. + +### Gateway in front of all user-facing services + +Centralizes TLS, auth tiers, rate limits, and payload caps. Internal services are not directly exposed on the host. + +### Separate job database on Postgres + +`proseqgo_jobs` coexists with the MLflow backend DB on the same Postgres instance but isolates embedding/training job history from MLflow schema. + +### CPU-portable base compose + +The default stack runs on CPU-only hosts (`CAFA_DEVICE=auto` → CPU). GPU is an opt-in overlay for local development. + +## Non-functional requirements + +| Concern | Approach | +|---------|----------| +| Latency | Sync sequence→GO default timeout 1800 s; embedding jobs async | +| Concurrency | RQ workers; GO inference sequential per orchestration call | +| Reproducibility | Fixed seeds, deterministic splits, config-driven pipelines | +| Observability | Prometheus metrics (`cafa5_*`), Grafana dashboards, alert rules | +| Failure recovery | Embedding worker requeues orphaned RQ jobs on startup | + +## Related documentation + +- [data.md](data.md) — dataset layout and versioning +- [training.md](training.md) — ML lifecycle and promotion +- [deployment.md](deployment.md) — environment setup and compose profiles +- [monitoring.md](monitoring.md) — metrics, dashboards, alerts +- Service READMEs: `services/embedding-api/`, `services/training-api/`, `services/streamlit-ui/` From 0f9623d406189a29a860a8a7b514237dd4231e3b Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 15:37:53 +0200 Subject: [PATCH 2/9] docs: Kaggle source, checksums, directory layout, versioning, ingestion, preprocess/split/embed pipeline, sequence normalization, data contracts, retention, reproducibility checklist --- docs/data.md | 220 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 docs/data.md diff --git a/docs/data.md b/docs/data.md new file mode 100644 index 0000000..d3e6b64 --- /dev/null +++ b/docs/data.md @@ -0,0 +1,220 @@ +# Data + +This document defines data sources, layout, versioning, preprocessing, and reproducibility requirements for ProSeqGO. + +## Data sources + +### Primary training dataset (Kaggle) + +| Field | Value | +|-------|-------| +| Dataset | [cafa-5-6-train-dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) | +| Owner | `behrouzmirabdi` | +| Access | Kaggle API (`~/.kaggle/kaggle.json`) or browser download | +| License | See Kaggle dataset page | + +**Required files** (paths relative to repo root, matching `configs/config.yaml`): + +```text +data/cafa-5-cafa-6-protein-function-prediction/ +└── Train/ + ├── train_sequences.fasta + └── train_terms.tsv +``` + +### Integrity checksums + +Verify files after download: + +```bash +sha256sum data/cafa-5-cafa-6-protein-function-prediction/Train/train_sequences.fasta \ + data/cafa-5-cafa-6-protein-function-prediction/Train/train_terms.tsv +``` + +| File | Expected sha256 | +|------|-----------------| +| `train_sequences.fasta` | `434addef94c14eb8fb263ad2f5801a73a43fcb69d10955e5463d20c6b8aaac82` | +| `train_terms.tsv` | `c9489b802b8955d3cb14c23cc465674de86e08ad23107296260c8a8040361535` | + +### External model weights (Hugging Face) + +Embedding backends download pretrained weights on first use: + +| Backend key | HF model | +|-------------|----------| +| `esm2` | `facebook/esm2_t33_650M_UR50D` | +| `protbert` | `Rostlab/prot_bert` | +| `t5` | `Rostlab/prot_t5_xl_uniref50` | + +Cache directory: `data/hf_cache/` (mounted in embedding containers). + +## Data layout + +```text +data/ +├── cafa-5-cafa-6-protein-function-prediction/ # Raw Kaggle data (gitignored) +│ └── Train/ +│ ├── train_sequences.fasta +│ └── train_terms.tsv +├── embeddings/ # Generated .npy embeddings +│ └── hf__/ # e.g. hf_esm2_mean/ +└── hf_cache/ # Hugging Face model cache + +outputs/ +├── splits/ # train/holdout ID arrays +│ ├── train_ids.npy +│ └── holdout_ids.npy +├── labels/ # Binary label matrix artifacts +├── checkpoints/ # Training checkpoints +├── service_artifacts/ # Embedding API job outputs +└── training_api/ # Training API job outputs +``` + +### Raw vs processed vs derived + +| Stage | Location | Regenerable | Git-tracked | +|-------|----------|-------------|-------------| +| Raw FASTA + terms | `data/.../Train/` | Re-download from Kaggle | No | +| Label matrix | `outputs/labels/` | `scripts/preprocess.py` | No | +| Splits | `outputs/splits/` | `scripts/split_train_holdout.py` | No | +| Embeddings | `data/embeddings/` | `scripts/embed_sequences.py` | No | +| HF cache | `data/hf_cache/` | Auto on first embed | No | +| Checkpoints / MLflow artifacts | `outputs/`, MinIO | Training pipeline | No | + +Serving (`make up`) does **not** require training data. Preprocess, embed, train, and holdout evaluation do. + +## Versioning strategy + +1. **Dataset version:** Pin the Kaggle dataset version or record the download date and checksums in MLflow run tags (training script logs file hashes when available). +2. **Config version:** All pipeline scripts accept `--config configs/config.yaml`; treat config changes as data/model contract changes. +3. **Embedding backend alignment:** `data.embeddings_source`, `embedding.backend`, and served model input dimension must match. Mismatch causes inference validation failures. + +Record in every training run: + +- `train_sequences.fasta` sha256 +- `train_terms.tsv` sha256 +- `embedding.backend` and `embedding.pooling` +- split seed / holdout fraction from config + +## Ingestion workflow + +### Download via Kaggle CLI + +```bash +mkdir -p data/cafa-5-cafa-6-protein-function-prediction/Train +kaggle datasets download -d behrouzmirabdi/cafa-5-6-train-dataset \ + -p data/cafa-5-cafa-6-protein-function-prediction/Train --unzip +``` + +### Validation checks + +After download: + +1. Confirm both files exist under `Train/`. +2. Run sha256 verification (table above). +3. Spot-check FASTA record count and terms file column structure. + +## Preprocessing pipeline + +### 1. Label matrix + +```bash +python scripts/preprocess.py --config configs/config.yaml +``` + +Builds a binary multi-label matrix from `train_terms.tsv`. Output paths are defined in `src/preprocess/preprocessing.py` and written under `outputs/`. + +Key config (`configs/config.yaml`): + +- `data.num_labels`: top-N GO terms (default 500) +- `data.train_val_split`: train/validation fraction within labeled set + +### 2. Train/holdout split + +```bash +python scripts/split_train_holdout.py --config configs/config.yaml +``` + +Produces deterministic `train_ids.npy` and `holdout_ids.npy` in `outputs/splits/` using `data.holdout_fraction` (default 0.1) and `training.seed` (default 42). + +### 3. Embedding generation + +```bash +python scripts/embed_sequences.py --config configs/config.yaml \ + --ids-npy outputs/splits/train_ids.npy --split train + +python scripts/embed_sequences.py --config configs/config.yaml \ + --ids-npy outputs/splits/holdout_ids.npy --split holdout +``` + +Outputs `.npy` arrays compatible with `ProteinSequenceDataset` under `data/embeddings/`. + +## Sequence normalization + +Applied at embedding time (`normalize_sequence` in `scripts/embed_sequences.py`): + +- Uppercase, whitespace stripped +- Canonical amino acids retained +- `X`, `U`, `O`, `B`, `Z`, `J`, and unknown symbols → `X` + +API endpoints (`/api/v1/predict-go-from-sequences`, FASTA upload) use the same normalization. + +## Data contracts + +### Training / inference inputs + +| Field | Requirement | +|-------|-------------| +| Protein ID | Non-empty string; matches FASTA header or JSON `id` | +| Sequence | Amino acid string; normalized as above | +| Embedding vector | Length must match model input dim for GO predictor | +| GO labels | `GO:#######` format in terms file | + +### Config-driven paths + +All paths are relative to repo root and defined in `configs/config.yaml`: + +```yaml +data: + data_dir: "data/cafa-5-cafa-6-protein-function-prediction" + train_fasta: "data/cafa-5-cafa-6-protein-function-prediction/Train/train_sequences.fasta" + embeddings_dir: "data/embeddings" + splits_dir: "outputs/splits" +``` + +Do not hardcode machine-specific absolute paths in scripts or configs committed to git. + +## Storage and retention + +| Environment | Raw data | Embeddings | Artifacts | +|-------------|----------|------------|-----------| +| Local dev | `./data/` bind mount | `./data/embeddings/` | `./outputs/` | +| Compose services | `./data`, `./outputs` volumes | `./data/hf_cache` | `./outputs/service_artifacts/` | +| MLflow (prod-like) | N/A | N/A | MinIO `mlflow-artifacts` bucket | + +**Regenerable without data loss:** embeddings, splits, label matrix, local checkpoints. + +**Must preserve for audit:** MLflow runs in Postgres, model versions in registry, promoted champion metadata. + +## Reproducibility checklist + +To rerun training on the same data: + +- [ ] Download dataset and verify sha256 +- [ ] Use unchanged `configs/config.yaml` (or document diffs) +- [ ] Set `training.seed: 42` +- [ ] Run preprocess → split → embed (same backend/pooling) → train +- [ ] Point `MLFLOW_TRACKING_URI` at the same tracking server +- [ ] Log dataset checksums from `train_run_summary.json` + +## Privacy and compliance + +- Training data is public competition data; confirm Kaggle license before redistribution. +- Do not commit raw data, credentials, or user-submitted sequences from production inference to git. +- Service artifacts under `outputs/service_artifacts/` may contain user sequences; treat as sensitive in shared environments. + +## Related documentation + +- [training.md](training.md) — how processed data feeds the training pipeline +- [architecture.md](architecture.md) — data flow through services +- [deployment.md](deployment.md) — volume mounts and data paths in Compose From df5037687441673f997e31a188d7f0d244aa0b54 Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:26:54 +0200 Subject: [PATCH 3/9] docs: ML objective, pipeline stages, config reference, MLflow tracking, registry/champion promotion, retrain options (CLI/API/hybrid), compute requirements, evaluation protocol, known limitations --- docs/training.md | 222 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 docs/training.md diff --git a/docs/training.md b/docs/training.md new file mode 100644 index 0000000..a5a62f7 --- /dev/null +++ b/docs/training.md @@ -0,0 +1,222 @@ +# Training + +This document describes the ML lifecycle in ProSeqGO: from preprocessing through experiment tracking, model registration, evaluation, and champion promotion. + +## Training objective + +**Task:** Multi-label classification — predict GO terms from protein sequence embeddings. + +**Primary promotion metric:** `holdout_f1_micro` on the holdout split (evaluated by `scripts/evaluate_holdout.py`). + +**Default promotion threshold:** `0.35` (`PROMOTION_THRESHOLD` in `.env`). + +**Registered model name:** `cafa-go-model` (`REGISTERED_MODEL_NAME`). + +Serving loads `models:/cafa-go-model@champion` unless `MODEL_URI` is overridden. + +## Pipeline stages + +```text +preprocess → split → embed → train → evaluate_holdout → promote_model +``` + +| Stage | Script | Output | +|-------|--------|--------| +| Labels | `scripts/preprocess.py` | Binary label matrix under `outputs/` | +| Split | `scripts/split_train_holdout.py` | `outputs/splits/{train,holdout}_ids.npy` | +| Embeddings | `scripts/embed_sequences.py` | `data/embeddings/hf_*/*.npy` | +| Train | `scripts/train.py` | Checkpoint, MLflow run, registered model version | +| Evaluate | `scripts/evaluate_holdout.py` | Holdout metrics, eval MLflow run | +| Promote | `scripts/promote_model.py` | `champion` alias if metric ≥ threshold | + +### One-shot retrain pipeline + +```bash +python scripts/retrain_pipeline.py --config configs/config.yaml \ + --promotion-threshold 0.35 \ + --model-name cafa-go-model +``` + +Runs train → evaluate → promote in sequence. Requires `MLFLOW_TRACKING_URI` and S3/MinIO env vars when using the Compose MLflow stack. + +## Configuration + +Global config: [`configs/config.yaml`](../configs/config.yaml). + +### Data + +```yaml +data: + num_labels: 500 + holdout_fraction: 0.1 + embeddings_source: "ESM2" +``` + +### Embedding + +```yaml +embedding: + backend: "esm2" # esm2 | protbert | t5 + pooling: "mean" # mean | cls + max_length: 1280 + batch_size: 8 + fp16: true +``` + +**Critical:** `embedding.backend` must match the embeddings used during training and the dimension expected by the GO predictor at inference time. + +### Model + +```yaml +model: + type: "cnn1d" # mlp | cnn1d + cnn_out_channels: [3, 8] + cnn_kernel_size: 3 +``` + +### Training + +```yaml +training: + epochs: 60 + batch_size: 256 + learning_rate: 0.001 + scheduler_factor: 0.1 + scheduler_patience: 3 + seed: 42 +``` + +### Environment variables + +| Variable | Purpose | +|----------|---------| +| `MLFLOW_TRACKING_URI` | Tracking server (default `file:./mlruns` for local CLI) | +| `MLFLOW_S3_ENDPOINT_URL` | MinIO endpoint for artifact I/O | +| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | S3 credentials for MLflow artifacts | +| `REGISTERED_MODEL_NAME` | Registry name (default `cafa-go-model`) | +| `PROMOTION_THRESHOLD` | Minimum `holdout_f1_micro` for promotion | +| `CAFA_DEVICE` | `auto`, `cpu`, or `cuda` for PyTorch | +| `TRAIN_RUN_ID` | Set by retrain pipeline for evaluate step | + +## Experiment tracking + +Training uses MLflow experiment **`cafa-train`**. + +Logged per run: + +- Hyperparameters from config +- Training metrics (loss, validation metrics) +- Model artifact (PyTorch) +- Dataset file checksums when available +- `train_run_summary.json` in `outputs/` with `train_run_id` + +Evaluation uses a separate MLflow run linked via `train_run_id` tag; metrics include `holdout_f1_micro`. + +**Tracking URI in Compose:** `http://mlflow:5000` (internal). External UI: `https://localhost/mlflow/` via gateway. + +## Model registry workflow + +1. `scripts/train.py` registers a new model version under `REGISTERED_MODEL_NAME`. +2. `scripts/evaluate_holdout.py` logs holdout metrics on a dedicated eval run. +3. `scripts/promote_model.py`: + - Reads `holdout_f1_micro` from the eval run + - Resolves model version from `train_run_id` + - Tags version with promotion metadata + - Sets `champion` alias if `metric ≥ threshold` + +```bash +python scripts/promote_model.py \ + --eval-run-id \ + --train-run-id \ + --model-name cafa-go-model \ + --threshold 0.35 +``` + +Tags written on the model version: `promotion_metric`, `promotion_value`, `promotion_threshold`, `train_run_id`, `eval_run_id`. + +### Rollback + +To revert serving to a previous version: + +```python +from mlflow.tracking import MlflowClient +client = MlflowClient("http://mlflow:5000") +client.set_registered_model_alias("cafa-go-model", "champion", "") +``` + +Restart or reload `go-prediction-api` if it caches the model in memory. + +## Retraining options + +### Option A: CLI (research iteration) + +```bash +python scripts/retrain_pipeline.py --config configs/config.yaml +``` + +Best for local experimentation with full control over each stage. + +### Option B: Training API (ops automation) + +```bash +make training-up +curl -sk -u ADMIN:ADMIN_PASS -X POST https://localhost/api/train/train \ + -H "Content-Type: application/json" \ + -d '{"config":"configs/config.yaml","mode":"retrain"}' +``` + +Poll `GET /api/train/jobs/{job_id}` for status and MLflow links. + +### Option C: Hybrid + +- Generate embeddings via Embedding API (`/api/v1/jobs`) for ad-hoc or online data +- Train/evaluate via CLI for flexibility +- Promote via `scripts/promote_model.py` or retrain pipeline + +## Compute requirements + +| Workload | CPU | GPU recommended | +|----------|-----|-----------------| +| Preprocess / split | Yes | No | +| Embedding generation | Yes (slow) | Yes | +| Training | Yes (slow) | Yes | +| Holdout evaluation | Yes | Optional | +| Serving (inference) | Yes | Yes for throughput | + +Compose defaults: `CAFA_DEVICE=auto` (GPU if available via `docker-compose.gpu.yml`). + +**Training job timeout:** `TRAINING_JOB_TIMEOUT_SEC` (default 86400 s). + +## Reproducibility + +- Set `training.seed: 42` in config; `set_seed()` called in training script +- Use fixed holdout split from `split_train_holdout.py` +- Pin embedding backend and record checksums of input FASTA/terms in MLflow +- Pin dependencies via `requirements.txt` / Docker images (`ghcr.io/behroooz/proseqgo-*`) + +### Rerun a past experiment + +1. Find `train_run_id` in MLflow UI or `outputs/train_run_summary.json` +2. Restore config used for that run (MLflow params or git commit) +3. Re-run with same data checksums and seed + +## Evaluation protocol + +- **Holdout split:** 10% of labeled proteins (`holdout_fraction: 0.1`), deterministic +- **No test-set tuning:** holdout is for final gate only; use train/val split inside training for early stopping +- **Promotion gate:** `holdout_f1_micro ≥ PROMOTION_THRESHOLD` +- **Metric name override:** `--metric-name` on `promote_model.py` (default `holdout_f1_micro`) + +## Known limitations + +- **Class imbalance:** top-500 GO terms still span wide frequency; threshold sensitivity affects rare terms +- **Embedding alignment:** switching backend without retraining breaks inference validation +- **Label coverage:** model only predicts terms present in the training label matrix (`num_labels`) +- **Multi-label bias:** high-frequency GO terms may dominate micro-F1 + +## Related documentation + +- [data.md](data.md) — dataset and preprocessing +- [architecture.md](architecture.md) — training API and worker architecture +- [deployment.md](deployment.md) — training profile and volumes +- [monitoring.md](monitoring.md) — training queue metrics when profile is active From 82ca05bed14d1003cf10ac97d129eefc98397c7c Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:34:47 +0200 Subject: [PATCH 4/9] docs:Environments, prerequisites, first-time setup, .env groups, NGINX routing, deployment patterns, CI/CD/GHCR, release/rollback, scaling, backups, Make targets --- docs/deployment.md | 256 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 docs/deployment.md diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..ea04743 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,256 @@ +# Deployment + +This document explains how to run and ship ProSeqGO across local development, CI, and production-like environments. + +## Supported environments + +| Environment | Compose files | Typical use | +|-------------|---------------|-------------| +| Local dev (CPU) | `docker-compose.yml` | PCs without NVIDIA GPU | +| Local dev (GPU) | `docker-compose.yml` + `docker-compose.gpu.yml` | `make up` auto-detects NVIDIA | +| CI / CPU smoke | `docker-compose.yml` + `docker-compose.ci.yml` | `make ci-up` | +| Full stack | base + GPU + `training` + `monitoring` profiles | Integration testing, demos | + +Differences: + +- **CI overlay:** forces `CAFA_DEVICE=cpu`, CPU PyTorch wheels, skips `postgres-backup` / `backup-offload` +- **GPU overlay:** adds `gpus: all` to embedding/training workers +- **Training profile:** starts `trainer-api` and `trainer-worker` +- **Monitoring profile:** starts Prometheus, Grafana, `redis-exporter` + +## Prerequisites + +| Requirement | Notes | +|-------------|-------| +| Docker + Docker Compose v2 | Required for all deployment modes | +| NVIDIA Container Toolkit | Optional; for GPU overlay | +| `make`, `bash`, `curl` | Convenience targets and smoke tests | +| Kaggle credentials | Only for downloading training data (not for serving) | +| Python 3.10+ | For CLI training outside containers | + +## First-time setup + +### 1. Environment and secrets + +```bash +make ci-env # copies .env.example → .env if missing +# Edit .env: Postgres, MinIO, GATEWAY_* passwords +make gateway-auth # writes nginx/.htpasswd-admin and .htpasswd-user +``` + +Never commit `.env` or htpasswd files with real credentials. + +### 2. Start core stack + +```bash +make up +``` + +Starts: `nginx`, `embedding-api`, `embedding-worker`, `go-prediction-api`, `streamlit-ui`, `mlflow`, `postgres`, `redis`, `minio`, backup sidecars (non-CI). + +Equivalent manual command: + +```bash +docker compose up -d --build +# GPU host: +docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d --build +``` + +### 3. Optional profiles + +```bash +make monitoring-up # Prometheus + Grafana +make training-up # Training API + worker +make all-up # default + monitoring + training +``` + +### 4. Verify access + +| Endpoint | URL | +|----------|-----| +| Gateway | `http://localhost` | +| Streamlit UI | `http://localhost/ui/` | +| MLflow | `http://localhost/mlflow/` | +| Prometheus | `http://localhost:9090` | +| Grafana | `http://localhost:3000` | + +Health checks: + +```bash +curl -sk -u admin:PASSWORD http://localhost/api/v1/health +curl -sk -u user:PASSWORD http://localhost/api/predict/health +``` + +## Configuration by environment + +### `.env` variables (required) + +See [`.env.example`](../.env.example) for the full list. Key groups: + +| Group | Variables | +|-------|-----------| +| Gateway auth | `GATEWAY_ADMIN_USER`, `GATEWAY_ADMIN_PASSWORD`, `GATEWAY_USER`, `GATEWAY_USER_PASSWORD` | +| Postgres | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | +| MinIO / S3 | `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | +| MLflow | `MLFLOW_S3_ENDPOINT_URL`, `MLFLOW_ARTIFACT_ROOT` | +| Model registry | `REGISTERED_MODEL_NAME`, `PROMOTION_THRESHOLD` | +| Job queues | `REDIS_URL`, `JOBS_DATABASE_URL`, `EMBEDDING_JOB_TIMEOUT_SEC`, `TRAINING_JOB_TIMEOUT_SEC` | + +### Service-specific overrides + +Set in `docker-compose.yml` or overlays; do not duplicate secrets in multiple files. + +| Service | Notable env | +|---------|-------------| +| `go-prediction-api` | `MODEL_URI=models:/${REGISTERED_MODEL_NAME}@champion` | +| `embedding-api` | `GO_PREDICTION_API_URL=http://go-prediction-api:8000` | +| `streamlit-ui` | `GATEWAY_BASE_URL=http://nginx` | + +## Networking and routing + +NGINX is the only public ingress (port 80). Internal services communicate on the `proseqgo` network. + +| Gateway path | Upstream | +|--------------|----------| +| `/ui/` | `streamlit-ui` | +| `/api/v1/*` | `embedding-api:8000` | +| `/api/predict/*` | `go-prediction-api:8000` | +| `/api/train/*` | `trainer-api:8000` (training profile) | +| `/mlflow/` | `mlflow:5000` | + +Auth tiers: + +- **Admin:** `/api/v1/*`, `/api/train*`, `/mlflow/` +- **User:** `/api/predict/*`, sync predict-go endpoints + +See [nginx/README.md](../nginx/README.md) for rate limits and body size caps. + +## Deployment patterns + +### 1. Monolith-like local stack (default) + +Single host, local volumes, all services in Compose. Best for development and reproducible demos. + +### 2. API-first production inference + +Run: `nginx`, `embedding-api`, `embedding-worker`, `go-prediction-api`, `mlflow`, data stores. Omit `training` profile in production inference clusters; retrain on separate compute. + +### 3. Training separated from serving + +- Training pipeline on GPU node or batch scheduler +- Push model versions to shared MLflow registry +- Serving stack consumes only `@champion` alias + +### 4. Monitoring-hardened + +Enable `monitoring` profile by default; add Alertmanager and external notifications for production. + +## CI/CD flow + +Documented in [`.github/CI.md`](../.github/CI.md). + +| Trigger | Actions | +|---------|---------| +| PR | Lint, unit tests, image builds (no push) | +| Push to `main` | Lint, unit tests, build + push to GHCR | + +**Registry:** `ghcr.io/behroooz/proseqgo-{embedding-api,go-prediction-api,streamlit-ui,trainer-api,mlflow}` + +**Tags:** `main`, `sha-` + +```bash +make pull-images # pull :main +GHCR_TAG=sha- make pull-images +``` + +Local build: + +```bash +make build-images # CUDA index by default (cu132) +TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu make build-images # CPU wheels +``` + +### CI smoke (Compose) + +```bash +make ci-up +make smoke +make ci-down +``` + +Smoke runs `tests/smoke/smoke_embedding_api.sh`. Does not start training profile or GPU jobs. + +## Release procedure + +1. Merge to `main` → CI publishes images to GHCR +2. Tag release in git if using versioned deploys +3. Pull images on target host: `GHCR_TAG=sha- make pull-images` +4. Update `.env` if config/secrets changed +5. `docker compose pull && docker compose up -d` +6. Run smoke/health checks +7. Confirm Grafana dashboards and MLflow registry + +## Rollback procedure + +### Service rollback + +```bash +GHCR_TAG=sha- make pull-images +docker compose up -d +``` + +### Model rollback + +Set `champion` alias to previous version in MLflow (see [training.md](training.md)), then restart `go-prediction-api`. + +### Database rollback + +Restore from `./backups/postgres/` or MinIO bucket `mlflow-db-backups`. Test restore procedure in staging before production need. + +## Scaling notes + +| Component | Scaling approach | +|-----------|------------------| +| `embedding-worker` | Add worker replicas (same Redis queue) | +| `go-prediction-api` | Horizontal replicas behind NGINX (shared model cache volume or pull from MLflow) | +| `trainer-worker` | Single worker recommended per GPU; scale via dedicated training nodes | +| Postgres / MinIO / Redis | Use managed services or clustered setups for production | + +Current Compose file targets single-host deployment; multi-host requires external orchestration (Kubernetes, etc.). + +## Backup and restore + +| Asset | Mechanism | Location | +|-------|-----------|----------| +| Postgres (MLflow + jobs) | `postgres-backup` sidecar | `./backups/postgres/` | +| Offsite DB dumps | `backup-offload` hourly | MinIO `mlflow-db-backups` | +| MLflow artifacts | MinIO volume | `minio_data` volume | +| Local outputs | Bind mount | `./outputs/` | + +**Existing Postgres volume without `proseqgo_jobs` DB:** + +```bash +docker compose exec postgres psql -U "$POSTGRES_USER" -c 'CREATE DATABASE proseqgo_jobs;' +``` + +Or recreate volume so `docker/postgres/init-proseqgo-jobs.sh` runs on first init. + +## Useful Make targets + +```bash +make up / make down +make ci-up / make ci-down +make training-up / make training-down +make monitoring-up / make monitoring-down +make lint / make test +make build-images / make pull-images +make smoke +make gateway-auth +``` + +## Related documentation + +- [architecture.md](architecture.md) — service topology +- [monitoring.md](monitoring.md) — observability setup +- [troubleshooting.md](troubleshooting.md) — common deployment failures +- [data.md](data.md) — training data setup (not required for serving-only deploy) From b3f8d02cfc6d96bd3abf4923dcce749623126227 Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:44:55 +0200 Subject: [PATCH 5/9] Prometheus/Grafana stack, scrape targets, metrics catalog, dashboards, alert rules (using actual ProSeqGO* alert names from alerts.yml), SLO guidance, per-alert runbooks, PromQL examples --- docs/monitoring.md | 226 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 docs/monitoring.md diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000..52dee18 --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,226 @@ +# Monitoring + +This document defines observability for ProSeqGO: what is monitored, where to look, and how alerts map to operational response. + +For dashboard export workflows and detailed PromQL examples, see also [monitoring/README.md](../monitoring/README.md). + +## Observability stack + +| Component | Role | Access | +|-----------|------|--------| +| Prometheus | Metrics collection, alert evaluation | `http://localhost:9090` | +| Grafana | Dashboards, visualization | `http://localhost:3000` (default `admin`/`admin`) | +| redis-exporter | Redis health and queue signals | Scraped by Prometheus (monitoring profile) | + +Monitoring is **profile-based** and isolated from the public NGINX ingress. Start with: + +```bash +make monitoring-up +``` + +## Service health model + +Health is determined by Prometheus scrape targets (`up` metric), not JSON `/health` endpoints. + +| Job name | Target | Port | +|----------|--------|------| +| `prometheus` | `localhost:9090` | 9090 | +| `embedding_api_metrics` | `embedding-api` | 8000 | +| `embedding_worker_metrics` | `embedding-worker` | 8001 | +| `go_prediction_api_metrics` | `go-prediction-api` | 8000 | +| `trainer_api_metrics` | `trainer-api` | 8000 (training profile) | +| `trainer_worker_metrics` | `trainer-worker` | 8001 (training profile) | +| `redis_exporter` | `redis-exporter` | 9121 | + +**Healthy:** `up == 1` for all expected jobs given active profiles. + +**Note:** `trainer_*` targets are only expected when the training profile is running. Absence is normal otherwise. + +## Metrics catalog + +### HTTP metrics (all APIs) + +| Metric family | Labels | Purpose | +|---------------|--------|---------| +| `http_requests_total` | `service`, `method`, `route`, `status_code` | Request volume and errors | +| `http_request_duration_seconds` | `service`, `method`, `route` | Latency histogram | +| `http_requests_in_flight` | `service` | Concurrency | + +Route labels are normalized to static templates (no raw UUIDs) to avoid cardinality explosion. + +### Embedding pipeline + +| Metric | Purpose | +|--------|---------| +| `embedding_queue_jobs` | Postgres queue depth by status | +| `rq_queue_length{queue="embedding-jobs"}` | Redis RQ queue length | +| Embedding job duration / outcome counters | Pipeline throughput and failures | + +### Training pipeline (training profile) + +| Metric | Purpose | +|--------|---------| +| Training queue depth | Backlog | +| Training job duration by mode | Retrain latency | +| Failure reason counters | Debug failed jobs | + +### Inference + +| Metric | Purpose | +|--------|---------| +| `inference_duration_seconds` | Latency by `model_version` | +| Validation failure counters | Embedding dimension / schema drift | +| `top_k` distribution | Usage patterns | + +### Redis + +| Metric | Purpose | +|--------|---------| +| `redis_up` | Redis availability | + +## Dashboards + +Provisioned from `monitoring/grafana/dashboards/`: + +| Dashboard | File | Focus | +|-----------|------|-------| +| CAFA5 Service Health | `service-health.json` | Target up/down, request rate, 5xx ratio, p95 latency, in-flight | +| CAFA5 Domain Pipelines | `domain-pipelines.json` | Embedding/training queues, inference by model version, validation failures | + +Grafana datasource UID: `prometheus` → `http://prometheus:9090`. + +**Start here during incidents:** Service Health dashboard, then Domain Pipelines if embedding or inference is involved. + +## Alerts + +Rules: [`monitoring/alerts.yml`](../monitoring/alerts.yml) + +| Alert | Condition | Severity | Meaning | +|-------|-----------|----------|---------| +| `ProSeqGOServiceMetricsTargetDown` | `up == 0` for >2m | critical | Scrape target unreachable | +| `ProSeqGOHighHttp5xxRatio` | 5xx ratio >5% for 10m with traffic floor | warning | User-visible API errors | +| `ProSeqGOEmbeddingQueueBacklogHigh` | queued jobs >20 for 10m | warning | Embedding pipeline saturated | +| `ProSeqGOEmbeddingWorkerDownWithBacklog` | worker down + RQ queue non-empty for 5m | critical | Jobs stuck with no worker | +| `ProSeqGORedisDown` | `redis_up == 0` for 2m | critical | Job dispatch broken | + +Verify rules and firing alerts: + +```bash +curl -s http://localhost:9090/-/ready +curl -s http://localhost:9090/api/v1/rules +curl -s http://localhost:9090/api/v1/alerts +``` + +**Production gap:** Alertmanager and external notifications (PagerDuty, Slack) are not wired in the default stack. Add Alertmanager for production paging. + +## SLO guidance (recommended) + +Define explicitly for your deployment; suggested starting points: + +| SLI | Target | +|-----|--------| +| API availability | `up{job=~"embedding_api_metrics|go_prediction_api_metrics"} == 1` for 99.5% / 30d | +| Prediction p95 latency | < 5s for single embedding inference (embedding precomputed) | +| 5xx ratio | < 1% over 1h under normal load | +| Embedding queue | < 20 queued jobs 95% of time | + +## Model observability + +When rolling a new model version: + +1. Confirm `model_version` label appears in inference metrics +2. Compare p95 latency by `model_version` +3. Watch validation failure reasons (embedding dimension mismatch is the most common) +4. Check 5xx ratio on `go-prediction-api` +5. Verify champion alias in MLflow matches expected version + +## Runbook: first actions per alert + +### `ProSeqGOServiceMetricsTargetDown` + +1. `docker compose ps` — is the container running? +2. `docker compose logs --tail=200 ` +3. Confirm `/metrics` responds inside the Docker network (not via NGINX) +4. Restart affected service if crash-looping + +### `ProSeqGOHighHttp5xxRatio` + +1. Identify service from alert label +2. Check recent deploys or model promotions +3. Inspect logs for stack traces +4. Query 5xx by route in Prometheus/Grafana + +### `ProSeqGOEmbeddingQueueBacklogHigh` + +1. Check `embedding-worker` logs and GPU/CPU utilization +2. Scale workers or reduce inbound job rate +3. Inspect failed jobs in Postgres `proseqgo_jobs` + +### `ProSeqGOEmbeddingWorkerDownWithBacklog` + +1. Restart `embedding-worker` +2. Confirm crash recovery requeues orphaned jobs (see smoke test) +3. Investigate OOM or GPU errors in worker logs + +### `ProSeqGORedisDown` + +1. `docker compose ps redis` +2. `docker compose logs redis` +3. Restart Redis; verify RQ workers reconnect + +## Reload and maintenance + +**Prometheus config/rules change:** + +```bash +curl -X POST http://localhost:9090/-/reload +# or +docker compose restart prometheus +``` + +**Grafana dashboard JSON change:** auto-refresh via provisioning; restart Grafana if panels do not update. + +**Retention:** Prometheus TSDB retention 15 days (`--storage.tsdb.retention.time=15d` in compose). + +## Useful PromQL queries + +Service availability: + +```promql +up{job=~"prometheus|embedding_api_metrics|go_prediction_api_metrics|trainer_api_metrics"} +``` + +HTTP 5xx ratio by service: + +```promql +sum by (service) (rate(http_requests_total{status_code=~"5.."}[5m])) +/ +clamp_min(sum by (service) (rate(http_requests_total[5m])), 0.001) +``` + +Embedding queue depth: + +```promql +cafa5_embedding_queue_jobs{status="queued"} +``` + +Inference p95 by model version: + +```promql +histogram_quantile( + 0.95, + sum by (le, model_version) (rate(inference_duration_seconds_bucket[5m])) +) +``` + +## Stop monitoring + +```bash +make monitoring-down +``` + +## Related documentation + +- [troubleshooting.md](troubleshooting.md) — extended diagnostic steps +- [architecture.md](architecture.md) — which services expose metrics +- [deployment.md](deployment.md) — starting the monitoring profile From 294663eb7a64295b0e1c3770c4a773ab162a7bf5 Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:47:05 +0200 Subject: [PATCH 6/9] =?UTF-8?q?Symptom=E2=86=92fix=20by=20layer:=20startup?= =?UTF-8?q?,=20gateway/auth,=20data,=20training,=20inference,=20monitoring?= =?UTF-8?q?,=20worker=20crash=20recovery,=20escalation=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/troubleshooting.md | 314 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 docs/troubleshooting.md diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..0d159b5 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,314 @@ +# Troubleshooting + +Symptom → cause → fix guide for ProSeqGO. Organized by layer. For alert-specific first actions, see [monitoring.md](monitoring.md). + +## Quick diagnostic commands + +```bash +docker compose ps +docker compose logs --tail=100 +curl -s http://localhost:9090/-/ready +curl -sk -u admin:PASS https://localhost/api/v1/health +curl -sk -u user:PASS https://localhost/api/predict/health +``` + +Replace `` with: `nginx`, `embedding-api`, `embedding-worker`, `go-prediction-api`, `mlflow`, `postgres`, `redis`, `minio`, `trainer-api`. + +--- + +## Startup failures + +### Compose services won't start + +**Symptoms:** `docker compose up` exits or containers restart loop. + +**Checks:** + +1. `.env` exists (`make ci-env`) +2. Required env vars set (Postgres, MinIO passwords) +3. Port conflicts on 80, 9090, 3000 +4. `make gateway-auth` ran (htpasswd files exist) + +**Fix:** + +```bash +make ci-env && make gateway-auth +docker compose logs postgres minio mlflow +``` + +### `proseqgo_jobs` database does not exist + +**Symptoms:** Embedding or training API fails with DB connection errors referencing `proseqgo_jobs`. + +**Cause:** Postgres volume created before init script added the jobs database. + +**Fix:** + +```bash +docker compose exec postgres psql -U "$POSTGRES_USER" -c 'CREATE DATABASE proseqgo_jobs;' +``` + +Or tear down with volumes on a dev machine only: `make ci-down` (removes volumes). + +### GPU not detected in containers + +**Symptoms:** Slow inference; logs show CPU device. + +**Checks:** + +```bash +nvidia-smi +docker compose -f docker-compose.yml -f docker-compose.gpu.yml config | grep -A2 gpus +``` + +**Fix:** Install NVIDIA Container Toolkit; use `make up` (auto-adds GPU overlay) or explicit GPU compose files. + +--- + +## Authentication and gateway + +### 401 Unauthorized from NGINX + +**Cause:** Wrong credentials or missing htpasswd files. + +**Fix:** + +```bash +# Ensure .env has GATEWAY_ADMIN_* and GATEWAY_USER_* +make gateway-auth +docker compose restart nginx +``` + +Use admin credentials for `/api/v1/*` and `/mlflow/`; user credentials for `/api/predict/*`. + +### 403 Forbidden + +**Cause:** Correct auth tier but route requires different tier (e.g. user creds on admin route). + +**Fix:** Match credential tier to route map in [deployment.md](deployment.md). + +### 502 Bad Gateway / 504 Gateway Timeout + +**Cause:** Upstream service down, slow, or unreachable from NGINX. + +**Checks:** + +```bash +docker compose ps embedding-api go-prediction-api mlflow +docker compose logs nginx --tail=50 +``` + +**Fix:** Restart failed upstream. For long jobs, confirm 600s NGINX timeouts are sufficient; increase client `timeout_seconds` for sync predict-go calls. + +### 413 Request Entity Too Large + +**Cause:** Payload exceeds per-route `client_max_body_size`. + +| Route | Limit | +|-------|-------| +| `/api/v1/predict-go-from-fasta` | 5 MB | +| `/api/predict/` | 8 MB | +| `/api/train` | 64 MB | +| `/api/v1/` (general) | 512 MB | + +**Fix:** Reduce payload, use async `/api/v1/jobs/fasta` for large inputs, or split requests. + +### 429 Too Many Requests + +**Cause:** NGINX rate limit exceeded. + +**Fix:** Back off and retry; adjust rate limit zones only in controlled environments. + +### TLS / certificate errors + +**Cause:** Self-signed cert on `https://localhost`. + +**Fix:** Use `curl -k` or add cert to trust store for local dev. Production should use real certificates. + +--- + +## Data issues + +### Kaggle authentication fails + +**Symptoms:** `403` or credential errors from `kaggle datasets download`. + +**Fix:** + +```bash +mkdir -p ~/.kaggle +chmod 600 ~/.kaggle/kaggle.json +``` + +Ensure API token is valid on kaggle.com → Account → API. + +### Missing CAFA files + +**Symptoms:** Preprocess or embed scripts fail with `FileNotFoundError`. + +**Fix:** Download dataset per [data.md](data.md) and verify paths match `configs/config.yaml`. + +### Checksum mismatch + +**Symptoms:** sha256 does not match expected values in [data.md](data.md). + +**Fix:** Re-download dataset; do not proceed with training until checksums match or divergence is documented in MLflow. + +### Split / embedding mismatch + +**Symptoms:** Training fails with shape errors or missing embedding files. + +**Fix:** Regenerate embeddings for the same `ids.npy` and backend as configured: + +```bash +python scripts/embed_sequences.py --config configs/config.yaml \ + --ids-npy outputs/splits/train_ids.npy --split train +``` + +--- + +## Training issues + +### MLflow unreachable from CLI + +**Symptoms:** Connection refused to `http://mlflow:5000` from host. + +**Fix:** From host use gateway or published port mapping; inside containers use `http://mlflow:5000`. Set: + +```bash +export MLFLOW_TRACKING_URI=http://127.0.0.1/mlflow # through gateway, with auth +# or direct if port-forwarded +``` + +For artifact upload, set `MLFLOW_S3_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`. + +### Training succeeds but no registered model + +**Checks:** + +1. MLflow logs in `scripts/train.py` output +2. MinIO bucket `mlflow-artifacts` exists (`minio-init` completed) +3. S3 credentials in environment + +### Promotion rejected + +**Symptoms:** `promote_model.py` prints metric below threshold; no `champion` update. + +**Fix:** Expected behavior when `holdout_f1_micro < PROMOTION_THRESHOLD`. Lower threshold only with ML review, or improve model/data. + +### Training API job stuck + +**Checks:** + +```bash +docker compose logs trainer-api trainer-worker +# Metrics: training queue depth in Grafana +``` + +**Fix:** Restart `trainer-worker`; check `TRAINING_JOB_TIMEOUT_SEC`; verify `data/` and `outputs/` mounts. + +--- + +## Inference issues + +### Empty or missing predictions + +**Checks:** + +1. `go-prediction-api` logs +2. Champion model exists in MLflow registry +3. Embedding dimension matches model input + +### Wrong model version served + +**Checks:** + +```bash +# Response includes model_version field +curl -sk -u user:PASS -X POST http://localhost/api/predict/predict \ + -H "Content-Type: application/json" \ + -d '{"embedding": [...], "top_k": 5}' +``` + +**Fix:** Verify `champion` alias in MLflow UI; restart `go-prediction-api` after alias change. + +### Embedding validation failures + +**Symptoms:** 4xx from predict API; validation failure metrics increase. + +**Cause:** Embedding length mismatch or NaN values. + +**Fix:** Align `embedding.backend` with training; regenerate embeddings; check `go-prediction-api` logs for expected dimension. + +### Sync predict-go timeout + +**Symptoms:** 504 or client timeout on `/api/v1/predict-go-from-sequences`. + +**Fix:** + +- Increase `timeout_seconds` in request (max 7200) +- Use async flow: `POST /api/v1/jobs/fasta` → poll → `POST /api/v1/jobs/{id}/predict-go` +- Reduce batch size or sequence count + +### Streamlit UI cannot reach API + +**Checks:** + +1. `streamlit-ui` env: `GATEWAY_BASE_URL=http://nginx` +2. `GATEWAY_USER` / `GATEWAY_USER_PASSWORD` match `make gateway-auth` output + +--- + +## Monitoring gaps + +### Prometheus target down + +See [monitoring.md](monitoring.md) — verify container running and `/metrics` on internal port (8000 API, 8001 workers). + +### Grafana shows no data + +**Checks:** + +1. Time range in Grafana +2. Datasource UID `prometheus` healthy +3. Query works in Prometheus UI directly +4. Training metrics absent if training profile not started (expected) + +### Alerts never fire + +**Checks:** + +1. Evaluate rule expression in Prometheus graph +2. Confirm `for:` duration elapsed +3. Traffic floor on 5xx ratio rule (low traffic suppresses alert) + +--- + +## Embedding worker crash recovery + +**Symptoms:** Jobs stuck in `running` after worker kill. + +**Expected behavior:** Worker startup requeues orphaned RQ jobs and resets Postgres status. + +**Verify:** + +```bash +./tests/smoke/test_embedding_worker_crash_recovery.sh +``` + +--- + +## Escalation path + +| Severity | Action | +|----------|--------| +| Serving down | Check NGINX → upstream health → restart services → rollback model if recent promotion | +| Data corruption | Stop training jobs; restore Postgres/MinIO from backup | +| Security incident | Rotate gateway and DB passwords; review NGINX access logs | + +## Related documentation + +- [deployment.md](deployment.md) — setup and rollback +- [monitoring.md](monitoring.md) — alerts and dashboards +- [training.md](training.md) — promotion and MLflow workflow +- [data.md](data.md) — dataset download and validation From 18dfda0d86b27d47e5d92f7fd8de2bacb0f47f09 Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:51:12 +0200 Subject: [PATCH 7/9] Dev setup, PR workflow, coding standards, testing expectations, change-type checklist, security, dependency policy --- docs/contributing.md | 184 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 docs/contributing.md diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..1ed31e9 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,184 @@ +# Contributing + +Guidelines for changing ProSeqGO safely: setup, workflow, standards, and what to update with each change type. + +## Contribution scope + +Contributions welcome for: + +- Bug fixes in pipelines, APIs, and infrastructure +- Tests for critical logic (validation, config, schemas) +- Documentation improvements +- Monitoring dashboards and alert tuning +- CI and Docker improvements + +**Requires design discussion before implementation:** + +- New public API endpoints or breaking request/response contracts +- Changes to model input/output schema or registry alias strategy +- New external dependencies (especially GPU/torch-related) +- Database schema changes for `proseqgo_jobs` +- Security model changes (auth tiers, TLS, exposed ports) + +## Development setup + +### Minimal setup (code changes only) + +```bash +python -m venv .venv && source .venv/bin/activate +pip install -e ".[dev]" +make lint +make test +``` + +Unit tests do **not** require Docker, GPU, or network access. + +### Full stack (integration work) + +```bash +make ci-env +make gateway-auth +make up +make monitoring-up # optional +make smoke # after stack is healthy +``` + +See [deployment.md](deployment.md) for profiles and secrets. + +## Branch and PR workflow + +1. Branch from `main` with a descriptive name (e.g. `fix/embedding-validation`, `docs/monitoring-runbook`) +2. Keep PRs focused; prefer small reviewable diffs +3. Ensure CI passes: lint, unit tests, image builds on PR +4. Include a **test plan** in the PR description (commands run, screenshots for UI) +5. Request review from a maintainer familiar with the affected area (ML, API, infra) + +**Merge policy:** squash or merge commit per team convention; `main` publishes images to GHCR. + +## Coding standards + +### Python + +- Target Python **3.10+** +- Style: [Ruff](https://docs.astral.sh/ruff/) (`make lint` → `ruff check src services scripts`) +- Line length: 100 (see `pyproject.toml`) +- Type hints where they clarify non-obvious contracts +- Match existing patterns in the module you edit + +### Layout conventions + +| Path | Purpose | +|------|---------| +| `src/` | Core ML, preprocessing, training, inference | +| `services/` | FastAPI/Streamlit service code | +| `scripts/` | CLI entrypoints | +| `configs/` | YAML configuration | +| `tests/unit/` | Fast pytest (CI) | +| `tests/smoke/` | Compose acceptance scripts (manual/CI Phase 3) | + +### Config and logging + +- Use `src/config.py` / `load_config()` for pipeline config +- No hardcoded secrets, paths, or machine-specific settings +- Use `setup_logger()` from `src/utils.py` for CLI scripts + +### Comments + +Prefer self-explanatory code. Comment only non-obvious business logic, protocol quirks, or operational constraints. + +## Testing expectations + +### Unit tests (`make test`) + +Located in `tests/unit/`. Current coverage areas: + +- Sequence normalization +- Config loading +- API schema bounds (Pydantic) +- Embedding vector validation +- UI input validation + +**Required before merge:** `make lint` and `make test` pass. + +### Smoke tests + +```bash +make ci-up && make smoke && make ci-down +``` + +Scripts in `tests/smoke/` — not part of unit test suite. Run when touching embedding API, workers, or compose wiring. + +### What CI does not run + +- Full training or retrain jobs +- GPU workloads +- Training API profile (see [`.github/CI.md`](../.github/CI.md)) + +Do not rely on CI to catch training regressions; document manual validation in PR test plan. + +## Change categories and required updates + +| Change type | Update | +|-------------|--------| +| New/changed API endpoint | Service README, request schemas, unit tests, [troubleshooting.md](troubleshooting.md) if user-facing | +| Config key added | `configs/config.yaml`, `src/config.py`, [training.md](training.md) or [data.md](data.md) | +| Model pipeline change | [training.md](training.md), reproducibility notes in PR | +| Compose / env var | `.env.example`, [deployment.md](deployment.md), `Makefile` if new target | +| New metric or alert | `monitoring/alerts.yml`, dashboard JSON, [monitoring.md](monitoring.md) | +| NGINX route change | `nginx/nginx.conf`, `nginx/README.md`, [architecture.md](architecture.md) | +| Dataset path or checksum | [data.md](data.md) only (do not commit raw data) | + +**Do not modify `README.md` unless the PR explicitly scopes documentation at the top level** — deep docs live under `docs/`. + +## Commit and PR description + +**Commit messages:** concise, imperative mood, explain *why* when not obvious. + +Examples: + +- `fix embedding-api route label cardinality for Prometheus` +- `docs: add deployment rollback procedure` +- `gate champion promotion on holdout_f1_micro threshold` + +**PR description should include:** + +- Summary of change (1–3 bullets) +- Test plan (commands executed) +- Rollout notes if deploy or model promotion is affected +- Screenshots for Streamlit/Grafana changes + +## Release and model promotion + +- **Image releases:** merging to `main` triggers GHCR publish (`sha-` and `main` tags) +- **Model promotion:** only after holdout evaluation passes threshold; document `train_run_id` and `eval_run_id` in change log or MLflow +- **Production config:** gateway passwords, Postgres, and MinIO credentials must be rotated via ops process — never in git + +Restrict who can: + +- Set `champion` alias in production MLflow +- Change production `.env` and NGINX configuration +- Modify alert thresholds that page on-call + +## Security practices + +- Never commit `.env`, `kaggle.json`, htpasswd files with real passwords, or raw datasets +- Use `.env.example` for variable names only +- Run `make gateway-auth` locally; use throwaway passwords in CI +- Report security issues privately to maintainers (do not open public issues for active vulnerabilities) + +## Dependency updates + +- Pin breaking changes (e.g. `mlflow==2.13.0` in `pyproject.toml`) +- Torch: local `make build-images` uses CUDA index; CI uses CPU wheels — test both paths when upgrading torch +- Avoid new dependencies unless necessary; prefer stdlib and existing stack + +## Getting help + +- Architecture questions → [architecture.md](architecture.md) +- Operational issues → [troubleshooting.md](troubleshooting.md) +- CI behavior → [`.github/CI.md`](../.github/CI.md) +- Service-specific behavior → `services/*/README.md` + +## License + +By contributing, you agree that your contributions are licensed under the project MIT license. From dfda10a6457a6922591d91a95aeb6dbfe5c762bc Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 17:54:44 +0200 Subject: [PATCH 8/9] fix: remove docs/ from it --- .gitignore | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 014fb92..8abfd37 100644 --- a/.gitignore +++ b/.gitignore @@ -123,9 +123,8 @@ docker-compose.override.yml Thumbs.db ########################## -# Docs / Notes (local only) +# Notes (local only) ########################## -docs/ notes/ ########################## From 6ba71135252745924b0791883413f43b1012c08c Mon Sep 17 00:00:00 2001 From: Behrouz Mirabdi Date: Tue, 28 Jul 2026 18:01:09 +0200 Subject: [PATCH 9/9] =?UTF-8?q?docs:=20Structure=20now=20Overview=20?= =?UTF-8?q?=E2=86=92=20features=20=E2=86=92=20architecture=20=E2=86=92=20r?= =?UTF-8?q?epo=20map=20=E2=86=92=20data=20=E2=86=92=20quickstart=20?= =?UTF-8?q?=E2=86=92=20config=20=E2=86=92=20training=20=E2=86=92=20serving?= =?UTF-8?q?=20=E2=86=92=20monitoring=20=E2=86=92=20CI=20=E2=86=92=20deploy?= =?UTF-8?q?ment=20=E2=86=92=20docs=20map=20=E2=86=92=20contributing=20?= =?UTF-8?q?=E2=86=92=20troubleshooting=20=E2=86=92=20Make=20targets=20?= =?UTF-8?q?=E2=86=92=20license?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 657 +++++++++++++++++++----------------------------------- 1 file changed, 228 insertions(+), 429 deletions(-) diff --git a/README.md b/README.md index 028d0e7..9338590 100644 --- a/README.md +++ b/README.md @@ -1,373 +1,236 @@ -# ProSeqGO: Protein Sequence Gene Onthology prediction +# ProSeqGO -End-to-end MLOps platform for protein function prediction (sequence -> embedding -> GO terms), with model lifecycle management, secured gateway routing, and production-oriented monitoring. +Production-oriented MLOps platform for **multi-label Gene Ontology (GO) prediction** from protein sequences: reproducible training, registry-based serving, secured gateway routing, async job queues, and observability. -## Problem This Project Solves +```text +sequence → embedding (ESM2 / ProtBERT / T5) → GO term predictions +``` + +| Audience | Entry point | +|----------|-------------| +| Product / lab users | Streamlit UI (`/ui/`) or predict APIs | +| ML engineers | CLI (`scripts/`) and Training API | +| Platform / ops | Docker Compose, NGINX, Prometheus / Grafana | +| Auditors | MLflow tracking and model registry (`/mlflow/`) | -Protein function annotation is a high-throughput, multi-label prediction problem where operational risks are as important as model quality: inconsistent embedding backends, untracked model promotions, and weak runtime observability can silently degrade prediction quality. +## Overview -This project provides: +Protein function annotation is a high-throughput, multi-label problem where operational risk matters as much as model quality: untracked promotions, embedding/model dimension drift, and weak runtime signals can silently degrade predictions. -- A reproducible training and retraining workflow. -- Online inference APIs for both embedding-level and sequence-level use cases. -- MLflow-backed experiment tracking and registry-based model serving. -- A secured NGINX gateway for TLS, auth, and rate/body constraints. -- Monitoring with Prometheus, Grafana dashboards, and actionable alert rules. +ProSeqGO provides: -## High-Level Workflow +1. Preprocess CAFA labels and deterministic train/holdout splits +2. Generate embeddings from protein sequences +3. Train and evaluate a multi-label GO predictor +4. Log runs and register models in MLflow +5. Promote a `champion` alias after metric threshold checks +6. Serve predictions (`embedding → GO` and `sequence → GO`) +7. Observe health, latency, errors, and queue backlog -1. Preprocess CAFA labels and generate deterministic train/holdout splits. -2. Generate embeddings from protein sequences (ESM2/ProtBERT/T5). -3. Train and evaluate multi-label GO predictor. -4. Log runs/artifacts to MLflow and register model versions. -5. Promote model alias (`champion`) after metric threshold checks. -6. Serve predictions: - - `embedding -> GO` via GO prediction API. - - `sequence -> GO` in one call via embedding API orchestration. -7. Observe service health and model-serving behavior with Prometheus/Grafana. +**Data source:** [cafa-5-6-train-dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) on Kaggle (required for training/evaluation; not required for inference-only serving once a champion model exists). -## Architecture Overview +## Key features + +- Reproducible training on a version-pinned Kaggle dataset (checksums documented) +- Config-driven preprocess → embed → train → evaluate → promote pipeline +- MLflow experiment tracking and model registry (`@champion` serving) +- Containerized multi-service stack (Compose base + GPU / CI overlays) +- Async embedding and training jobs (Postgres history + Redis/RQ) +- NGINX gateway with Basic Auth tiers, rate limits, and body-size controls +- Streamlit UI for interactive sequence → GO prediction +- Prometheus / Grafana monitoring with alert rules +- CI: lint, unit tests, image builds; GHCR publish on `main` + +## Architecture ```text User / Client | v -NGINX (TLS + Basic Auth + Rate Limit + Routing) +NGINX (Basic Auth + Rate Limit + Routing) :80 |-----------------------> /ui/ -----------------------> Streamlit UI |-----------------------> /api/v1/* ------------------> Embedding API - | |-> Go Prediction API (/predict) + | |-> Go Prediction API |-----------------------> /api/predict/* -------------> Go Prediction API |-----------------------> /api/train* ----------------> Training API (profile: training) |-----------------------> /mlflow/* ------------------> MLflow UI / Registry -Prometheus <---------------- /metrics from embedding/go/training/prometheus -Grafana <------------------- Prometheus datasource +Prometheus <---------------- /metrics (APIs, workers, redis-exporter) +Grafana <------------------- Prometheus ``` -## Repository Structure +| Layer | Components | +|-------|------------| +| Ingress | `nginx` | +| Inference | `embedding-api`, `embedding-worker`, `go-prediction-api`, `streamlit-ui` | +| ML lifecycle | `mlflow`, `trainer-api` / `trainer-worker` (training profile) | +| Data plane | `postgres` (MLflow + `proseqgo_jobs`), `redis` (RQ), `minio` (artifacts) | +| Observability | `prometheus`, `grafana`, `redis-exporter` (monitoring profile) | + +Serving loads `models:/cafa-go-model@champion` by default (`REGISTERED_MODEL_NAME` / `MODEL_URI`). + +**Details:** [docs/architecture.md](docs/architecture.md) + +## Repository structure ```text -CAFA-5-MLOps-solution/ -├── configs/ # Global YAML config for data/model/train/inference -├── data/ # CAFA data, embeddings, HF cache -├── docker/ # Service-specific Dockerfiles -├── docs/ # Additional project docs -├── examples/ # Example sequences/inputs -├── monitoring/ # Prometheus, Grafana provisioning, alert rules -├── nginx/ # Gateway config, TLS certs, htpasswd files -├── outputs/ # Splits, labels, checkpoints, artifacts, submissions -├── scripts/ # CLI pipeline entrypoints (preprocess/train/evaluate/predict) -├── tests/ -│ ├── unit/ # Fast pytest suite (no Docker/GPU; CI Phase 1B) -│ └── smoke/ # Compose smoke/acceptance checks (not unit tests) -├── .github/ -│ ├── workflows/ci.yml # PR/main CI (lint today; tests/images later) -│ └── CI.md # CI scope, GHCR plan, non-goals +proseqgo/ +├── configs/ # Global YAML (data / model / train / inference) +├── data/ # Kaggle raw data, embeddings, HF cache (gitignored payloads) +├── docker/ # Service Dockerfiles and Postgres init +├── docs/ # Architecture, data, training, deploy, ops +├── examples/ # Sample FASTA / inputs +├── monitoring/ # Prometheus, Grafana, alert rules +├── nginx/ # Gateway config and htpasswd (generated) +├── outputs/ # Splits, labels, checkpoints, service artifacts +├── scripts/ # CLI: preprocess, embed, train, evaluate, promote ├── services/ -│ ├── embedding-api/ # Async embedding jobs + sequence->GO orchestration endpoint -│ ├── go-prediction-api/ # Embedding->GO inference API -│ ├── streamlit-ui/ # Interactive UI over gateway endpoint -│ └── training-api/ # Async train/retrain job API -├── src/ # Core modeling/training/inference modules -├── docker-compose.yml # Portable serving stack (CPU-safe base) -├── docker-compose.gpu.yml # GPU overlay (gpus: all for inference workers) -├── docker-compose.ci.yml # CI overlay (CPU torch, CAFA_DEVICE=cpu, trim backups) -├── Makefile # Convenience targets for compose profiles +│ ├── embedding-api/ # Async embeddings + sequence→GO orchestration +│ ├── go-prediction-api/ # Embedding→GO inference +│ ├── streamlit-ui/ # Product UI +│ └── training-api/ # Async train / retrain jobs +├── src/ # Core modeling / training / inference libraries +├── tests/ +│ ├── unit/ # Fast pytest (CI; no Docker/GPU) +│ └── smoke/ # Compose acceptance scripts +├── .github/ # CI workflows and CI notes +├── docker-compose.yml # Portable base stack +├── docker-compose.gpu.yml # GPU overlay +├── docker-compose.ci.yml # CPU CI / smoke overlay +├── Makefile # up / train / monitor / lint / test / smoke └── README.md ``` -## Quick Start +## Data source and versioning -### 1) Bring core stack up +| Field | Value | +|-------|-------| +| Dataset | [cafa-5-6-train-dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) | +| Expected layout | `data/cafa-5-cafa-6-protein-function-prediction/Train/{train_sequences.fasta,train_terms.tsv}` | +| Access | Kaggle API (`~/.kaggle/kaggle.json`) or browser download | +| Integrity | SHA-256 checksums in [docs/data.md](docs/data.md) | ```bash -make up +mkdir -p data/cafa-5-cafa-6-protein-function-prediction/Train +kaggle datasets download -d behrouzmirabdi/cafa-5-6-train-dataset \ + -p data/cafa-5-cafa-6-protein-function-prediction/Train --unzip ``` -`make up` uses the portable base compose file and **automatically adds** `docker-compose.gpu.yml` when `nvidia-smi` is available. On CPU-only hosts the stack still starts (inference uses `CAFA_DEVICE=auto` → CPU). +Serving (`make up`) does **not** require these files. Training, embedding generation, and holdout evaluation do. -Manual compose (equivalent): +Raw vs processed vs feature data are separated under `data/` and `outputs/`. Do not hardcode machine-specific paths; use `configs/config.yaml` and environment variables. -```bash -# CPU-only / portable -docker compose up --build - -# GPU host (explicit overlay) -docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build -``` +**Details:** [docs/data.md](docs/data.md) -Core services started by default: `nginx`, `embedding-api`, `embedding-worker`, `go-prediction-api`, `streamlit-ui`, `mlflow`. - -**CI / CPU smoke** (Phase 3): - -```bash -make ci-up # base + docker-compose.ci.yml (forces CAFA_DEVICE=cpu, CPU torch wheels) -make smoke -make ci-down -``` +## Quickstart -### 2) Bring monitoring up +### Prerequisites -```bash -docker compose --profile monitoring up -d -``` +- Docker + Docker Compose v2 +- `make`, `bash`, `curl` +- NVIDIA Container Toolkit (optional; GPU overlay) +- Kaggle credentials (only if downloading training data) +- Python 3.10+ (for CLI training outside containers) -or: +### 1. Environment and gateway auth ```bash -make monitoring-up +make ci-env # .env.example → .env if missing +# Edit .env: Postgres, MinIO, GATEWAY_* passwords (admin ≠ user) +make gateway-auth # writes nginx/.htpasswd-admin and .htpasswd-user ``` -### 3) Optional: bring training API profile up +Never commit `.env` or real htpasswd files. -```bash -docker compose --profile training up -d --build -``` - -or: +### 2. Start the core stack ```bash -make training-up +make up ``` -### 4) Training data (preprocess / train) - -Serving (`make up`) does **not** need these files. They are required for CLI preprocess, embedding generation, training, and holdout evaluation. - -Dataset on Kaggle: [cafa-5-6-train-dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) - -**Expected layout** (matches `configs/config.yaml`): +`make up` uses the portable base compose file and **adds** `docker-compose.gpu.yml` when `nvidia-smi` is available. On CPU-only hosts, inference uses `CAFA_DEVICE=auto` → CPU. -```text -data/cafa-5-cafa-6-protein-function-prediction/ -└── Train/ - ├── train_sequences.fasta - └── train_terms.tsv -``` +Default services: `nginx`, `embedding-api`, `embedding-worker`, `go-prediction-api`, `streamlit-ui`, `mlflow`, `postgres`, `redis`, `minio` (plus backup sidecars outside CI). -**Download via Kaggle CLI** (requires [`kaggle`](https://github.com/Kaggle/kaggle-cli) and `~/.kaggle/kaggle.json`): +### 3. Optional profiles ```bash -mkdir -p data/cafa-5-cafa-6-protein-function-prediction/Train -kaggle datasets download -d behrouzmirabdi/cafa-5-6-train-dataset \ - -p data/cafa-5-cafa-6-protein-function-prediction/Train --unzip +make monitoring-up # Prometheus + Grafana +make training-up # Training API + worker +make all-up # default + monitoring + training ``` -Alternatively, download the zip from the Kaggle page in a browser and extract so the two files end under `Train/` as shown above. +### 4. Access points -**Verify integrity** (sha256 of the files this repo was developed against): +| Endpoint | URL | +|----------|-----| +| Gateway | `http://localhost` | +| Streamlit UI | `http://localhost/ui/` | +| MLflow | `http://localhost/mlflow/` | +| Prometheus | `http://localhost:9090` | +| Grafana | `http://localhost:3000` | ```bash -sha256sum data/cafa-5-cafa-6-protein-function-prediction/Train/train_sequences.fasta \ - data/cafa-5-cafa-6-protein-function-prediction/Train/train_terms.tsv +curl -sk -u admin:PASSWORD http://localhost/api/v1/health +curl -sk -u user:PASSWORD http://localhost/api/predict/health ``` -Expected digests: - -| File | sha256 | -|------|--------| -| `train_sequences.fasta` | `434addef94c14eb8fb263ad2f5801a73a43fcb69d10955e5463d20c6b8aaac82` | -| `train_terms.tsv` | `c9489b802b8955d3cb14c23cc465674de86e08ad23107296260c8a8040361535` | - -## Access Points - -- Gateway root: `https://localhost` -- Streamlit UI: `https://localhost/ui/` -- MLflow via gateway: `https://localhost/mlflow/` -- Prometheus: `http://localhost:9090` -- Grafana: `http://localhost:3000` - -## NGINX Architecture - -`nginx` is the single public ingress on ports `80/443` and enforces operational policy: - -- HTTP->HTTPS redirect. -- Basic auth tiers: - - Admin routes: `/api/v1/*`, `/api/train*`, `/mlflow/`. - - User routes: `/api/predict/*`, `/api/v1/predict-go-from-sequences`, `/api/v1/predict-go-from-fasta`. -- Per-route body size controls: - - `/api/v1/`: 512 MB - - `/api/v1/predict-go-from-fasta`: 5 MB - - `/api/train`: 64 MB - - `/api/predict/`: 8 MB - - `/mlflow/`: 32 MB -- Request rate limits: - - admin zone `15 r/s` (burst 40) - - predict zone `30 r/s` (burst 80) -- Long-job compatible upstream timeouts (`600s` read/send). -- Trace headers forwarded upstream (`X-Trace-Id`, auth tier/user context). - -## MLflow Architecture - -MLflow runs as an internal service and is exposed through gateway path `/mlflow/`. - -- **Backend store:** PostgreSQL (`postgres` service, named volume `postgres_data`) -- **Job history DB:** separate database `proseqgo_jobs` on the same Postgres (embedding + training durable jobs). Init script: `docker/postgres/init-proseqgo-jobs.sh` (first volume only). Existing volumes: `docker compose exec postgres psql -U "$POSTGRES_USER" -c 'CREATE DATABASE proseqgo_jobs;'` -- **Transient job dispatch:** Redis/RQ (`redis`, AOF volume `redis_data`); workers `embedding-worker` / `trainer-worker` -- **Artifact store:** MinIO S3-compatible bucket `mlflow-artifacts` (named volume `minio_data`) -- **Tracking URI for clients:** `http://mlflow:5000` (clients need S3 env vars for direct artifact I/O to MinIO) -- **Secrets:** copy [`.env.example`](.env.example) to `.env` and set Postgres/MinIO credentials before first run -- **Backups:** `postgres-backup` dumps to `./backups/postgres/` daily; `backup-offload` copies dumps to MinIO bucket `mlflow-db-backups` -- Model registry: - - registered model name defaults to `cafa-go-model` (override via `REGISTERED_MODEL_NAME` in `.env`). - - serving API (`go-prediction-api`) loads `models:/cafa-go-model@champion` by default. -- Training API returns MLflow run/model URLs in job status payload when available. - -Legacy file-backed runs (if any) are archived under `archives/` before migration; restore from tarball only for rollback. - -## Monitoring Architecture - -Monitoring is profile-based and isolated from public ingress: - -- Prometheus scrapes: - - `prometheus:9090` - - `embedding-api:8000/metrics` - - `embedding-worker:8001/metrics` - - `go-prediction-api:8000/metrics` - - `trainer-api:8000/metrics` / `trainer-worker:8001/metrics` (when training profile is active) - - `redis-exporter:9121` -- Grafana datasource is provisioned from `monitoring/grafana/provisioning/datasources/prometheus.yml`. -- Dashboards are file-provisioned from `monitoring/grafana/dashboards`. -- Metrics family includes: - - HTTP request/latency/in-flight metrics (`cafa5_http_*`) - - embedding/training Postgres queue depth and RQ queue length - - inference latency, validation failures, top_k distribution - -## Alert Rules - -Defined in `monitoring/alerts.yml`: - -- `Cafa5ServiceMetricsTargetDown` - - condition: `up == 0` for monitored jobs for >2m - - severity: `critical` -- `Cafa5HighHttp5xxRatio` - - condition: service 5xx ratio >5% over 5m and enough traffic, sustained 10m - - severity: `warning` -- `Cafa5EmbeddingQueueBacklogHigh` - - condition: queued embedding jobs >20 for 10m - - severity: `warning` -- `Cafa5EmbeddingWorkerDownWithBacklog` - - condition: embedding worker down while RQ queue non-empty for 5m - - severity: `critical` -- `Cafa5RedisDown` - - condition: `redis_up == 0` for 2m - - severity: `critical` - -Verify: +### 5. CI / CPU smoke ```bash -curl -s http://localhost:9090/-/ready -curl -s http://localhost:9090/api/v1/rules -curl -s http://localhost:9090/api/v1/alerts -``` - -## APIs and Route Map - -All gateway examples below use TLS and basic auth. - -- Embedding API (admin): `/api/v1/...` - - `/api/v1/health` - - `/api/v1/jobs` - - `/api/v1/jobs/fasta` - - `/api/v1/jobs/{job_id}` - - `/api/v1/jobs/{job_id}/artifacts/{name}` - - `/api/v1/jobs/{job_id}/predict-go` - - `/api/v1/predict-go-from-sequences` - - `/api/v1/predict-go-from-fasta` -- GO prediction API (user/admin): `/api/predict/...` - - `/api/predict/health` - - `/api/predict/predict` -- Training API (admin, training profile): `/api/train/...` - - `/api/train/health` - - `/api/train/train` - - `/api/train/jobs/{job_id}` - -## Practical Testing Playbook - -### 1) Health checks - -```bash -curl -sk -u USER:PASS https://localhost/api/v1/health -curl -sk -u USER:PASS https://localhost/api/predict/health -curl -sk -u USER:PASS https://localhost/api/train/health +make ci-up +make smoke +make ci-down ``` -### 2) Async embeddings from sequences +**Full deploy guide:** [docs/deployment.md](docs/deployment.md) -```bash -curl -sk -u ADMIN:ADMIN_PASS -X POST https://localhost/api/v1/jobs \ - -H "Content-Type: application/json" \ - -d '{ - "stage": "test", - "backend": "esm2", - "pooling": "mean", - "batch_size": 2, - "max_length": 1280, - "sequences": [ - {"id": "P1", "sequence": "MKTAYIAKQRQISFVKSHFSRQ"}, - {"id": "P2", "sequence": "GAVLIPFYWSTCMNQDEKRH"} - ] - }' -``` +## Configuration -Poll job and download artifacts: +| Group | Variables (see [`.env.example`](.env.example)) | +|-------|--------------------------------------------------| +| Gateway auth | `GATEWAY_ADMIN_*`, `GATEWAY_USER_*` | +| Postgres | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | +| Jobs DB | `JOBS_DATABASE_URL` | +| Redis / RQ | `REDIS_URL`, `EMBEDDING_JOB_TIMEOUT_SEC`, `TRAINING_JOB_TIMEOUT_SEC` | +| MinIO / S3 | `MINIO_*`, `AWS_*`, `MLFLOW_S3_ENDPOINT_URL`, `MLFLOW_ARTIFACT_ROOT` | +| Registry | `REGISTERED_MODEL_NAME`, `PROMOTION_THRESHOLD` | -```bash -curl -sk -u ADMIN:ADMIN_PASS https://localhost/api/v1/jobs/ -curl -sk -u ADMIN:ADMIN_PASS -o test_ids.npy \ - https://localhost/api/v1/jobs//artifacts/test_ids.npy -curl -sk -u ADMIN:ADMIN_PASS -o test_embeddings.npy \ - https://localhost/api/v1/jobs//artifacts/test_embeddings.npy -``` +Pipeline hyperparameters live in [`configs/config.yaml`](configs/config.yaml). Keep `embedding.backend` aligned with training embeddings and the GO predictor’s expected dimension. -### 3) Direct embedding -> GO inference +## Training workflow -```bash -python - <<'PY' -import json -import numpy as np -import requests - -embedding = np.load("test_embeddings.npy")[0].astype(float).tolist() -r = requests.post( - "https://localhost/api/predict/predict", - auth=("USER", "USER_PASS"), - json={"embedding": embedding, "top_k": 10}, - verify=False, - timeout=60, -) -print(r.status_code) -print(json.dumps(r.json(), indent=2)) -PY +```text +preprocess → split → embed → train → evaluate_holdout → promote_model ``` -### 4) Sequence -> GO in one call (main integration endpoints) +| Mode | When to use | +|------|-------------| +| CLI (`scripts/retrain_pipeline.py`) | Research iteration, full control | +| Training API (`/api/train/train`) | Ops automation (training profile) | +| Hybrid | Embed via API; train/promote via CLI | -Two sync wrappers share the same response contract (`PredictGoResponse`): embed → wait for job completion → call GO prediction API per sequence. +Primary promotion metric: `holdout_f1_micro` (default threshold `0.35`). Serving consumes the `champion` alias unless `MODEL_URI` is overridden. -#### Option A: JSON sequences +**Details:** [docs/training.md](docs/training.md) -Endpoint: `POST /api/v1/predict-go-from-sequences` +## Serving and UI -Request contract: +| Path | Purpose | Auth tier | +|------|---------|-----------| +| `/ui/` | Streamlit product UI | gateway | +| `/api/predict/*` | Embedding → GO | user / admin | +| `/api/v1/predict-go-from-sequences` | Sequence → GO (JSON) | user / admin | +| `/api/v1/predict-go-from-fasta` | Sequence → GO (FASTA) | user / admin | +| `/api/v1/jobs*` | Async embedding jobs | admin | +| `/api/train*` | Async train / retrain | admin (training profile) | +| `/mlflow/*` | Tracking + registry UI | admin | -- `backend`: `esm2 | protbert | t5` -- `pooling`: `mean | cls` -- `batch_size`: `1..128` -- `max_length`: `8..8192` -- `top_k`: `1..500` -- `sequences`: non-empty list of `{id, sequence}` -- optional: - - `indices`: subset prediction indices - - `fail_fast`: `true|false` - - `timeout_seconds`: `5..7200` (default `1800`) - - `poll_interval_seconds`: `0.1..5.0` (default `1.0`) - -Example: +Example (sequence → GO): ```bash -curl -sk -u USER:USER_PASS -X POST \ - https://localhost/api/v1/predict-go-from-sequences \ +curl -sk -u user:PASSWORD -X POST \ + http://localhost/api/v1/predict-go-from-sequences \ -H "Content-Type: application/json" \ -d '{ "backend": "esm2", @@ -375,177 +238,113 @@ curl -sk -u USER:USER_PASS -X POST \ "batch_size": 2, "max_length": 1280, "top_k": 10, - "fail_fast": true, "sequences": [ - {"id": "P1", "sequence": "MKTAYIAKQRQISFVKSHFSRQ"}, - {"id": "P2", "sequence": "GAVLIPFYWSTCMNQDEKRH"} + {"id": "P1", "sequence": "MKTAYIAKQRQISFVKSHFSRQ"} ] }' ``` -#### Option B: FASTA upload - -Endpoint: `POST /api/v1/predict-go-from-fasta` (`multipart/form-data`) +Service-specific notes: -Form fields mirror the JSON endpoint (`backend`, `pooling`, `batch_size`, `max_length`, `top_k`, `fail_fast`, `timeout_seconds`, `poll_interval_seconds`) plus required `fasta_file`. There is no `indices` form field — all parsed records are predicted. `sequence_id` in results is the first token after `>` in each FASTA header. +- [services/embedding-api/README.md](services/embedding-api/README.md) +- [services/training-api/README.md](services/training-api/README.md) +- [services/streamlit-ui/README.md](services/streamlit-ui/README.md) -Example: +## Monitoring and operations ```bash -curl -sk -u USER:USER_PASS -X POST \ - https://localhost/api/v1/predict-go-from-fasta \ - -F "fasta_file=@examples/small_sequences.fasta" \ - -F "backend=esm2" \ - -F "pooling=mean" \ - -F "batch_size=2" \ - -F "max_length=1280" \ - -F "top_k=10" \ - -F "fail_fast=true" -``` - -**Residue normalization (both endpoints):** Sequences are normalized at embedding time: uppercase, whitespace stripped, canonical amino acids kept, and `X`/`U`/`O`/`B`/`Z`/`J` plus any other unknown symbols mapped to `X` (see `normalize_sequence` in `scripts/embed_sequences.py`). - -**Operational limits:** - -- FASTA upload cap: **5 MB** (API + NGINX on `/api/v1/predict-go-from-fasta`). Larger uploads return `413` (`FASTA_FILE_TOO_LARGE`). -- No sequence-count cap on the FASTA endpoint — only file size. Dense FASTA files may still time out because GO inference is sequential. -- Default sync timeout: **1800 s** (embedding wait + per-sequence GO calls). -- Large or proteome-scale jobs: `POST /api/v1/jobs/fasta` → poll → `POST /api/v1/jobs/{job_id}/predict-go`. - -What happens internally: - -1. `embedding-api` creates and runs an embedding job (from JSON sequences or parsed FASTA). -2. It waits for completion with polling (`timeout_seconds`, `poll_interval_seconds`). -3. It loads generated `test_embeddings.npy`. -4. For each selected item, it calls `go-prediction-api /predict`. -5. It returns aggregated predictions and per-item failures. - -Response shape (simplified): - -```json -{ - "job_id": "uuid", - "status": "succeeded", - "model_version": "12", - "top_k": 10, - "results": [ - { - "index": 0, - "sequence_id": "P1", - "predictions": [{"go_term": "GO:0000000", "score": 0.82}] - } - ], - "failures": [] -} -``` - -### 5) Trigger retraining via API - -```bash -curl -sk -u ADMIN:ADMIN_PASS -X POST https://localhost/api/train/train \ - -H "Content-Type: application/json" \ - -d '{"config":"configs/config.yaml","mode":"retrain"}' -``` - -Poll: - -```bash -curl -sk -u ADMIN:ADMIN_PASS https://localhost/api/train/jobs/ +make monitoring-up ``` -## Retraining Workflow Options +| Signal | Why it matters | +|--------|----------------| +| Target `up` | Service scrape health | +| HTTP 5xx ratio / latency | Serving quality | +| Embedding / training queue depth | Backlog and worker health | +| Inference validation failures | Embedding dimension / schema drift | +| `redis_up` | Job dispatch dependency | -### Option A: Pure CLI retraining (recommended for research iteration) +Alert rules live in `monitoring/alerts.yml` (service down, high 5xx, queue backlog, worker-down-with-backlog, Redis down). -```bash -python scripts/retrain_pipeline.py --config configs/config.yaml \ - --promotion-threshold 0.35 \ - --model-name cafa-go-model -``` +**Details:** [docs/monitoring.md](docs/monitoring.md) · [monitoring/README.md](monitoring/README.md) -### Option B: API-driven retraining (recommended for ops automation) +## Testing and CI -- Start `training` profile. -- Submit `/api/train/train` with mode `retrain`. -- Poll until completion. -- Use MLflow links in final job payload for audit and model version traceability. +| Check | Command | Notes | +|-------|---------|-------| +| Lint | `make lint` | Ruff on `src/`, `services/`, `scripts/` | +| Unit tests | `make test` | `tests/unit` — no Docker/GPU/network | +| Smoke | `make smoke` | Compose stack must already be up | +| Images | `make build-images` / `make pull-images` | Local build or GHCR pull | -### Option C: Hybrid flow +GitHub Actions (PR / `main`): lint → unit tests → parallel image builds. Merges to `main` publish to GHCR (`sha-` + `main`). See [`.github/CI.md`](.github/CI.md). -- Generate embeddings in service mode (`/api/v1/jobs`) for online or ad-hoc data. -- Train/evaluate in CLI for maximum flexibility. -- Promote registry alias after metric gates. +CI does **not** run GPU training or full retrain jobs. -## Different Ways to Implement/Deploy This Project +## Deployment -### 1) Monolith-like local stack (current default) +| Environment | Compose | Typical use | +|-------------|---------|-------------| +| Local CPU | `docker-compose.yml` | Dev without NVIDIA | +| Local GPU | base + `docker-compose.gpu.yml` | `make up` auto-detects | +| CI / smoke | base + `docker-compose.ci.yml` | `make ci-up` | +| Full stack | base (+ GPU) + `training` + `monitoring` | Integration / demos | -- One compose file, one host, local volumes. -- Best for development and reproducible demos. +Suggested production posture: keep NGINX + inference + MLflow always on; run training on dedicated compute; promote only via metric gates; enable monitoring by default. -### 2) API-first deployment +**Details:** [docs/deployment.md](docs/deployment.md) -- Keep `nginx + embedding-api + go-prediction-api + mlflow`. -- Add `training-api` only in restricted environments. -- Good for production inference where retraining is decoupled. +## Documentation -### 3) Training separated from serving +| Doc | Contents | +|-----|----------| +| [docs/architecture.md](docs/architecture.md) | Services, flows, security boundaries | +| [docs/data.md](docs/data.md) | Kaggle source, layout, checksums, versioning | +| [docs/training.md](docs/training.md) | Pipeline, metrics, promotion, MLflow | +| [docs/deployment.md](docs/deployment.md) | Environments, secrets, Compose, GHCR | +| [docs/monitoring.md](docs/monitoring.md) | Metrics, alerts, operational response | +| [docs/troubleshooting.md](docs/troubleshooting.md) | Symptom → cause → fix | +| [docs/contributing.md](docs/contributing.md) | Dev setup, PR workflow, standards | -- Run training pipeline on dedicated compute (GPU/HPC/batch scheduler). -- Push selected model versions to shared MLflow registry. -- Serving stack only consumes `@champion` alias. +## Contributing -### 4) Monitoring-hardened setup +```bash +python -m venv .venv && source .venv/bin/activate +pip install -e ".[dev]" +make lint && make test +``` -- Enable monitoring profile by default. -- Add Alertmanager and external notification integrations. -- Formalize SLOs around 5xx ratio, p95 latency, and queue backlog. +Prefer small, reviewable PRs. Discuss design first for public API contract changes, registry/alias strategy, new heavy dependencies, jobs DB schema, or security model changes. -## Reproducibility and QC Recommendations +**Details:** [docs/contributing.md](docs/contributing.md) -- Keep `embedding.backend` and trained model embedding dimension aligned to avoid invalid inference inputs. -- Use deterministic seeds and fixed splits for comparable retrains. -- Monitor class imbalance and threshold sensitivity (multi-label GO bias risk). -- Track data snapshot/version metadata in MLflow to prevent annotation drift confusion. -- Watch inference validation failure reasons for upstream schema drift. +## Troubleshooting -## Useful Make Targets +Start here when something fails: ```bash -make up # Default stack (GPU overlay if NVIDIA detected) -make down -make ci-up # CPU CI/smoke stack (base + ci overlay) -make ci-down # Stop CI stack and remove volumes -make training-up -make training-down -make monitoring-up -make monitoring-down -make lint # Ruff on src/ services/ scripts/ -make test # Unit tests (tests/unit) -make build-images # Build all five product images -make pull-images # Pull product images from GHCR (GHCR_TAG=main|sha-...) -make smoke # Smoke scripts (Compose stack must already be up) -make ci-env # Copy .env.example → .env if missing -make gateway-auth # Write nginx/.htpasswd-* from GATEWAY_* in .env +docker compose ps +docker compose logs --tail=100 +curl -s http://localhost:9090/-/ready +curl -sk -u admin:PASS http://localhost/api/v1/health ``` -## CI (GitHub Actions) - -PR and `main` pushes run **lint**, **unit tests**, and **parallel image builds**. Merges to `main` also **publish to GHCR** (`sha-` + `main`). See [`.github/CI.md`](.github/CI.md). +Common issues covered in the runbook: missing `.env` / htpasswd, `proseqgo_jobs` DB, gateway 401/502, GPU not visible, MLflow / MinIO artifact errors, embedding–model dimension mismatch, queue backlog. -- Registry: **GHCR** (`ghcr.io/behroooz/proseqgo-*`) -- Compose: portable **base** + **`docker-compose.ci.yml`** for CPU smoke (`make ci-up`) -- GPU local dev: **base** + **`docker-compose.gpu.yml`** (`make up` auto-detects NVIDIA) -- CI does **not** run training/GPU/retrain jobs -- Compose secrets: `.env` from `.env.example`; gateway Basic Auth via `make gateway-auth` (`GATEWAY_ADMIN_*` ≠ `GATEWAY_USER_*`) -- Local image rebuild: `make build-images`; pull published: `make pull-images` +**Details:** [docs/troubleshooting.md](docs/troubleshooting.md) -## Service-Specific Documentation +## Useful Make targets -- `services/embedding-api/README.md` -- `services/training-api/README.md` -- `services/streamlit-ui/README.md` -- `monitoring/README.md` +```bash +make up / make down +make ci-up / make ci-down +make training-up / make training-down +make monitoring-up / make monitoring-down +make all-up / make all-down +make lint / make test / make smoke +make build-images / make pull-images +make ci-env / make gateway-auth +``` ## License