Refactor/streamlit

docker-compose.yml

@@ -321,6 +321,8 @@ services:
       # Predict-route credentials (public user). Must match make gateway-auth / .htpasswd-user.
       GATEWAY_USER: ${GATEWAY_USER:-user}
       GATEWAY_USER_PASSWORD: ${GATEWAY_USER_PASSWORD:-change-me-gateway-user}
+      # Compose uses plain HTTP to nginx; disable TLS verification explicitly.
+      GATEWAY_VERIFY_TLS: "false"
     depends_on:
       - embedding-api
       - go-prediction-api

docs/data.md

@@ -6,12 +6,14 @@ This document defines data sources, layout, versioning, preprocessing, and repro
 
 ### Primary training dataset (Kaggle)
 
-| Field | Value |
-|-------|-------|
+
+| Field   | Value                                                                                           |
+| ------- | ----------------------------------------------------------------------------------------------- |
 | Dataset | [cafa-5-6-train-dataset](https://www.kaggle.com/datasets/behrouzmirabdi/cafa-5-6-train-dataset) |
-| Owner | `behrouzmirabdi` |
-| Access | Kaggle API (`~/.kaggle/kaggle.json`) or browser download |
-| License | See Kaggle dataset page |
+| Owner   | `behrouzmirabdi`                                                                                |
+| Access  | Kaggle API (`~/.kaggle/kaggle.json`) or browser download                                        |
+| License | See Kaggle dataset page                                                                         |
+
 
 **Required files** (paths relative to repo root, matching `configs/config.yaml`):
 
@@ -22,6 +24,8 @@ data/cafa-5-cafa-6-protein-function-prediction/
     └── train_terms.tsv
 ```
 
+
+
 ### Integrity checksums
 
 Verify files after download:
@@ -31,20 +35,26 @@ sha256sum data/cafa-5-cafa-6-protein-function-prediction/Train/train_sequences.f
           data/cafa-5-cafa-6-protein-function-prediction/Train/train_terms.tsv
 ```
 
-| File | Expected sha256 |
-|------|-----------------|
+
+| File                    | Expected sha256                                                    |
+| ----------------------- | ------------------------------------------------------------------ |
 | `train_sequences.fasta` | `434addef94c14eb8fb263ad2f5801a73a43fcb69d10955e5463d20c6b8aaac82` |
-| `train_terms.tsv` | `c9489b802b8955d3cb14c23cc465674de86e08ad23107296260c8a8040361535` |
+| `train_terms.tsv`       | `c9489b802b8955d3cb14c23cc465674de86e08ad23107296260c8a8040361535` |
+
+
+
 
 ### External model weights (Hugging Face)
 
 Embedding backends download pretrained weights on first use:
 
-| Backend key | HF model |
-|-------------|----------|
-| `esm2` | `facebook/esm2_t33_650M_UR50D` |
-| `protbert` | `Rostlab/prot_bert` |
-| `t5` | `Rostlab/prot_t5_xl_uniref50` |
+
+| Backend key | HF model                       |
+| ----------- | ------------------------------ |
+| `esm2`      | `facebook/esm2_t33_650M_UR50D` |
+| `protbert`  | `Rostlab/prot_bert`            |
+| `t5`        | `Rostlab/prot_t5_xl_uniref50`  |
+
 
 Cache directory: `data/hf_cache/` (mounted in embedding containers).
 
@@ -70,16 +80,20 @@ outputs/
 └── training_api/                                 # Training API job outputs
 ```
 
+
+
 ### Raw vs processed vs derived
 
-| Stage | Location | Regenerable | Git-tracked |
-|-------|----------|-------------|-------------|
-| Raw FASTA + terms | `data/.../Train/` | Re-download from Kaggle | No |
-| Label matrix | `outputs/labels/` | `scripts/preprocess.py` | No |
-| Splits | `outputs/splits/` | `scripts/split_train_holdout.py` | No |
-| Embeddings | `data/embeddings/` | `scripts/embed_sequences.py` | No |
-| HF cache | `data/hf_cache/` | Auto on first embed | No |
-| Checkpoints / MLflow artifacts | `outputs/`, MinIO | Training pipeline | No |
+
+| Stage                          | Location           | Regenerable                      | Git-tracked |
+| ------------------------------ | ------------------ | -------------------------------- | ----------- |
+| Raw FASTA + terms              | `data/.../Train/`  | Re-download from Kaggle          | No          |
+| Label matrix                   | `outputs/labels/`  | `scripts/preprocess.py`          | No          |
+| Splits                         | `outputs/splits/`  | `scripts/split_train_holdout.py` | No          |
+| Embeddings                     | `data/embeddings/` | `scripts/embed_sequences.py`     | No          |
+| HF cache                       | `data/hf_cache/`   | Auto on first embed              | No          |
+| Checkpoints / MLflow artifacts | `outputs/`, MinIO  | Training pipeline                | No          |
+
 
 Serving (`make up`) does **not** require training data. Preprocess, embed, train, and holdout evaluation do.
 
@@ -96,8 +110,12 @@ Record in every training run:
 - `embedding.backend` and `embedding.pooling`
 - split seed / holdout fraction from config
 
+
+
 ## Ingestion workflow
 
+
+
 ### Download via Kaggle CLI
 
 ```bash
@@ -106,6 +124,8 @@ kaggle datasets download -d behrouzmirabdi/cafa-5-6-train-dataset \
   -p data/cafa-5-cafa-6-protein-function-prediction/Train --unzip
 ```
 
+
+
 ### Validation checks
 
 After download:
@@ -114,8 +134,12 @@ After download:
 2. Run sha256 verification (table above).
 3. Spot-check FASTA record count and terms file column structure.
 
+
+
 ## Preprocessing pipeline
 
+
+
 ### 1. Label matrix
 
 ```bash
@@ -129,6 +153,8 @@ Key config (`configs/config.yaml`):
 - `data.num_labels`: top-N GO terms (default 500)
 - `data.train_val_split`: train/validation fraction within labeled set
 
+
+
 ### 2. Train/holdout split
 
 ```bash
@@ -161,14 +187,20 @@ API endpoints (`/api/v1/predict-go-from-sequences`, FASTA upload) use the same n
 
 ## Data contracts
 
+
+
 ### Training / inference inputs
 
-| Field | Requirement |
-|-------|-------------|
-| Protein ID | Non-empty string; matches FASTA header or JSON `id` |
-| Sequence | Amino acid string; normalized as above |
-| Embedding vector | Length must match model input dim for GO predictor |
-| GO labels | `GO:#######` format in terms file |
+
+| Field            | Requirement                                         |
+| ---------------- | --------------------------------------------------- |
+| Protein ID       | Non-empty string; matches FASTA header or JSON `id` |
+| Sequence         | Amino acid string; normalized as above              |
+| Embedding vector | Length must match model input dim for GO predictor  |
+| GO labels        | `GO:#######` format in terms file                   |
+
+
+
 
 ### Config-driven paths
 
@@ -186,11 +218,13 @@ Do not hardcode machine-specific absolute paths in scripts or configs committed
 
 ## Storage and retention
 
-| Environment | Raw data | Embeddings | Artifacts |
-|-------------|----------|------------|-----------|
-| Local dev | `./data/` bind mount | `./data/embeddings/` | `./outputs/` |
-| Compose services | `./data`, `./outputs` volumes | `./data/hf_cache` | `./outputs/service_artifacts/` |
-| MLflow (prod-like) | N/A | N/A | MinIO `mlflow-artifacts` bucket |
+
+| Environment        | Raw data                      | Embeddings           | Artifacts                       |
+| ------------------ | ----------------------------- | -------------------- | ------------------------------- |
+| Local dev          | `./data/` bind mount          | `./data/embeddings/` | `./outputs/`                    |
+| Compose services   | `./data`, `./outputs` volumes | `./data/hf_cache`    | `./outputs/service_artifacts/`  |
+| MLflow (prod-like) | N/A                           | N/A                  | MinIO `mlflow-artifacts` bucket |
+
 
 **Regenerable without data loss:** embeddings, splits, label matrix, local checkpoints.
 
@@ -207,14 +241,19 @@ To rerun training on the same data:
 - [ ] Point `MLFLOW_TRACKING_URI` at the same tracking server
 - [ ] Log dataset checksums from `train_run_summary.json`
 
+
+
 ## Privacy and compliance
 
 - Training data is public competition data; confirm Kaggle license before redistribution.
 - Do not commit raw data, credentials, or user-submitted sequences from production inference to git.
 - Service artifacts under `outputs/service_artifacts/` may contain user sequences; treat as sensitive in shared environments.
 
+
+
 ## Related documentation
 
 - [training.md](training.md) — how processed data feeds the training pipeline
 - [architecture.md](architecture.md) — data flow through services
 - [deployment.md](deployment.md) — volume mounts and data paths in Compose
+

scripts/build_go_term_metadata.py

@@ -0,0 +1,121 @@
+#! /usr/bin/env python
+"""Build the GO term metadata for the GO term prediction model from go-basic.obo file
+and for outputs/label_matrix_top500/term_names.npy GO term names.
+"""
+
+import argparse
+from pathlib import Path
+import numpy as np
+import pandas as pd
+
+def parse_obo(go_basic_obo_file: Path) -> list[dict]:
+    """Parse a go-basic.obo file into a list of term dicts."""
+
+    terms = []
+    current = None
+    in_term = False
+
+    with open(go_basic_obo_file, 'r', encoding='utf-8') as f:
+        for line in f:
+            line = line.rstrip('\n')
+            stripped = line.strip()
+
+            if stripped.startswith('['):
+                if current is not None and 'id' in current:
+                    terms.append(current)
+                if stripped == '[Term]':
+                    current = {}
+                    in_term = True
+                else:
+                    current = None
+                    in_term = False
+                continue
+
+            if not in_term or stripped == '':
+                continue
+
+            if ':' not in line:
+                continue
+
+            key, _, value = line.partition(':')
+            key = key.strip()
+            value = value.strip()
+
+            if key == 'id':
+                current['id'] = value
+            elif key == 'name':
+                current['name'] = value
+            elif key == 'namespace':
+                current['namespace'] = value
+            elif key == 'def':
+                if value.startswith('"'):
+                    end_quote = value.find('"', 1)
+                    current['def'] = value[1:end_quote]
+                else:
+                    current['def'] = value
+
+        if current is not None and 'id' in current:
+            terms.append(current)
+
+    return terms
+
+
+def build_dataframe(obo_path, npy_path):
+    """Load the array of GO term names we want to filter on."""
+    term_names = np.load(npy_path, allow_pickle=True)
+    term_names_set = set(term_names)
+
+    all_terms = parse_obo(obo_path)
+
+    rows = []
+    for t in all_terms:
+        go_id = t.get('id')
+        if go_id is not None and go_id in term_names_set:
+            rows.append({
+                'GO_term': go_id,
+                'name': t.get('name'),
+                'namespace': t.get('namespace'),
+                'def': t.get('def'),
+            })
+
+    df = pd.DataFrame(rows, columns=['GO_term', 'name', 'namespace', 'def'])
+    return df
+
+
+def make_dir_if_not_exists(path: Path, exist_ok: bool = True) -> None:
+    """Make a directory if it doesn't exist."""
+    if not path.exists():
+        path.mkdir(parents=True, exist_ok=exist_ok)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Build the GO term metadata for the GO term prediction model from go-basic.obo file and for outputs/label_matrix_top500/term_names.npy GO term names.")
+    parser.add_argument(
+        "--go-basic-obo-file", 
+        type=str, 
+        default="data/cafa-6-protein-function-prediction/Train/go-basic.obo", 
+        help="Path to the go-basic.obo file")
+    parser.add_argument(
+        "--term-names-file", 
+        type=str, 
+        default="outputs/label_matrix_top500/term_names.npy", 
+        help="Path to the term_names.npy file")
+    parser.add_argument(
+        "--output-file", 
+        type=str, 
+        default="services/streamlit-ui/metadata/go_term_metadata.csv", 
+        help="Path to the output file")
+    args = parser.parse_args()
+
+    go_basic_obo_file = Path(args.go_basic_obo_file)
+    term_names_file = Path(args.term_names_file)
+
+    make_dir_if_not_exists(Path(args.output_file).parent)
+
+    df = build_dataframe(go_basic_obo_file, term_names_file)
+    df.to_csv(args.output_file, index=False)
+
+    print(f"GO term metadata saved to {args.output_file}")
+
+if __name__ == "__main__":
+    main()