Introduction

Language can be used a biomarker for early detection of psychiatric illnesses, but methods for capturing linguistic signal are still in their infancy. model-experiments contains experiments for utilizing language as a biomarker, specifically predicting Clinical High Risk - Psychosis (shorthand: CHR) versus Community Control (CC) status.

Language models with classifier heads

Language models already have robust internal representations of language that can be utilized for downstream classification tasks. Challenges include variable length and inconsistent data, imbalanced classes, and overall lack of training and validation data. These models attempt to balance generalization with predictive power. Loosely inspired by these papers, but with significant adaptions to accomodate language data and psychosis-risk dataset:

Human whole epigenome modelling for clinical applications with Pleiades (2025)

Using Interpretability to Identify a Novel Class of Alzheimer's Biomarkers (2026)

All variants share one frozen Hugging Face base model as a feature extractor — only the lightweight classifier head trains. The backbone is model-agnostic and swappable per run via a --model CLI argument (with --load-mode for 8/4-bit quantization of large models), so the same pipeline works with any AutoModel.

Models

classifier_mean_pool.py: Utilizes mean pooling strategy for all participant utterance embeddings for final classification

classifier_attention_pool.py: Embeds each utterance separately and uses a single learned attention query to take a weighted average over a patient's utterances before final classification

classifier_attention_pool_multihead.py: Extends attention pooling with several independent attention queries whose pooled vectors are concatenated, letting different heads attend to different linguistic signals

classifier_attention_pool_ffn.py: Attention pooling with a small feed-forward layer (Linear → GELU → Dropout → Linear) before the final logit, adding nonlinear mixing of the pooled representation

classifier_attention_pool_multihead_ffn.py: Multi-head attention pooling with the same small feed-forward layer before the final logit

Running

Each script runs standalone (python classifier_mean_pool.py --model <hf-id>), or run every methodology in sequence with:

python run_all.py --model google/gemma-4-E2B-it

Only the flags you pass (--model, --load-mode, --extract-batch, --gpu, --max-length, --seed) are forwarded, so omitted flags keep each script's own defaults. Results land in results/<model>/<methodology>/len<λ>_seed<σ>/, and every run upserts its row into results/<model>/summary.csv — one row per (methodology, max_length, seed) with all accuracy statistics — so the summary fills completely after a sweep or incrementally when scripts are run by hand.

Hyperparameter sweeps

run_all.py can vary MAX_LENGTH and seed across a whole sweep. Length is per-featurization: --max-length-utterance goes to the four attention scripts, --max-length-transcript to mean_pool. --seeds a b c repeats the sweep once per seed (embedding caches are seed-independent, so only the first seed pays extraction):

python run_all.py --model google/gemma-4-E2B-it \
    --max-length-utterance 1024 --max-length-transcript 16384 --seeds 42 43 44

Supporting tools:

truncation_stats.py: tokenizes the cohort with a backbone's tokenizer and reports token-length percentiles and the fraction of transcripts/utterances truncated at each candidate MAX_LENGTH. Run it first — if nothing is truncated at the current limit, a longer limit cannot change the embeddings.

compare_results.py: aggregates every results/<model>/summary.csv into results/comparison.csv (seed replicates reduced to mean ± sd) plus an AUROC-vs-max_length plot per methodology, for reading length and model-size effects side by side.