AstroVision — LoRA fine-tuning, and an audit of the pipeline it came from

A clean LoRA fine-tune of Llama-3.2-11B-Vision for astronomy image captioning on one free T4 — and a written record of the five defects I found in the published pipeline I reproduced it from, including one that made its public demo serve the base model instead of the fine-tune.

Live demo · Debugging log → · Model card

---

Why this repository has two halves

Anyone can call FastVisionModel.from_pretrained and post a BLEU score. The harder and more useful skill is noticing when your own number is wrong. So this repository is deliberately two things:

1. A clean pipeline — notebook/finetune_astrovision.ipynb. EDA, caption normalization, a train/valid/test split made before anything is trained, LoRA fine-tuning with validation loss, evaluation through a single shared inference function, adapter export.

2. An audit — DEBUGGING.md. Five findings, each with symptom, hypothesis, root cause, fix and evidence. Every claim about the audited pipeline cites committed execution output, reproduced verbatim under evidence/ so none of it has to be taken on trust.

The whole audit started from one number that did not fit: length_ratio = 2.05.

---

The findings

Full write-up in DEBUGGING.md. In short:

#	Finding	Impact
1	The evaluation decode echoed the prompt into every prediction	Every reported metric was measuring the instruction as well as the caption
2	A correct 200/25/25 split was computed, then overwritten by a later cell	Every evaluated example had been trained on; the training banner said Num examples = 250
3	temperature passed to generate() without do_sample	Decoding behaviour came from the checkpoint, so the same code changes behaviour with the model id
4	The published demo never loaded its fine-tuned adapter	The deployed model was the base model. Silent: LoRA initialises B to zero, so a missing adapter behaves exactly like no adapter
5	The documented class balance did not match the run	Mars: 54 in the README against Mars 42, Unknown 12 in the output

Finding 4 is the one I would lead with in conversation, and it needed no GPU — only reading the committed output of a notebook that was already public. Four independent lines of evidence: no adapter file in the download log, the create-a-new-adapter code path taken, a freshly constructed adapter in the printed module tree, and output in unmistakable base-model style.

Findings 1–3 are fixed in this pipeline. Finding 4's fix is scripts/verify_adapter_loading.py plus a startup check that app.py shows on the page.

---

What the fixes look like

The three pipeline defects share a shape: correct code existed, and nothing required it to be used. So the fixes mostly remove the option rather than add a reminder.

One inference path. generate_caption() is the only model.generate call in the repository, and a test enforces that by parsing every .py and .ipynb:

# astrovision_core.py -- imported by the notebook, the ablation runner and app.py
prompt_len = int(inputs["input_ids"].shape[1])
output_ids = model.generate(**inputs, **settings.to_generate_kwargs())
caption = tokenizer.decode(output_ids[0][prompt_len:], skip_special_tokens=True).strip()

A split you cannot separate from its held-out sets, plus a guardrail on the object actually handed to the trainer:

splits = core.split_records(records, test_size=0.2, seed=42)   # all three, together
train_dataset = Dataset.from_list([core.build_conversation(r, DATA_ROOT) for r in splits.train])
core.assert_no_leakage(train_dataset, test_conversations, "train_dataset", "test")

Sampling stated, never inherited — and only passed when it takes effect, so a temperature in a log is always a temperature that applied:

kwargs = {"max_new_tokens": ..., "do_sample": self.temperature > 0, "use_cache": True}
if self.do_sample:
    kwargs["temperature"] = self.temperature
    kwargs["top_p"] = self.top_p

Validation loss, which the original pipeline had none of — making overfitting literally unobservable on a 250-example corpus with ~67 M trainable parameters (a deterministic consequence of the LoRA config, not a measurement).

---

Pipeline

1. Rebuild the dataset locally from Hugging Face (the original read a private Drive archive, so nobody else could run it)
2. Audit the corpus — integrity, length distribution, heuristic topic labels with the Unknown bucket reported rather than folded away
3. Normalize captions conservatively; the aggressive upstream cleaner is kept behind a flag and its cost is shown
4. Validate images with verify() and load() — a header-only check passes truncated files that then fail during training
5. Split into 200/25/25 before building any training dataset, and assert it
6. Load the 4-bit base model, attach LoRA (r=16, alpha=16)
7. Verify what the loss is computed on — the unmasked label positions should decode to exactly the caption
8. Train 30 steps at an effective batch of 8, with validation loss
9. Evaluate the held-out split through the shared generate_caption(), greedy
10. Save the adapter (tens of MB, not a 20 GB+ merge)

---

Running it

Fine-tuning — open notebook/finetune_astrovision.ipynb in Colab on a T4 runtime and run the cells in order. The 2026-08-21 run took 20.1 min for training and evaluation, plus the dependency install and a ~7 GB model download.

Tests — no GPU, no ML stack, no model download:

pip install -r requirements-dev.txt && python -m pytest tests/ -q

Test	What it actually verifies
test_inference_consistency.py	model.generate is called in exactly one file, once, and no notebook calls it. Prompt slicing and explicit sampling, against a stub model with known prompt and continuation tokens
test_no_leakage.py	Split sizes, determinism, disjointness. Includes a regression test that reproduces Finding 2 and asserts the guardrail fires
test_collator_masking.py	The loss-masking rule, including a deliberately broken mask to show the check has teeth. Real-collator check is opt-in via ASTROVISION_RUN_GPU_TESTS=1

Ablation — does training the vision tower help at 250 examples? ablation/vision_tower_ablation.md has the design and a pre-registered prediction. A T4 cannot hold two 11B models, so the arms run separately:

python ablation/run_ablation.py --arm vision_on

Verify an adapter actually loaded — the check the upstream deployment needed:

python scripts/verify_adapter_loading.py --adapter SamHung/astrovision-lora --image sample.jpg

Demo — see spaces/DEPLOY.md.

---

Results

Tesla T4, 2026-08-21. Raw log committed at evidence/runs/2026-08-21-t4-mainline.json — my own numbers are checkable on the same terms I asked of the pipeline I audited.

The controlled result. One set of generations from the 25 held-out images, decoded two ways. Only the decode differs, so the whole gap is the prompt echo:

Metric	Prompt echoed (the defect)	Prompt sliced (fixed)
BLEU	0.0348	0.0722
length_ratio	1.9927	0.9561
ROUGE-1	0.2778	0.3735
ROUGE-L	0.2413	0.3361

The bug was hiding half the BLEU score. And the unsliced length_ratio of 1.9927 lands next to the 2.0548 that started the audit upstream — a different environment and a separately trained adapter, and the echo still doubles measured length. That is the diagnosis reproducing where it was not fitted.

Training. 200/25/25 split, 30 steps, 20.1 min, peak 9.227 of 14.563 GB. Validation loss fell monotonically 2.2722 → 0.7232 with its minimum at the final step: no overfitting at this budget, undertrained if anything.

Still open, and marked as such: the vision-tower ablation has not been run, and the adapter has not been published, so the three serving-path checks in scripts/verify_adapter_loading.py have not run against a real artifact.

One prediction in MODEL_CARD.md turned out wrong — I expected scores below the audited pipeline's and got scores above it. The mistake and its cause are recorded there rather than edited out; a repository about honest numbers does not get to quietly fix its own bad call.

The upstream figures quoted in DEBUGGING.md are a different thing: they are real, they are cited to specific cells, and they describe the pipeline being audited rather than this one. The two are not directly comparable — different environment, different dataset reconstruction path, different caption normalization — which is why the before/after comparison that matters is constructed as a controlled one instead: generate once, decode two ways, and the only variable is the decode.

---

Layout

astrovision_core.py    # every shared piece: generate_caption, splitting,
                       # the guardrail, cleaning, metrics, adapter checks
app.py                 # Gradio demo; imports the same generate_caption
notebook/              # the clean pipeline
DEBUGGING.md           # the audit
evidence/              # verbatim upstream output, plus the extractor
ablation/              # vision-tower ablation: design, runner, results
scripts/               # adapter load verification
tests/                 # runnable without a GPU
spaces/                # Space card and deployment steps

---

Provenance

Coursework project. Reproduced and extended from public work, and it is worth being precise about which parts are whose:

• Dataset — AIOmarRehan/space-multimodal-dataset, 250 astronomy image/caption pairs.
• Pipeline reproduced — AIOmarRehan/Unsloth_Llama_3.2_11B_Vision_Instruct_Astronomy (MIT, © Omar Rehan). The architecture, hyperparameters and conversation format are theirs. I worked from my own fork of it — so to be unambiguous: the audited code and every execution log cited under evidence/ are the original author's, not mine. The findings are about their pipeline; the fork is just the checkout I read.
• Mine — the audit and its evidence, the three pipeline fixes, validation loss, the single shared inference path, the test suite, the ablation design, and the adapter verification.

I found and fixed three data/evaluation defects and one deployment defect in that pipeline, and documented a documentation discrepancy. That is the point of the repository, and hiding where it came from would remove the part worth showing: auditing someone else's working, published code is a normal and valuable thing to be able to do. The findings are stated with their evidence and without editorialising — the upstream project is public work that was useful enough to build on.

Base model — unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit, governed by the Llama 3.2 Community License. My code here is MIT (LICENSE); any adapter weights derived from Llama 3.2 remain subject to that model's license.