# VLM Attribute Extraction — Attribute-Grounded Multimodal Fashion Search

A two-phase project building toward open-vocabulary fashion product search using a fine-tuned vision-language model. Phase 1 established a confidence-aware color/pattern extractor on t-shirts. Phase 2 upgrades to a real catalog vocabulary, larger dataset, and open-vocabulary query resolution — directly addressing Flipkart Immerse's stated limitation of "predefined and limited" modification prompts.

---

## What this project does

**Phase 1 (original):** Extract color and pattern from t-shirt images using Qwen2.5-VL-3B-Instruct fine-tuned with LoRA on 435 weakly-labeled samples.

**Phase 2 (this upgrade):** Replace the fixed color list with the catalog's own 46-color vocabulary, fine-tune on 1,545 clean-labeled images across all fashion categories, embed the vocabulary with sentence-transformers, and prove that open-vocabulary query resolution (free text → nearest catalog color) outperforms a fixed-list approach — quantified with precision@5.

---

## Architecture

### Phase 1

HF Fashion Dataset (700 t-shirts) → Weak Label Parsing (regex on titles) → Labeled Split: 180 train / 47 test → Zero-Shot Baseline (Qwen2.5-VL-3B, 4-bit) → LoRA Fine-Tuning (unsloth, 180 samples) → Pseudo-Labeling (469 unlabeled, conf >= 0.95 → 255 accepted) → Expanded LoRA Training (435 samples) → FastAPI Inference Server (/predict, /health)

### Phase 2

Fashion Product Images Dataset (1,792 stratified images, 42 colors, 45 subcategories) → Clean label loading (baseColour field, no regex needed) → Stratified split: 1,545 train / 247 test (by subCategory + baseColour) → Zero-Shot Baseline (Qwen2.5-VL-3B-Instruct, 4-bit, 336px) → LoRA Fine-Tuning (unsloth, lr=2e-5, 1 epoch, 336px) → Fine-Tuned Evaluation (same test split, same metrics) → Attribute Vocabulary Embedding (46 colors, all-MiniLM-L6-v2) → Open-Vocabulary Query Resolution (free text → nearest catalog color) → Retrieval + Precision@5 Evaluation (vs fixed-list baseline) → FastAPI Server (/predict, /resolve, /search, /health) → Web UI (interface.html)

---

## Results

### Phase 1 — T-Shirt Color + Pattern Extraction

| Metric      | Zero-Shot | LoRA-180 | LoRA-435 |
|-------------|-----------|----------|----------|
| Color Acc   | 66.1%     | 66.0%    | **68.1%** |
| Pattern Acc | 53.7%     | 89.4%    | **89.4%** |
| Color F1    | 0.514     | 0.439    | 0.449    |
| Pattern F1  | 0.562     | 0.822    | **0.822** |
| Confidence  | 0.840     | 0.953    | **0.965** |

### Phase 2 — Multi-Category Color Extraction (247 test images, 42 classes)

| Metric          | Zero-Shot | LoRA (1 epoch) | Delta   |
|-----------------|-----------|----------------|---------|
| Color Accuracy  | 60.16%    | 59.35%         | -0.81%  |
| Color F1 Macro  | 0.3307    | **0.3444**     | +0.0137 |
| Avg Confidence  | 0.9990    | 1.0000         | +0.001  |
| Parse Fails     | 1/247     | 1/247          | —       |

> Macro F1 improvement reflects better coverage of minority color classes (Maroon, Magenta, Green, Lavender). Accuracy regression is within 2-sample margin on 247 images.

### Phase 2 — Open-Vocabulary Resolution vs Fixed-List (15 synonym queries)

| Metric                  | Open-Vocab | Fixed-List | Delta    |
|-------------------------|------------|------------|----------|
| Query Coverage          | **100%**   | 0%         | +100%    |
| Mean Precision@5        | **0.827**  | 0.000      | +0.827   |
| OOV Synonym Accuracy    | 67%        | 0%         | +67%     |

> Fixed-list baseline accepts only exact catalog color name matches. None of the 15 natural language queries ("mustard yellow", "burgundy red", "off-white" etc.) matched exactly. Open-vocab resolved all 15.

### Phase 2 — Top Confusion Pairs (LoRA)

| Ground Truth | Predicted    | Count |
|-------------|--------------|-------|
| Grey        | Grey Melange | 10    |
| Grey        | Charcoal     | 4     |
| Black       | Red          | 3     |
| Navy Blue   | Blue         | 3     |
| Red         | Pink         | 3     |

---

## Stack

| Component      | Detail                                      |
|----------------|---------------------------------------------|
| Model          | Qwen2.5-VL-3B-Instruct                      |
| Fine-tuning    | LoRA via unsloth (r=16, lr=2e-5, 1 epoch)   |
| Embeddings     | all-MiniLM-L6-v2 (sentence-transformers)    |
| Dataset P1     | ashraq/fashion-product-images-small (HF)    |
| Dataset P2     | Fashion Product Images (Kaggle, 44k items)  |
| Training       | Google Colab T4 (free tier)                 |
| Serving        | FastAPI + uvicorn                           |
| UI             | Vanilla HTML/JS (dark theme)                |

---

## Project Structure

vlm_attribute_extraction/ ├── data/ │ ├── color_vocab.json # 46-color catalog vocabulary │ ├── color_embeddings.npy # Sentence-transformer embeddings (46x384) │ ├── baseline_metrics.json # Phase 2 zero-shot metrics │ ├── baseline_predictions.csv # Phase 2 zero-shot predictions │ ├── lora_metrics.json # Phase 2 LoRA metrics │ ├── lora_predictions.csv # Phase 2 LoRA predictions │ ├── retrieval_metrics.json # Precision@5 results │ ├── retrieval_results.csv # Per-query retrieval results │ ├── resolver_results.json # OOV synonym resolution results │ ├── labeled_test.json # Phase 1 test labels │ ├── lora_expanded_test_results.json # Phase 1 LoRA-435 predictions │ ├── zero_shot_results.json # Phase 1 zero-shot predictions │ └── test_images/ # Phase 1 test images ├── models/ │ ├── lora_adapter_v2/ # Phase 2 LoRA adapter (336px, lr=2e-5) │ │ ├── adapter_config.json │ │ ├── adapter_model.safetensors │ │ └── tokenizer files... │ └── (adapter_config.json etc.) # Phase 1 LoRA-435 adapter (flat) ├── scripts/ │ ├── serve.py # Phase 1 FastAPI server │ ├── serve_v2.py # Phase 2 FastAPI server (/resolve, /search) │ ├── interface.html # Web UI (open via localhost:8000) │ ├── predict.py # Phase 1 single image inference │ ├── visualize_results.py # Phase 1 visual proof grid │ └── download_test_images.py # Phase 1 image downloader ├── notebooks/ # Colab training notebooks ├── logs/ │ └── visual_proof.png ├── DATA.md ├── EXPERIMENTS.md ├── requirements.txt └── README.md

---

## Setup

```bash
python -m venv venv
venv\Scripts\activate
pip install transformers accelerate peft pillow datasets fastapi uvicorn
pip install python-multipart requests sentence-transformers aiofiles

---

Usage

Phase 2 Server (Resolve + Search)

# Start server
cd scripts
python serve_v2.py

# Open UI
# Navigate to http://localhost:8000 in browser

# Health check
Invoke-WebRequest -Uri http://localhost:8000/health -UseBasicParsing | Select-Object -ExpandProperty Content

# Resolve a free-text color query
Invoke-WebRequest -Uri http://localhost:8000/resolve `
  -Method POST -ContentType "application/json" `
  -Body '{"query": "mustard yellow", "top_k": 3}' `
  -UseBasicParsing | Select-Object -ExpandProperty Content

# Expected: {"query":"mustard yellow","resolved":"Mustard","top_k":[["Mustard",0.8094],...]}

Phase 1 Server (Predict)

python -m uvicorn scripts.serve:app --host 0.0.0.0 --port 8000

# Predict color + pattern from image
python -c "import requests; f=open('data/test_images/18237.jpg','rb'); r=requests.post('http://localhost:8000/predict',files={'file':('18237.jpg',f,'image/jpeg')}); print(r.json())"

---

API Reference (Phase 2)

Endpoint	Method	Description
/health	GET	Server status, vocab size
/resolve	POST	Free-text query → nearest catalog color
/search	POST	Image + text query → catalog matches
/predict	POST	Color from image (requires GPU runtime)

---

Adapters

Adapter	Location	Training	Size
Phase 1 LoRA-435	models/ (flat)	435 samples, t-shirts only	~114MB
Phase 2 LoRA-v2	models/lora_adapter_v2/	1,545 samples, all categories	~114MB

Base model: Qwen/Qwen2.5-VL-3B-Instruct (auto-downloaded from HuggingFace, ~7.5GB). GPU required for /predict. /resolve and /search run on CPU.

---

Key Design Decisions

• No pattern attribute in Phase 2 — not present in the Kaggle dataset's structured fields; color-only this iteration by decision, not oversight.
• Clean labels over weak labels — Phase 2 uses baseColour field directly (99.97% populated), replacing Phase 1's regex parsing on titles.
• Fixed-list comparison is simulated — the fixed-list baseline accepts only exact catalog color name matches in the query string, replicating Flipkart Immerse's stated limitation.
• Image resize to 336px — all images resized before inference to avoid CUDA device-side assert on high-resolution inputs (1080×1440, 1800×2400).
• LoRA language model only — vision encoder frozen; color is a language-grounded output task.