SFT → DPO Fine-Tuning Pipeline

Two-phase fine-tuning pipeline that trains Qwen3-8B on structured JSON extraction using QLoRA. Phase 1 (SFT) teaches format consistency. Phase 2 (DPO) corrects the failure modes that SFT alone doesn't fix.

---

What This Is

A pipeline that takes a base LLM and makes it reliably extract structured data from unstructured text. The target task: given messy free-text input (job postings, medical records, product descriptions), extract a defined JSON schema.

The training data is synthetically generated — 3,000 SFT pairs and 1,000 DPO preference triples, covering 8 entity schemas with 15% deliberate refusal examples for unanswerable inputs.

---

Why Fine-Tuning (Not Prompting)

Prompting works for occasional, well-formed inputs. It breaks in three specific ways that matter for production:

1. Schema inconsistency. A prompted base model sometimes uses "phone_number", sometimes "phone", sometimes "phoneNum". Unusable downstream without normalisation.
2. Hallucinated fields. When a field isn't present in the text, the model invents a plausible value rather than returning null or refusing.
3. Prompt sensitivity. Reordering instructions or rephrasing slightly produces different JSON structures. Fine-tuning makes the extraction stable regardless of minor input variation.

RAG doesn't help here — the extraction task is format learning, not knowledge retrieval. Context stuffing doesn't help either: the model already has the relevant text, it just doesn't know the right output format. Fine-tuning is the correct tool.

---

Model: Qwen3-8B

Qwen3-8B was chosen because it's the best-performing open-weight model at 8B parameters on structured output tasks as of April 2026, it runs on a single A100 in 4-bit NF4, and the model weights are publicly available on HuggingFace. The 8B scale is the practical ceiling for QLoRA on accessible hardware — larger models would require multi-GPU setups that change the engineering problem significantly.

---

Training Approach: QLoRA + TRL

QLoRA lets you fine-tune a large model on a single GPU by quantising the base weights to 4-bit NF4 and training only the LoRA adapters in full precision. The base model weights are frozen — you're training roughly 1% of the total parameters.

Config used:

• LoRA rank r=16, alpha=32
• Target modules: q_proj, v_proj, k_proj, o_proj
• 4-bit NF4 quantisation with double quantisation enabled
• 3 epochs SFT, 1 epoch DPO

TRL (Transformer Reinforcement Learning) provides the SFTTrainer and DPOTrainer wrappers. SFTTrainer handles the instruction-formatted dataset and packing. DPOTrainer takes the (chosen, rejected) preference pairs and applies the DPO sigmoid loss with β=0.1.

Why β=0.1: lower β means the model deviates less from the SFT reference policy. Starting conservative avoids reward hacking on the DPO phase.

---

Dataset Design

Synthetically generated — no external dataset or API calls needed.

SFT pairs (3,000 examples): Each example is (instruction, unstructured_text, target_json). The generator creates varied text per entity type, introduces common noise (abbreviations, missing fields, inconsistent formatting), and produces the ground-truth JSON extraction. 15% of examples have no extractable data — the correct output is a structured refusal: {"extractable": false, "reason": "..."}.

DPO preference triples (1,000 examples): Each triple is (prompt, chosen, rejected). The chosen response is the correct extraction. The rejected response is one of four failure modes, sampled proportionally:

• Hallucinated field values (40%)
• Wrong key names (25%)
• Invalid JSON syntax (20%)
• Missed refusal — extraction when none was possible (15%)

8 entity schemas: Person, Company, Product, Job Posting, Medical Event, Financial Transaction, Location, Event. Schema definitions are in src/schemas/.

---

Architecture

┌─────────────────────────────────────────────────────┐
│                  Dataset Generation                  │
│  3,000 SFT pairs + 1,000 DPO preference triples     │
│  8 entity schemas · 15% refusal examples             │
└──────────────────────┬──────────────────────────────┘
                       │
          ┌────────────▼────────────┐
          │   Phase 1: SFT (QLoRA)  │
          │   Qwen3-8B · LoRA r=16  │
          │   4-bit NF4 · 3 epochs  │
          └────────────┬────────────┘
                       │
              ┌────────▼────────┐
              │   Evaluation    │
              │   5 metrics     │
              └────────┬────────┘
                       │
          ┌────────────▼────────────┐
          │   Phase 2: DPO          │
          │   beta=0.1 · LR=5e-6   │
          │   Sigmoid loss          │
          └────────────┬────────────┘
                       │
              ┌────────▼────────┐
              │  Re-Evaluation  │
              │  SFT vs DPO     │
              └─────────────────┘

---

Evaluation Metrics

Metric	What It Measures
JSON Validity Rate	% of outputs that parse as valid JSON
Schema Exact Match	% with all required keys and correct types
Field-Level F1	Per-field precision/recall across all extractions
Refusal Correctness	% of correct refusals on unanswerable prompts
Hallucination Rate	% of outputs with fabricated keys or values

---

What I Learned

DPO doesn't help unless SFT is solid first. The first DPO run made things worse because the SFT model wasn't stable enough — the reference policy wasn't reliable, so the beta constraint was working against good outputs. The fix was running SFT for a full 3 epochs before starting DPO, not 1.

Refusal examples matter disproportionately. Without them, the model extracts from everything — including inputs where nothing extractable exists. Adding 15% refusal examples to SFT dropped the false-extraction rate significantly. The model learned "no data here" as a valid output state.

Schema exact match is a harder metric than it looks. A model that gets all values right but uses "phone_number" instead of "phone" scores 0 on exact match. This is intentional — downstream systems doing record["phone"] will break. But it means the metric punishes minor key name drift harshly, which revealed that the SFT dataset needed more key name variation to make the model robust to paraphrasing.

Local dataset generation beats finding a public dataset for narrow tasks. I spent time looking for structured extraction datasets before writing the generator. The public datasets were either too narrow (one domain) or too noisy (real-world web data with inconsistent labelling). Generating synthetic data with controlled failure modes gave me exactly what the evaluation metrics needed to be meaningful.

---

How To Run It

git clone https://github.com/DOWNEY7/Specialised-Fine-Tuning
cd Specialised-Fine-Tuning
pip install -r requirements.txt

# Generate dataset (no GPU or API keys needed)
python src/generate_dataset.py --validate

# Phase 1: SFT
# Use --dry-run to verify setup without a GPU
python src/train_sft.py --config configs/sft_config.yaml

# Evaluate SFT model
python src/evaluate.py \
  --model outputs/sft_model \
  --output outputs/sft_eval_results.json

# Phase 2: DPO
python src/train_dpo.py --config configs/dpo_config.yaml

# Compare SFT vs DPO
python src/evaluate.py \
  --model outputs/dpo_model \
  --output outputs/dpo_eval_results.json \
  --compare outputs/sft_eval_results.json

Hardware: Default config targets A100 (40GB). For T4 (16GB): set bf16: false, fp16: true, batch size 1, gradient accumulation 16, remove flash_attention_2.

Stack: Qwen3-8B · TRL (SFTTrainer + DPOTrainer) · PEFT + BitsAndBytes 4-bit NF4 · Matplotlib / Seaborn