Temporal LLM Baseline

A minimal starter project for fine-tuning an instruction-tuned language model with temporal metadata.

pip install -r requirements.txt

Generate a tiny demo dataset

We can generate a small JSONL dataset to smoke-test the pipeline:

python src/main.py --write_demo_data data/data_sample.jsonl

This creates examples with temporal metadata such as date, year, decade, newspaper, language, instruction, input text, and structured output.

Training

Minimal LoRA fine-tuning experiment:

python src/main.py \
  --data_path data/data_sample.jsonl \
  --model_name Qwen/Qwen2.5-3B-Instruct \
  --output_dir outputs/temporal-llm-lora \
  --num_train_epochs 1 \
  --bf16

Input data format

Each training example is a JSONL:

{
  "id": "GDL-1912-001",
  "date": "1912-05-03",
  "year": 1912,
  "decade": "1910s",
  "century": "20C",
  "newspaper": "Gazette de Lausanne",
  "language": "fr",
  "source_type": "historical newspaper",
  "instruction": "Link the entity mention according to the document date.",
  "mention": "M. Poincaré",
  "input_text": "M. Poincaré a déclaré aujourd'hui que la situation politique exigeait de la prudence.",
  "output": {
    "surface": "M. Poincaré",
    "type": "person",
    "qid": "Q191974",
    "temporal_grounding": "The document is dated 1912, when Raymond Poincaré was a central French political figure."
  }
}

The data.py module converts this raw example into a chat-style supervised fine-tuning example:

system: You are a temporal language model for historical documents...
user: <YEAR_1912> <DECADE_1910s> <LANG_fr> + metadata block + task + text
assistant: structured target output

Temporal metadata prompting

The baseline supports two complementary temporal conditioning strategies.

First, it adds a natural-language metadata block:

Temporal metadata:
- Document date: 1912-05-03
- Year: 1912
- Decade: 1910s
- Century: 20C
- Newspaper: Gazette de Lausanne
- Language: fr
- Source type: historical newspaper

Second, it will prepend (different) temporal tokens:

<YEAR_1912> <DECADE_1910s> <CENTURY_20C> <LANG_fr>

This lets us compare whether explicit temporal metadata, compact temporal tokens, or both improve performance (one by one).