PhyBench Pipeline

A comprehensive LLM performance evaluation pipeline that generates model solutions and evaluates them using expression edit distance (EED) metrics.

Overview

PhyBench Pipeline consists of two main components:

• API Caller: Generates solutions to physics problems using large language model (LLM) API
• Evaluation: Scores model solutions against ground truth using symbolic expression comparison

Features

• Parallel Processing: Multi-threaded API calls and CPU-optimized evaluation
• Expression Distance: Non-binary LaTeX expression comparison using EED metrics
• Flexible Configuration: TOML-based config with CLI override support

Quick Start

Environment Setup

Method 1: uv (Recommended)

Step 1: Install uv

Choose one installation method based on your platform:

# Option 1: Using pip (cross-platform)
pip install uv

# Option 2: Linux/Mac using curl
curl -LsSf https://astral.sh/uv/install.sh | sh

# Option 3: Mac using Homebrew
brew install uv

# Option 4: Windows using PowerShell
irm https://astral.sh/uv/install.ps1 | iex

Step 2: Set up environment

# Create virtual environment and install dependencies
uv sync

# Activate environment (Windows: .venv\Scripts\activate)
source .venv/bin/activate

Method 2: pip

# Create virtual environment
python -m venv .venv

# Activate environment (Windows: .venv\Scripts\activate)
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Configuration

1. Copy and edit the configuration file:

# Copy the sample configuration
cp config.toml.sample config.toml

# Edit config.toml with your API key and settings

2. Key configuration sections:

• [[providers]]: API provider settings (name, base_url, api_key, models)
• [api_caller.model]: Model settings (model, openai_o_model_keywords)
• [api_caller.paths]: Input/output paths (input_file, output_dir)
• [api_caller.execution]: Processing settings (num_consumers, chat_timeout, repeat_count, max_retries)
• [evaluation.paths]: Ground truth and model answer files (gt_file, model_answers_file, output_file, log_file)
• [evaluation.eed]: Parameters for the Expression Edit Distance algorithm
• [evaluation.execution]: Processing settings (num_processes, skip_problem_ids)
• [logging]: Logging settings (log_dir, log_file, console_level, file_level)

Usage

Note: All commands should be run from the root of the project directory. You can either activate the virtual environment first (source .venv/bin/activate or \.venv\Scripts\activate on Windows) or prefix each command with uv run.

Generate Model Solutions

# Generate solutions using configured settings
python -m phybench.api_caller

# Custom parameters
python -m phybench.api_caller --model gpt-4o --input-file data/ground_truth/test.json --output-dir data/model_solutions --repeat-count 3

Evaluate Solutions

# Evaluate using configured settings
python -m phybench.evaluation

# Custom evaluation with specific parameters
python -m phybench.evaluation --gt-file data/ground_truth/test.json --model-answers-file data/model_solutions/gpt-4o.json --output-dir data/evaluation_results/custom_results.json --initial-score 60 --scoring-slope 100 --log-file logs/custom_evaluation.log --num-processes 4

Testing

# Run all tests
uv run pytest

Data Format

API Caller Input (Ground Truth)

[
  {
    "id": 495,
    "tag": "OPTICS",
    "content": "Physics problem statement with LaTeX equations...",
  }
]

API Caller Output (Model Solutions)

[
  {
    "id": 495,
    "model": "gpt-4o",
    "model_solution": "Complete solution text generated by the model...",
    "model_answer": "\\frac{NZ e^3}{2c\\varepsilon_0 m^2 n} \\cdot \\frac{\\omega^2}{(\\omega_0^2 - \\omega^2)^2}",
    "timestamp": 1736156789.123,
    "time_taken": 12.5,
    "repeat_index": 0,
    "prompt_tokens": 850,
    "completion_tokens": 420,
    "total_tokens": 1270
  }
]

Evaluation Output

[
  {
    "id": 495,
    "tag": "OPTICS",
    "content": "Physics problem statement...",
    "answer": "\\frac{NZ e^3}{2c\\varepsilon_0 m^2 n} \\cdot \\frac{\\omega^2}{(\\omega_0^2 - \\omega^2)^2}",
    "model_name": ["gpt-4o"],
    "model_score": [85.2],
    "model_answer": ["\\frac{NZ e^3}{2c\\varepsilon_0 m^2 n} \\cdot \\frac{\\omega^2}{(\\omega_0^2 - \\omega^2)^2}"],
    "model_distance": [0.15],
    "model_score_var": 0.0,
    "answer_size": 12
  }
]

Scoring System

The evaluation uses Expression Edit Distance (EED). The scoring behavior is highly configurable via the [evaluation.eed] section of your config.toml file.

• initial_score: Base score before any distance penalty is applied (e.g., 100.0).
• scoring_slope: A multiplier for the distance penalty. Higher values mean a steeper penalty curve.
• insert_cost, delete_cost, update_cost: Dictionaries defining the cost for tree operations (insertion, deletion, update) on different node types (number, symbol, operator, function).
• change_type_cost: The cost for updating a node to a different type (e.g., number to symbol).
• simplify_time_limit, equals_time_limit: Timeouts in seconds for symbolic simplification and equality checks to prevent excessively long computations.

The final score is calculated as: max(0, initial_score - scoring_slope * (distance / tree_size))

Development

Requirements

• Python ≥ 3.13
• Dependencies: See pyproject.toml

Development Setup

Code Quality Tools

The project uses Ruff for linting and formatting, mypy for basic type checking, with pre-commit hooks for automatic checks:

# Install development dependencies (includes Ruff and mypy)
uv sync

# Install pre-commit hooks
pre-commit install

# Run linting manually
uv run ruff check phybench/

# Run formatting manually
uv run ruff format phybench/

# Run type checking manually (lenient mode for existing codebase)
uv run mypy phybench/

# Run both linting and formatting on all files
pre-commit run --all-files

Pre-commit Hooks

Pre-commit hooks automatically run on each commit and include:

• Ruff Linter: Checks code style, imports, and common issues
• Ruff Formatter: Ensures consistent code formatting
• mypy: Static type checking

Configuration files:

• .pre-commit-config.yaml: Pre-commit hook configuration
• pyproject.toml: Ruff configuration under [tool.ruff]
• mypy.ini: mypy configuration

Configuration Management

• Copy config.toml.sample to config.toml and edit your settings
• All CLI arguments can override config values

License

This project follows standard academic research practices. Please cite appropriately if used in research.