Log-Scale Arithmetic Model Evaluation

An experiment for measuring how accurately an OpenAI model performs arithmetic across values spanning many orders of magnitude.

The project generates log-uniform and log-Gaussian number distributions, creates balanced arithmetic questions, sends those questions to a selected model, compares answers at the precision returned by the model, and displays the results in a web dashboard.

View the private results dashboard

What the project includes

• Reproducible log-uniform and log-Gaussian datasets
• Evenly balanced addition, subtraction, multiplication, and division
• Configurable calculation and evaluation counts
• Configurable OpenAI model selection
• Significant-digit-aware answer comparison
• Scatter plots, two-sigma regions, and binned heatmaps
• An HTML dashboard with JSON loading, filtering, and CSV export

Workflow

flowchart LR
    A[Generate distributions] --> B[Generate calculations]
    B --> C[Query OpenAI model]
    C --> D[Compare returned precision]
    D --> E[Save JSON report]
    E --> F[Review in dashboard]

Loading

Project structure

.
├── data/                         # Distribution and calculation JSON
├── outputs/                      # Generated plot images
├── styles/                       # Light and dark Matplotlib themes
├── dashboard/                    # Results web application
├── log_distribution.py          # Generate source distributions
├── generate_calculations.py      # Create balanced arithmetic questions
├── tool_use_test.py              # Evaluate model answers and write a report
├── openai_call.py                # Reusable Responses API helper
├── tool_call_example.py          # Function-calling calculator example
├── plot_log_chart.py             # Symmetric-log sample chart
├── plot_scientific_distribution.py
├── plot_distribution_heatmaps.py
├── project_paths.py              # Shared project directories
└── requirements.txt

Requirements

• Python 3.10 or newer; Python 3.12 is recommended
• Node.js 22 or newer for the dashboard
• An OpenAI API key for live evaluation

Python setup

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Set OPENAI_API_KEY as a Windows user environment variable, then open a new terminal. Never commit the key or paste it into source code.

Run the experiment

Generate 100 values for each distribution:

python log_distribution.py

Generate a chosen number of arithmetic questions per distribution:

python generate_calculations.py --number 100

Evaluate questions from the log-uniform calculation set:

python tool_use_test.py --number 20 --model gpt-4.1-mini

The evaluation writes data/test_results.json. The report contains the model, prompt, raw output, detected significant digits, expected value at the same precision, full expected value, and pass/fail status.

API calls may incur usage charges. Start with a small --number value.

Generate the plots

python plot_log_chart.py
python plot_scientific_distribution.py
python plot_distribution_heatmaps.py

Plots are written to outputs/.

Scientific-notation distribution

Binned distribution heatmaps

Run the dashboard locally

cd dashboard
npm ci
npm run dev

Open the local URL printed by the development server and select Load JSON report to load data/test_results.json.

Precision comparison

Model answers are compared using the precision encoded in the returned text. For example:

Model output:     -3.39e-05
Detected:         3 significant digits
Full expected:    -3.3901515151515155e-05
Rounded expected: -3.39e-05
Result:           PASS

Trailing zeroes are preserved as meaningful precision, so 3.390 is treated as four significant digits.

GitHub preparation

Generated virtual environments, caches, dashboard dependencies, private evaluation reports, and local Sites history are excluded through .gitignore. The sample datasets, plots, dashboard source, and example dashboard report are safe to include in the repository.

Before publishing, review the staged files and confirm that no API keys or private reports are present.