Describe the Bug
A JSON report generated by OpenAgent Eval 0.4.9 identifies itself as version 0.1.0 in metadata.version.
The CLI correctly prints OpenAgent Eval v0.4.9, so a downstream archive or comparison tool sees inconsistent producer versions for the same run.
To Reproduce
At revision c8b9452faa27d7ca0a13ce06d311d9bd0fc4a7ee:
- Run
uv sync --group dev.
- Create a two-row JSON dataset with
question, ground_truth, context, and ground_truth_contexts fields.
- Use the documented offline configuration:
dataset:
path: data/questions.json
format: json
llm:
provider: mock
model: mock-model
retriever:
provider: mock
report:
output: json
output_dir: reports
parallel: false
- Run
oaeval run config.yaml.
- Inspect the generated JSON report.
Observed CLI output:
OpenAgent Eval v0.4.9
Items: 2 | Errors: 0
Observed report metadata:
{
"metadata": {
"engine": "openagent-eval",
"version": "0.1.0"
}
}
I also ran the focused offline integration test:
pytest tests/integration/test_pipeline/test_mock_e2e.py -o 'addopts=' -q
3 passed in 0.03s
Expected Behavior
Generated reports should use the installed package version (0.4.9 here), ideally from the same single version source used by the CLI.
Source Location
openagent_eval/core/engine.py currently sets the report metadata value directly:
metadata={
"version": "0.1.0",
"engine": "openagent-eval",
...
}
openagent_eval/__init__.py already exposes __version__ = "0.4.9".
Environment
- macOS 26.5.2, arm64
- Python 3.13.13
- OpenAgent Eval 0.4.9
- Source install via
uv sync --group dev
Describe the Bug
A JSON report generated by OpenAgent Eval 0.4.9 identifies itself as version
0.1.0inmetadata.version.The CLI correctly prints
OpenAgent Eval v0.4.9, so a downstream archive or comparison tool sees inconsistent producer versions for the same run.To Reproduce
At revision
c8b9452faa27d7ca0a13ce06d311d9bd0fc4a7ee:uv sync --group dev.question,ground_truth,context, andground_truth_contextsfields.oaeval run config.yaml.Observed CLI output:
Observed report metadata:
{ "metadata": { "engine": "openagent-eval", "version": "0.1.0" } }I also ran the focused offline integration test:
Expected Behavior
Generated reports should use the installed package version (
0.4.9here), ideally from the same single version source used by the CLI.Source Location
openagent_eval/core/engine.pycurrently sets the report metadata value directly:openagent_eval/__init__.pyalready exposes__version__ = "0.4.9".Environment
uv sync --group dev