Skip to content

[BUG] Generated reports hard-code version 0.1.0 while CLI is 0.4.9 #291

Description

@mioimotoai-lgtm

Describe the Bug

A JSON report generated by OpenAgent Eval 0.4.9 identifies itself as version 0.1.0 in metadata.version.

The CLI correctly prints OpenAgent Eval v0.4.9, so a downstream archive or comparison tool sees inconsistent producer versions for the same run.

To Reproduce

At revision c8b9452faa27d7ca0a13ce06d311d9bd0fc4a7ee:

  1. Run uv sync --group dev.
  2. Create a two-row JSON dataset with question, ground_truth, context, and ground_truth_contexts fields.
  3. Use the documented offline configuration:
dataset:
  path: data/questions.json
  format: json
llm:
  provider: mock
  model: mock-model
retriever:
  provider: mock
report:
  output: json
  output_dir: reports
parallel: false
  1. Run oaeval run config.yaml.
  2. Inspect the generated JSON report.

Observed CLI output:

OpenAgent Eval v0.4.9
Items: 2 | Errors: 0

Observed report metadata:

{
  "metadata": {
    "engine": "openagent-eval",
    "version": "0.1.0"
  }
}

I also ran the focused offline integration test:

pytest tests/integration/test_pipeline/test_mock_e2e.py -o 'addopts=' -q
3 passed in 0.03s

Expected Behavior

Generated reports should use the installed package version (0.4.9 here), ideally from the same single version source used by the CLI.

Source Location

openagent_eval/core/engine.py currently sets the report metadata value directly:

metadata={
    "version": "0.1.0",
    "engine": "openagent-eval",
    ...
}

openagent_eval/__init__.py already exposes __version__ = "0.4.9".

Environment

  • macOS 26.5.2, arm64
  • Python 3.13.13
  • OpenAgent Eval 0.4.9
  • Source install via uv sync --group dev

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions