Skip to content

TaskProtocol.max_retries and TimeoutAction.RETRY/EXTEND are declared but never consumed #62

Description

@cemde

Summary

Two retry-related fields on TaskProtocol are exposed in the public API and documented as functional, but the execution loop never reads them. They are silent no-ops for users.

Affected fields

  1. TaskProtocol.max_retriesmaseval/core/task.py:45-52. Docstring says "Maximum retry attempts for transient failures (not timeouts)." Grepping maseval/core/ shows the only references are the dataclass field itself and its to_dict() serialization. Benchmark._execute_task_repetition (maseval/core/benchmark.py:1069-1255) never reads it.

  2. TimeoutAction.RETRY and TimeoutAction.EXTENDmaseval/core/task.py:21-26. The TaskTimeoutError handler at benchmark.py:1139-1159 records the timeout and returns. It does not branch on task.protocol.timeout_action. So SKIP, RETRY, and EXTEND all behave identically (skip).

Reproduction

```python
from maseval import Task
from maseval.core.task import TaskProtocol, TimeoutAction

task = Task(
query="...",
protocol=TaskProtocol(max_retries=5, timeout_action=TimeoutAction.RETRY, timeout_seconds=1.0),
)

Run a benchmark where this task raises a transient error or times out.

Observed: a single failed report is produced, max_retries / RETRY are ignored.

Expected: up to 5 retries on transient errors; one retry on timeout.

```

Suggested resolution

Either:

  • Wire the fields into `_execute_task_repetition` (retry loop around the execution `try` block; branch on `timeout_action` in the timeout handler), or
  • Mark the fields as "not implemented" in their docstrings and raise on non-default values, so downstream users do not silently rely on them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcoreIn regards to the core package `maseval/core`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions