Skip to content

v3 Slice 1: Scaffold + spine - package, registries, tagged-text extractor, status taxonomy #14

Description

@mark-allwyn

Parent

#13

What to build

The foundational skeleton every other slice imports. Not a full vertical slice on its own - it stands up the package, the extension points, and the shared primitives so the tracer bullet stays thin.

  • A new benchpress package at the repo root (the v2 system stays under legacy/), with a CLI stub exposing the command surface (generate, run, score, stats, export, audit) - subcommands may be stubs that exit cleanly.
  • Core data types: Item, Part, PartResult, ItemResult, ModuleMeta (plain serializable dataclasses matching the PRD shapes).
  • The three registries that provide extensibility (generalizing the legacy CHECKERS dict): a module registry, a part-scorer registry keyed by part type, and a metric registry - each with a registration decorator, populated by import side-effect.
  • The tagged-text extractor: parse a model response into a dict of label -> raw field, tolerant (case-insensitive labels, markdown-tolerant, last occurrence wins, extra prose ignored).
  • The status-taxonomy classifier: map (stop_reason, stop_details, extraction outcome) -> one of ok | refusal | invalid_answer | truncated | api_error.
  • Root config model registry reused from the existing config.yaml shape (provider, model, company, type, launch_date, api_key_env, params).

Acceptance criteria

  • benchpress imports cleanly; pytest exits 0 at the repo root (a trivial placeholder test is fine)
  • Registering a module / part-scorer / metric makes it discoverable via its registry
  • Tagged-text extractor unit-tested: case-insensitive labels, markdown fences, repeated labels (last wins), missing label -> absent key
  • Status classifier unit-tested across all five statuses, including the refusal stop_reason path
  • No secrets committed (.env stays gitignored)

Blocked by

None - can start immediately

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentReady for autonomous agent pickup

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions