Skip to content

Create eval suite with test fixtures and okf:test skill #9

Description

@siculo

Problem

No fixtures, runner, or CI gate exist. Skills are actively iterated but regressions are discovered only during live sessions.

Proposed Change

Directory structure: evals/

evals/
├── README.md
├── okf-create/
│   ├── 01-single-source/
│   │   ├── input.md                   ← sample source document
│   │   ├── expected-concepts.yaml     ← expected list: path, type, title
│   │   └── expected-log-entry.yaml    ← expected log.md entry
│   └── 02-multi-source/
│       ├── input-a.md
│       ├── input-b.md
│       └── expected-concepts.yaml
├── okf-update/
│   ├── 01-new-concept/
│   │   ├── bundle-snapshot/
│   │   ├── input.md
│   │   └── expected-changes.yaml      ← NEW: 1, UPDATES: 0, SPLITS: 0, CONFLICTS: 0
│   └── 02-split/
│       ├── bundle-snapshot/
│       ├── input.md
│       └── expected-changes.yaml
├── okf-reconcile/
│   ├── 01-stale-detection/
│   │   ├── bundle-snapshot/
│   │   ├── sources/
│   │   └── expected-report.yaml
│   └── 02-new-in-sources/
│       ├── bundle-snapshot/
│       ├── sources/
│       └── expected-report.yaml
├── okf-validate/
│   ├── 01-missing-type/
│   │   ├── bundle-snapshot/
│   │   └── expected-errors.yaml       ← ERRORS: 1, WARNINGS: 0
│   └── 02-conformant/
│       ├── bundle-snapshot/
│       └── expected-errors.yaml       ← ERRORS: 0, WARNINGS: 0
└── okf-search/
    ├── 01-full-text/
    │   ├── bundle-snapshot/
    │   └── expected-results.yaml      ← rank 1: path, title
    └── 02-type-filter/
        ├── bundle-snapshot/
        └── expected-results.yaml

New skill: skills/test/SKILL.md

A new okf:test skill that:

  1. Parses optional arguments skill:<name> and evals:<path>.
  2. Discovers all fixtures under the evals directory.
  3. Runs each fixture against the skill's classification/proposal logic on a temporary copy — never writes to the real bundle.
  4. Reports pass/fail with full diffs for failures.

Example output:

OKF Skill Eval
─────────────────────────────────────────
Fixtures run : 10
Passed       : 9
Failed       : 1

FAILED
  ✗ okf-update / 02-split
      expected: SPLITS: 1
      actual:   SPLITS: 0
      note: the skill classified the concept as UPDATE instead of SPLIT

On failure, the skill prints the full diff and does not auto-fix anything, since failures indicate a regression in the skill logic that requires manual review.

Files Affected

  • New directory: evals/ with fixture structure above
  • New file: skills/test/SKILL.md

Estimated Effort

~4–6 hours to create initial fixtures; ~1 hour for the test skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions