Problem
No fixtures, runner, or CI gate exist. Skills are actively iterated but regressions are discovered only during live sessions.
Proposed Change
Directory structure: evals/
evals/
├── README.md
├── okf-create/
│ ├── 01-single-source/
│ │ ├── input.md ← sample source document
│ │ ├── expected-concepts.yaml ← expected list: path, type, title
│ │ └── expected-log-entry.yaml ← expected log.md entry
│ └── 02-multi-source/
│ ├── input-a.md
│ ├── input-b.md
│ └── expected-concepts.yaml
├── okf-update/
│ ├── 01-new-concept/
│ │ ├── bundle-snapshot/
│ │ ├── input.md
│ │ └── expected-changes.yaml ← NEW: 1, UPDATES: 0, SPLITS: 0, CONFLICTS: 0
│ └── 02-split/
│ ├── bundle-snapshot/
│ ├── input.md
│ └── expected-changes.yaml
├── okf-reconcile/
│ ├── 01-stale-detection/
│ │ ├── bundle-snapshot/
│ │ ├── sources/
│ │ └── expected-report.yaml
│ └── 02-new-in-sources/
│ ├── bundle-snapshot/
│ ├── sources/
│ └── expected-report.yaml
├── okf-validate/
│ ├── 01-missing-type/
│ │ ├── bundle-snapshot/
│ │ └── expected-errors.yaml ← ERRORS: 1, WARNINGS: 0
│ └── 02-conformant/
│ ├── bundle-snapshot/
│ └── expected-errors.yaml ← ERRORS: 0, WARNINGS: 0
└── okf-search/
├── 01-full-text/
│ ├── bundle-snapshot/
│ └── expected-results.yaml ← rank 1: path, title
└── 02-type-filter/
├── bundle-snapshot/
└── expected-results.yaml
New skill: skills/test/SKILL.md
A new okf:test skill that:
- Parses optional arguments
skill:<name> and evals:<path>.
- Discovers all fixtures under the evals directory.
- Runs each fixture against the skill's classification/proposal logic on a temporary copy — never writes to the real bundle.
- Reports pass/fail with full diffs for failures.
Example output:
OKF Skill Eval
─────────────────────────────────────────
Fixtures run : 10
Passed : 9
Failed : 1
FAILED
✗ okf-update / 02-split
expected: SPLITS: 1
actual: SPLITS: 0
note: the skill classified the concept as UPDATE instead of SPLIT
On failure, the skill prints the full diff and does not auto-fix anything, since failures indicate a regression in the skill logic that requires manual review.
Files Affected
- New directory:
evals/ with fixture structure above
- New file:
skills/test/SKILL.md
Estimated Effort
~4–6 hours to create initial fixtures; ~1 hour for the test skill.
Problem
No fixtures, runner, or CI gate exist. Skills are actively iterated but regressions are discovered only during live sessions.
Proposed Change
Directory structure:
evals/New skill:
skills/test/SKILL.mdA new
okf:testskill that:skill:<name>andevals:<path>.Example output:
On failure, the skill prints the full diff and does not auto-fix anything, since failures indicate a regression in the skill logic that requires manual review.
Files Affected
evals/with fixture structure aboveskills/test/SKILL.mdEstimated Effort
~4–6 hours to create initial fixtures; ~1 hour for the test skill.