Summary
The current eval/score.py has a no-op syntax_check that runs true (always passes) and only two eval dimensions. This inflates the composite score and leaves the factory blind to capability surface and research grounding.
Changes
- Replace
eval_syntax_check: use ast.parse on .py files in scripts/ and bash -n on .sh files
- Add
eval_capability_surface: count slash commands in .claude/commands/, skills with SKILL.md in skills/, and templates in Templates/. Score = count / 50 (capped at 1.0)
- Add
eval_research_grounding: count archive entries in .factory/archive/. Score = entries / 10 (capped at 1.0)
- Update weights: syntax_check=0.20, observability=0.20, capability_surface=0.30, research_grounding=0.30
- Preserve existing
eval_observability function exactly as-is
From H1 in .factory/strategy/current.md.
Summary
The current
eval/score.pyhas a no-opsyntax_checkthat runstrue(always passes) and only two eval dimensions. This inflates the composite score and leaves the factory blind to capability surface and research grounding.Changes
eval_syntax_check: useast.parseon.pyfiles inscripts/andbash -non.shfileseval_capability_surface: count slash commands in.claude/commands/, skills withSKILL.mdinskills/, and templates inTemplates/. Score = count / 50 (capped at 1.0)eval_research_grounding: count archive entries in.factory/archive/. Score = entries / 10 (capped at 1.0)eval_observabilityfunction exactly as-isFrom H1 in
.factory/strategy/current.md.