Skip to content

Replace no-op syntax_check eval with real validation, add capability_surface and research_grounding #4

Description

@lukeinglis

Summary

The current eval/score.py has a no-op syntax_check that runs true (always passes) and only two eval dimensions. This inflates the composite score and leaves the factory blind to capability surface and research grounding.

Changes

  • Replace eval_syntax_check: use ast.parse on .py files in scripts/ and bash -n on .sh files
  • Add eval_capability_surface: count slash commands in .claude/commands/, skills with SKILL.md in skills/, and templates in Templates/. Score = count / 50 (capped at 1.0)
  • Add eval_research_grounding: count archive entries in .factory/archive/. Score = entries / 10 (capped at 1.0)
  • Update weights: syntax_check=0.20, observability=0.20, capability_surface=0.30, research_grounding=0.30
  • Preserve existing eval_observability function exactly as-is

From H1 in .factory/strategy/current.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions