[Feature] Evaluation metrics for synthesized program quality — how do we know KAPSO's solutions are actually better?

[Ruthwik-Data]

Background

I'm exploring KAPSO as a framework for autonomous ML/AI program synthesis. The knowledge-grounded approach is compelling, but I've run into an evaluation gap that I think is fundamental to the framework:

Once KAPSO synthesizes a program/solution, how do we assess whether it's actually better than a baseline — and in what dimensions?

The Problem

Right now the evaluation story seems to be: run it, see if it works, compare loss/metric on the target task. But that's a black-box eval that misses several important quality dimensions for synthesized programs:

1. Knowledge grounding quality: Did the synthesis actually use the retrieved knowledge correctly, or did it hallucinate a pattern that looks valid but isn't grounded in the retrieved papers/examples?

2. Generalization vs. overfitting to search: Did the synthesized program improve because it's genuinely better, or because it was optimized for the specific eval queries used during synthesis?

3. Compositional correctness: For synthesized programs that compose known techniques, did it compose them in a way that's theoretically sound (e.g., compatible assumptions, correct ordering of operations)?

4. Efficiency-accuracy tradeoff tracking: Often a synthesized program gets better accuracy at the cost of 10x compute. There's no structured way to surface this tradeoff alongside the quality score.

Proposed Solution

Add an EvalReport struct to synthesis outputs that surfaces these dimensions:

@dataclass
class SynthesisEvalReport:
    task_id: str
    baseline_score: float
    synthesized_score: float
    improvement_delta: float
    
    # New fields
    grounding_score: float       # 0-1: how well synthesis used retrieved knowledge
    generalization_score: float  # held-out query set score vs. synthesis query set score
    composition_validity: bool   # did it compose techniques with compatible assumptions?
    compute_ratio: float         # synthesized_flops / baseline_flops
    
    flagged_issues: list[str]    # e.g. ["grounding_gap", "efficiency_regression"]

This would make it much easier to:

• Catch regressions where synthesis finds a locally optimal but globally fragile program
• Trust the synthesized program enough to deploy it vs. keep iterating
• Compare synthesis runs across different knowledge bases

Why Now

As KAPSO moves toward production use cases (see issue #74 about Hugging Face release), users will need to trust that synthesized programs are genuinely better — not just better on the synthesis task. An evaluation layer at synthesis time is the right point to add this signal.

Happy to contribute a PR with a basic EvalReport implementation and a worked example on one of the existing KAPSO benchmarks if the team is interested.