Add case-based chord progression evaluations and diagnostic analysis

[laceyp99]

Summary

Refocus Conductor Eval's growing harmonic-evaluation workflow around named, structured chord-progression cases and expand the analysis dashboard so it explains chord performance across models and reasoning configurations.

This follows the enhancement noted in the review on PR #29. The existing deterministic chord checks are a useful foundation, but Evaluator.evaluate() currently applies one shared test_params mapping to every prompt, and the dashboard exposes only all-or-nothing pass rates for chord checks.

The initial scope is deliberately limited to explicitly requested sustained block chords. More general chord-window analysis for arpeggiation, rhythmic comping, anticipations, and rearticulation should remain a documented follow-up.

Goals

1. Package each prompt with the deterministic test parameters that define its expected progression.
2. Allow each case to restrict the scale modes against which it runs.
3. Preserve the existing prompts plus shared test_params evaluation interface.
4. Keep strict full-case pass rate as the headline harmonic result.
5. Add diagnostic chord accuracy and error visualizations that reveal differences among models, reasoning modes, and progression cases.
6. Provide a small, safe local-model chord-progression example without changing the existing scripts/test_eval.py.

Proposed case interface

Add a structured cases input to Evaluator.evaluate(). A case packages a stable name, prompt, optional scales, and its test parameters:

cases=[
    {
        "name": "major_1564",
        "prompt": "Four sustained block chords following I-V-vi-IV, one chord per bar",
        "scales": ["major"],
        "test_params": {
            "polyphony": {"min_voices": 3},
            "chord_progression": {
                "progression": ["I", "V", "vi", "IV"],
                "beats_per_chord": 4,
                "strict": True,
            },
            "harmonic_rhythm": {"expected_onsets": [0, 4, 8, 12]},
            "chord_event_positions": {
                "expected_starts": [0, 4, 8, 12],
                "expected_ends": [4, 8, 12, 16],
            },
        },
    },
]

Interface behavior

• A caller supplies either cases or the existing prompts input, not both.
• Existing prompts, tests, and shared test_params behavior remains supported and unchanged.
• case["scales"] is optional. When omitted, it defaults to the evaluator's existing major/minor scale set.
• Explicit case scales must be validated against the evaluator's supported scales.
• A case's selected checks should be derived from or validated against its test_params; required scale checking remains enforced.
• Case names must be non-empty and unique within a run so they can serve as stable analysis dimensions.
• Invalid or incomplete cases should fail before a run directory is created or any provider request begins.
• Case configuration must be defensively copied so task generation cannot mutate caller input or leak parameters between cases.

A typed internal representation (for example, a small dataclass or validation helper) is preferred over passing loosely validated dictionaries throughout the evaluator. Avoid introducing a generic suite/config framework beyond what these cases require.

Task routing and persistence

Task generation should expand each case only across the case's selected scales, the run's selected roots, selected models, and applicable reasoning variations.

Every task and persisted result should carry at least:

• case_name;
• the original case prompt;
• selected scale;
• resolved case-level test parameters;
• progression metadata needed by analysis.

Include case identity and all resolved case inputs in the task fingerprint so distinct cases cannot share artifact identity. Persist case information explicitly; analysis must not infer it from directory names or prompt text. Run configuration should retain normalized cases so a result set remains reproducible and inspectable.

Strict headline scoring

Keep strict full-case pass rate as the primary harmonic benchmark:

• A case passes only when every selected eligible deterministic check passes.
• A progression with three correct chords out of four remains a failed case.
• Generation errors, check errors, and ineligible results remain distinguishable from validation failures and excluded from denominators according to existing eligibility semantics.
• Existing overall pass-rate reporting continues to work for non-chord evaluations.
• The numerator and eligible denominator are visible wherever a rate is presented.

Chord diagnostic metrics

Under the strict headline result, calculate and expose diagnostics from existing per-bar chord results:

• correct chord positions / total evaluated chord positions;
• chord accuracy by model;
• chord accuracy by reasoning mode or effort;
• chord accuracy by named case;
• accuracy by Roman numeral;
• accuracy by progression position;
• missing expected pitch-class tone counts/rates;
• extra pitch-class tone counts/rates;
• failure category, distinguishing at least missing tone, extra tone, both missing and extra, harmonic-rhythm mismatch, event-position mismatch, insufficient polyphony, and generation/check/eligibility failures outside chord identity.

Preserve raw counts and denominators rather than persisting only percentages. Multiple missing or extra pitch classes in one position must not silently become multiple failed chords.

Dashboard direction

Follow existing model- and reasoning-comparison patterns while adding a chord-focused analysis section. At minimum, provide:

1. Strict case pass rate by model with passed/eligible counts.
2. Per-chord accuracy by model immediately beneath the headline.
3. Case comparison, such as a model-by-case heatmap, with chord accuracy and strict pass context.
4. Reasoning comparison across standard and reasoning-effort variants using established model ordering.
5. Roman-numeral performance with sample counts.
6. Progression-position performance comparable across cases of different lengths.
7. Chord-tone error profile showing missing versus extra tone counts/rates.
8. Failure-category view covering chord identity, harmonic rhythm, event position, polyphony, generation, check, and eligibility failures.

Existing filters should continue to support model/provider/root/scale/reasoning dimensions and add named case where appropriate. Hover text and exported HTML should include counts, denominators, and full labels.

Prefer a small number of complementary charts over one chart per grouping. Reuse shared dashboard aggregation, ordering, theming, eligibility, and export helpers rather than creating a parallel reporting system.

Dedicated local example

Add scripts/chord_progression_eval.py as a small intentional evaluation example:

• contain 3-4 representative major and/or minor block-chord cases;
• use explicit local Ollama model names only;
• never expand implicitly to cloud or paid providers;
• keep the evaluation matrix visibly small;
• use sustained triads, one chord per clearly specified harmonic window;
• demonstrate case-level scale selection and case-specific test parameters;
• document task-count multiplication before execution;
• retain the direct-import guard so importing the script cannot start an evaluation.

Leave scripts/test_eval.py unchanged; it remains useful for lightweight local arpeggiator evaluation.

The exact example progressions can be selected during implementation, but they should exercise more than one progression shape and include both major and minor cases without becoming a canonical benchmark suite.

Deferred enhancements / non-goals

Do not broaden the initial deterministic definition beyond sustained block chords. Record these as follow-up opportunities:

• chord-window recognition across the full harmonic interval rather than sampling only the boundary;
• arpeggiated chord recognition;
• rhythmic comping and permitted within-window rearticulation;
• anticipations, suspensions, passing tones, and non-chord tones;
• inversions, voicing quality, voice leading, extensions, borrowed harmony, or nondiatonic Roman-numeral semantics;
• automatic extraction of expected progressions from natural-language prompts;
• a repository-wide canonical benchmark catalog;
• paid-provider evaluation runs.

The current checks can remain strict for the block-chord contract. Documentation and chart labels should not present them as general-purpose harmony recognition.

Acceptance criteria

Case API and routing

• Evaluator.evaluate() accepts named structured cases with prompt, case-level test parameters, and optional scales.
• Omitting case scales retains existing major/minor expansion; specifying scales limits expansion.
• Supplying both cases and prompts is rejected clearly.
• Existing prompt-based callers behave unchanged.
• Invalid names, duplicate names, unsupported scales, malformed parameters, and missing required chord parameters fail before generation.
• Per-case parameters reach only that case's tasks and are not mutated.
• Task counts equal cases ? selected case scales ? roots ? models ? applicable reasoning variations.
• Case identity and resolved expectations are persisted in run configuration and individual results and participate in task fingerprints.

Scoring and analysis data

• Strict eligible full-case pass rate remains the headline result.
• Per-chord correctness is aggregated without weakening full-case pass semantics.
• Analysis exposes model, reasoning, case, Roman numeral, and progression-position dimensions.
• Missing tones, extra tones, and failure categories retain raw counts and meaningful denominators.
• Generation errors, check errors, ineligible results, and deterministic validation failures remain distinguishable.
• Older runs without case metadata still load and render with sensible legacy behavior.

Dashboard and exports

• The dashboard includes strict case pass rate and per-chord accuracy views.
• It includes useful case, reasoning, numeral, position, tone-error, and failure-category exploration without unnecessary duplication.
• Every rate communicates its numerator and denominator in labels or hover details.
• Case filtering integrates with existing filters and exported HTML.
• Empty states render cleanly for runs without chord checks or case metadata.

Example and validation

• A separate scripts/chord_progression_eval.py demonstrates 3-4 explicit block-chord cases with local models only.
• scripts/test_eval.py is unchanged.
• No test or validation step calls paid providers.
• Deterministic tests cover case validation, routing, persistence, backward compatibility, analysis aggregation, chart contents, denominators, filters, and empty/legacy states.
• The locked validation suite and package build pass.

Suggested delivery order

1. Add and validate the case contract while preserving the legacy API.
2. Route cases through task expansion, fingerprints, configuration, and persisted results.
3. Add the dedicated local example.
4. Flatten case and per-chord diagnostic fields in analysis loading with backward compatibility.
5. Implement reusable chord aggregations and deterministic tests.
6. Add focused dashboard views, filters, and export coverage.
7. Update README documentation with the strict block-chord contract and deferred generalized-harmony behavior.

No broad model-matrix expansion or provider evaluation is part of this issue.