Skip to content

fix(evaluator): simplify provider concurrency handling #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laceyp99 merged 4 commits into from
Aug 14, 2026
Merged

fix(evaluator): simplify provider concurrency handling #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0caaff9
fix(evaluator): simplify provider concurrency
laceyp99 Aug 13, 2026
1dc97e3
docs(evaluator): clarify concurrency behavior
laceyp99 Aug 13, 2026
1227e9a
docs(evaluator): clarify concurrency limits
laceyp99 Aug 13, 2026
257ae38
fix(dashboard): simplify pass rate summary
laceyp99 Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,8 @@ Aggregated statistics for the entire run:
"total_generations": 48,
"successful_generations": 45,
"failed_generations": 3,
"generation_error_generations": 3,
"generation_error_generations": 2,
"rate_limited_generations": 1,
"check_error_generations": 0,
"validation_failed_generations": 6,
"ineligible_generations": 3,
Expand Down Expand Up @@ -409,9 +410,9 @@ Individual results for each generation:
```

Overall pass rates use only eligible results as their denominator. Each result persists
an `overall_status` of `passed`, `failed`, `ineligible`, `generation_error`, or
`check_error`. A check with no examined notes is ineligible and can never make
`overall_pass` true. A checker exception is a `check_error`, is excluded from pass-rate
an `overall_status` of `passed`, `failed`, `ineligible`, `generation_error`,
`rate_limited`, or `check_error`. A check with no examined notes is ineligible and can never
make `overall_pass` true. A checker exception is a `check_error`, is excluded from pass-rate
denominators, and is reported separately from a musical validation failure.


Expand Down Expand Up @@ -494,7 +495,13 @@ The evaluator continues on failures, logging errors and saving partial results:

## Performance Notes

- **Cloud providers** run asynchronously with rate limiting based on RPM from `model_list.json`
- **Cloud providers** run asynchronously with at most four concurrent requests per provider by
default. Set `max_cloud_concurrency` on `evaluate()` to adjust this cap. This is a concurrency
guard, not a request-rate limiter, and it does not guarantee compliance with RPM, TPM, or RPD
quotas. Provider SDK retry behavior varies and is not guaranteed by Eval. Reduce
`max_cloud_concurrency` or split work into smaller evaluations when using lower account limits or
running expensive workloads. Persistent provider throttling is recorded as `rate_limited`
rather than hidden as a generation failure.
- **Ollama** runs synchronously, sorted by model to minimize GPU memory swaps
- A live Rich progress table displays during evaluation with per-model pass rates, latency, and cost
- Large evaluations (many models x many prompts x many roots) can take significant time and incur API costs
88 changes: 70 additions & 18 deletions src/conductor_eval/analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
)
from dash import Input, Output, dcc, html

from conductor_eval.outcomes import get_overall_status
from conductor_eval.paths import get_evaluations_dir

PLOTLY_BG = "#1a1a2e"
Expand Down Expand Up @@ -115,6 +116,36 @@ def _overall_statuses(df):
return df["overall_pass"].map(lambda passed: "passed" if passed else "failed")


def _format_pass_rate_summary(passed, eligible_count, exception_counts):
"""Return compact pass-rate text and color for overview summaries."""
if eligible_count:
pass_rate = round(passed / eligible_count * 100, 1)
value = f"{pass_rate}%"
eligible_summary = f"{passed} / {eligible_count} eligible passed"
color = "#2ecc71" if pass_rate >= 50 else "#e74c3c"
else:
value = "N/A"
eligible_summary = "No eligible generations"
color = "#5dade2"

nonzero_exceptions = [(name, count) for name, count in exception_counts if count]
if not nonzero_exceptions:
exception_summary = "No generation errors"
elif len(nonzero_exceptions) > 1:
exception_summary = f"{sum(count for _, count in nonzero_exceptions)} run exceptions"
else:
name, count = nonzero_exceptions[0]
labels = {
"ineligible": "ineligible",
"generation_error": "generation error" if count == 1 else "generation errors",
"rate_limited": "rate limited",
"check_error": "check error" if count == 1 else "check errors",
}
exception_summary = f"{count} {labels[name]}"

return value, eligible_summary, exception_summary, color


# Plotly textposition options mapped to angles (degrees, counter-clockwise from +x axis).
# The label is placed in the direction of the angle relative to the marker.
_TEXT_POSITIONS = [
Expand Down Expand Up @@ -467,13 +498,7 @@ def load_run(run_path):
chord_progression_test = tests.get("chord_progression", {})
harmonic_rhythm_test = tests.get("harmonic_rhythm", {})
chord_event_positions_test = tests.get("chord_event_positions", {})
overall_status = (
"generation_error"
if result.get("error")
else tests.get(
"overall_status", "passed" if tests.get("overall_pass", False) else "failed"
)
)
overall_status = get_overall_status(result)

row = {
"task_id": result.get("task_id", ""),
Expand Down Expand Up @@ -2213,6 +2238,8 @@ def make_metric_card(title, value, subtitle="", color="#5dade2"):
Returns:
dbc.Card: Dash Bootstrap card component.
"""
subtitles = subtitle if isinstance(subtitle, (list, tuple)) else [subtitle]

return dbc.Card(
dbc.CardBody(
[
Expand All @@ -2230,7 +2257,11 @@ def make_metric_card(title, value, subtitle="", color="#5dade2"):
className="mb-0",
style={"color": color, "fontWeight": "bold"},
),
html.Small(subtitle, style={"color": "#666"}) if subtitle else None,
*[
html.Small(line, style={"color": "#999", "display": "block"})
for line in subtitles
if line
],
]
),
style={
Expand Down Expand Up @@ -2487,11 +2518,22 @@ def update_overview(models, roots, scales, variations):
eligible = _eligible_overall_rows(filtered)
passed = int(eligible["overall_pass"].sum())
statuses = _overall_statuses(filtered)
validation_failed = int((statuses == "failed").sum())
ineligible = int((statuses == "ineligible").sum())
failed_gen = int((statuses == "generation_error").sum())
rate_limited = int((statuses == "rate_limited").sum())
check_errors = int((statuses == "check_error").sum())
pass_rate = round(passed / len(eligible) * 100, 1) if len(eligible) > 0 else 0
pass_rate_value, eligible_summary, exception_summary, pass_rate_color = (
_format_pass_rate_summary(
passed,
len(eligible),
[
("ineligible", ineligible),
("generation_error", failed_gen),
("rate_limited", rate_limited),
("check_error", check_errors),
],
)
)
total_cost = filtered["cost"].sum()
known_costs = int(filtered["cost"].notna().sum())
avg_latency = filtered["api_latency"].mean()
Expand All @@ -2518,11 +2560,9 @@ def update_overview(models, roots, scales, variations):
dbc.Col(
make_metric_card(
"Pass Rate",
f"{pass_rate}%",
f"{passed} passed / {validation_failed} failed / "
f"{ineligible} ineligible / {failed_gen} generation errors / "
f"{check_errors} check errors",
color="#2ecc71" if pass_rate >= 50 else "#e74c3c",
pass_rate_value,
[eligible_summary, exception_summary],
color=pass_rate_color,
),
md=2,
),
Expand Down Expand Up @@ -2936,8 +2976,20 @@ def _build_combined_html(figures, run_name, timestamp, totals, df):
validation_failed = int((statuses == "failed").sum())
ineligible = int((statuses == "ineligible").sum())
generation_errors = int((statuses == "generation_error").sum())
rate_limited = int((statuses == "rate_limited").sum())
check_errors = int((statuses == "check_error").sum())
pass_rate = round(passed / len(eligible) * 100, 1) if len(eligible) > 0 else 0
pass_rate_value, eligible_summary, exception_summary, pass_rate_color = (
_format_pass_rate_summary(
passed,
len(eligible),
[
("ineligible", ineligible),
("generation_error", generation_errors),
("rate_limited", rate_limited),
("check_error", check_errors),
],
)
)
total_reported_cost = df["cost"].sum()
known_costs = int(df["cost"].notna().sum())
escaped_run_name = escape(str(run_name))
Expand Down Expand Up @@ -2971,8 +3023,8 @@ def _build_combined_html(figures, run_name, timestamp, totals, df):
<p style="color: #666">Run: {escaped_timestamp} | {total} generations | {len(df["model"].unique())} models</p>
<div class="stats">
<div class="stat-card"><div class="label">Total</div><div class="value">{total}</div></div>
<div class="stat-card"><div class="label">Pass Rate</div><div class="value" style="color: {"#2ecc71" if pass_rate >= 50 else "#e74c3c"}">{pass_rate}%</div><div class="label">{passed}/{len(eligible)} eligible</div></div>
<div class="stat-card"><div class="label">Outcomes</div><div class="value">{passed} / {validation_failed}</div><div class="label">passed / failed</div><div class="label">{ineligible} ineligible / {generation_errors} generation errors / {check_errors} check errors</div></div>
<div class="stat-card"><div class="label">Pass Rate</div><div class="value" style="color: {pass_rate_color}">{pass_rate_value}</div><div class="label">{eligible_summary}</div><div class="label">{exception_summary}</div></div>
<div class="stat-card"><div class="label">Outcomes</div><div class="value">{passed} / {validation_failed}</div><div class="label">passed / failed</div><div class="label">{ineligible} ineligible / {generation_errors} generation errors / {rate_limited} rate limited / {check_errors} check errors</div></div>
<div class="stat-card"><div class="label">Total Reported Cost</div><div class="value">${total_reported_cost:.4f}</div><div class="label">{known_costs}/{total} costs reported</div></div>
</div>
{"".join(chart_divs)}
Expand Down
Loading
Loading