-
Notifications
You must be signed in to change notification settings - Fork 66
Send the Quality grade to the cloud, and stop the cloud inventing an empty one #5121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
eb986cb
e1db022
02ea09c
ee686f4
2dcfe5e
5ed2a00
83f4720
12edcaf
3e70015
3e2be7d
193d12e
cc66a17
3c7185a
ee9a5ff
3ad83cb
242b436
ded039f
fee103a
04a0112
af24c02
8a3b642
14e5e5c
e6e405f
35a2888
19022a6
4d849ee
4a83ad7
1541ad6
54f1e84
606dcd4
2010858
30b2bfd
331af14
d35f6d4
b0008ef
0609d51
cb3e39c
114a482
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17682,6 +17682,98 @@ def _reliability_score_session(events): | |
| } | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon implements _build_quality_snapshot() to emit node-wide and per-runtime quality report cards as an encrypted snapshot slice for cloud parity, but this new responsibility and the "read once, compose many" optimization pattern are not documented in the Local Observability Service blueprint. |
||
|
|
||
|
|
||
| _QUALITY_SNAPSHOT_WINDOW_HOURS = 168 # 7d — what the Quality tab asks for | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon implements _build_quality_snapshot() to emit node-wide and per-runtime quality report cards as an encrypted snapshot slice for cloud parity, but this new responsibility and the "read once, compose many" optimization pattern are not documented in the Local Observability Service blueprint. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon now implements a QualitySnapshotBuilder (_build_quality_snapshot) to emit quality report cards in the encrypted snapshot for cloud parity, but the Local Observability Service blueprint does not document this new component responsibility or its behavior. |
||
|
|
||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon implements _build_quality_snapshot() to emit node-wide and per-runtime quality report cards as an encrypted snapshot slice for cloud parity, but this new responsibility and the "read once, compose many" optimization pattern are not documented in the blueprint. |
||
| def _build_quality_snapshot(): | ||
| """The Quality report card, node-wide and per runtime, for the snapshot. | ||
|
|
||
| Founder live-hit 2026-08-22: the hosted Quality tab said "Nothing to grade | ||
| yet" for a machine whose local tab showed an A over 119 graded runs. The | ||
| grade had never ridden the snapshot at all, and the hosted container | ||
| answered the request from its OWN DuckDB — which exists but is empty, so | ||
| it reported "no runs" instead of failing. An empty answer and an | ||
| unreachable machine looked identical on screen and meant opposite things. | ||
|
|
||
| Read once, compose many: the node's sessions are queried ONCE (current | ||
| window, prior window, and a 30-day history for calibration) and grouped in | ||
| Python, so a card per runtime costs no extra queries. Sessions are graded | ||
| at ingest, so composing is mostly dict lookups; the bounded deep scan runs | ||
| once over the node's rows and every per-runtime card reuses that map. | ||
|
|
||
| Returns ``{}`` on any failure — a snapshot must never fail to build over | ||
| one optional slice, and the cloud falls back to saying the grade lives on | ||
| your machine. | ||
| """ | ||
| try: | ||
| from datetime import timedelta # module scope imports datetime/timezone only | ||
|
|
||
| from clawmetry import local_store as _ls | ||
| from clawmetry import quality_thresholds as _qt | ||
| from routes.quality import _assess_rows, compose_report_card | ||
|
|
||
| store = _ls.get_store() | ||
| if store is None: | ||
| return {} | ||
|
|
||
| hours = _QUALITY_SNAPSHOT_WINDOW_HOURS | ||
| now = datetime.now(timezone.utc) | ||
| since = (now - timedelta(hours=hours)).isoformat() | ||
| prior_since = (now - timedelta(hours=hours * 2)).isoformat() | ||
| hist_since = (now - timedelta(hours=24 * 30)).isoformat() | ||
|
|
||
| rows = store.query_quality_sessions(since=since, limit=400) or [] | ||
| prior_rows = store.query_quality_sessions( | ||
| since=prior_since, until=since, limit=400) or [] | ||
| hist_rows = store.query_quality_sessions(since=hist_since, limit=1500) or [] | ||
|
|
||
| by_rt_hist = {} | ||
| for h in hist_rows: | ||
| by_rt_hist.setdefault(h.get("runtime") or "openclaw", []).append(h) | ||
| thresholds = _qt.calibrate_all(by_rt_hist) | ||
| assessments = _assess_rows(rows, thresholds) | ||
| prior_assessments = _assess_rows(prior_rows, thresholds, deep_limit=0) | ||
|
|
||
| def _card(sub_rows, sub_prior, runtime): | ||
| return compose_report_card( | ||
| sub_rows, sub_prior, hist_rows, | ||
| window_hours=hours, runtime=runtime, | ||
| assessments=assessments, prior_assessments=prior_assessments, | ||
| ) | ||
|
|
||
| out = { | ||
| "window_hours": hours, | ||
| "all": _card(rows, prior_rows, None), | ||
| "byRuntime": {}, | ||
| } | ||
| # Calibration is identical in every card (same 30-day history), so it | ||
| # is carried ONCE here and re-attached client-side. Left inline it was | ||
| # ~1.9 kB duplicated across fourteen cards, a quarter of the slice | ||
| # spent saying the same thing. | ||
| out["thresholds"] = out["all"].get("thresholds") or {} | ||
| # Per-runtime cards so the hosted tab stays honest under the runtime | ||
| # switcher: a grade shown while a single runtime is selected must be | ||
| # THAT runtime's grade, never the node's total wearing its name. | ||
| # Every runtime the machine has run in the last 30 days, not only those | ||
| # with sessions THIS week: a runtime that was quiet this week needs a | ||
| # card saying so in its own name. Without one the hosted tab would fall | ||
| # back to the node-wide card and show another runtime's grade under | ||
| # this runtime's filter. | ||
| runtimes = {(r.get("runtime") or "openclaw") for r in rows} | ||
| runtimes |= {(h.get("runtime") or "openclaw") for h in hist_rows} | ||
| for rt in runtimes: | ||
| sub = [r for r in rows if (r.get("runtime") or "openclaw") == rt] | ||
| sub_prior = [r for r in prior_rows | ||
| if (r.get("runtime") or "openclaw") == rt] | ||
| out["byRuntime"][rt] = _card(sub, sub_prior, rt) | ||
| for _c in [out["all"], *out["byRuntime"].values()]: | ||
| _c.pop("thresholds", None) | ||
| return out | ||
| except Exception as exc: | ||
| log.debug(f"quality snapshot build failed (continuing): {exc}") | ||
| return {} | ||
|
|
||
|
|
||
| def _build_reliability(limit_sessions=25, min_sessions=4): | ||
| """Agent Reliability score for the cloud Pro Reliability tab (P1). | ||
|
|
||
|
|
@@ -21540,6 +21632,7 @@ def sync_system_snapshot(config: dict, state: dict, paths: dict) -> int: | |
| "governance": _build_governance(), | ||
| "dailyUsage": _du, # #2142: computed once above, shared with `spending` | ||
| "reliability": _build_reliability(), | ||
| "quality": _build_quality_snapshot(), | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon adds a quality snapshot slice to sync_system_snapshot() for the cloud dashboard, but the Cloud Fleet Dashboard blueprint does not document a CloudQualityInterceptor component for consuming the quality slice alongside the existing CloudSpendFlowInterceptor and CloudAttentionInterceptor. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sync daemon adds a quality snapshot slice to sync_system_snapshot() for the cloud dashboard, but the Cloud Fleet Dashboard blueprint does not document a CloudQualityInterceptor component for consuming the quality slice alongside the existing CloudSpendFlowInterceptor and CloudAttentionInterceptor. |
||
| "memoryAccess": _build_memory_access(), | ||
| "traces": _build_traces(), | ||
| "turnAnatomy": _build_turn_anatomy(), | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code emits a quality snapshot slice from the daemon via sync_system_snapshot() to reach the cloud dashboard, but the Cloud Fleet Dashboard blueprint documents CloudSpendFlowInterceptor and CloudAttentionInterceptor but does not document a corresponding CloudQualityInterceptor component to handle the quality slice on the cloud side.