Skip to content

fix(log): #0044 log-hygiene batch — artifact planner, ClientDisconnect, default-service setup probe (0.21.135) - #1051

Merged
oeway merged 1 commit into
mainfrom
fix/log-hygiene-0044-artifact-clientdisconnect-setup
Aug 13, 2026
Merged

fix(log): #0044 log-hygiene batch — artifact planner, ClientDisconnect, default-service setup probe (0.21.135)#1051
oeway merged 1 commit into
mainfrom
fix/log-hygiene-0044-artifact-clientdisconnect-setup

Conversation

@oeway

@oeway oeway commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Three prod log-hygiene findings from kth-k8s:zippy-goat's nightly review (2026-08-13). Prod runs 0.21.107, main ~0.21.134 — sub-findings 1a/1b were already fixed on main; 1c, 2, 3 are genuinely present and fixed here. None is a real fault — each restates a benign/expected condition at a level (INFO/ERROR) that either floods the retained kubectl logs forensic window or poisons the ERROR metric that health thresholds key on. Same log-hygiene family as #8 / #19 / #20 / #21.

1c — hypha/artifact.py::search query-planner narration → DEBUG

6 lines (search: stage parameter value, Adding stage filter condition, Adding condition to return ONLY STAGED/COMMITTED …) fired at INFO on every search() — a hot listing path (~300k lines/24h in prod). Internal planner breadcrumbs, no steady-state operator value → demoted INFO→DEBUG.

2 — hypha/http_rpc.py::_handle_rpc_post catches ClientDisconnect

When a client hangs up before/while its request body arrives, await request.body() raises starlette.requests.ClientDisconnect. The broad except Exception logged it ERROR + full traceback + 500 (618/24h). It's a normal network event → caught specifically (before the broad arm), logged at DEBUG with no traceback, answered 499 (no client left to answer).

3 — hypha/core/workspace.py default-service setup probe (the reusable code trap)

The remote-service proxy (get_remote_service → Munch/ObjectProxy) RAISES AttributeError('setup') when a default service simply defines no setup(). The old if svc.setup: inside a broad except Exception as e: logger.error(f"… {e}") turned that benign, expected case into a per-client ERROR whose {e} interpolated to the useless bare string "setup" (~20/24h, unactionable by construction). Fix: setup = getattr(svc, "setup", None) + if callable(setup): (no-setup → silent no-op), a narrower except around the resolve step, and every caught detail reformatted f"{type(e).__name__}: {e!r}" so a setup() that actually raises is diagnosable — naming the real failure (RemoteException: RemoteException('RemoteError:boom-in-setup…')), never the attribute name.

hypha/workers/__init__.py:38 has the same if svc.setup: shape but is intentionally left untouched — worker services define setup; out of scope.

Tests

tests/test_log_hygiene_0044.py — 5 real tests, Docker-free (FakeRedis), no mocks. All conftest fastapi_server* fixtures are subprocess.Popen (caplog can't reach them), so each path is driven in-process: a real ArtifactController(store, s3_controller=None); real HTTPStreamingRPCServer routes over httpx.ASGITransport with a valid Bearer token + monkeypatched Request.body raising ClientDisconnect; two real event-bus create_rpc peers registering a default service (built-in registry key seeded so the real :default@ block runs).

Reproduce-before-fix verified: stashing the 3 fixes makes the 3 regression guards fail, with the captured log showing the exact prod signature ERROR workspace:workspace.py:2401 Failed to run setup for default service …: setup.

5 passed in 3.06s

Version

Bumped 0.21.134 → 0.21.135 across the 10 packaging files. CLAUDE.md and .svamp/issues/0003.md mentions of 0.21.134 are #43 history and left as-is.

Merging does not publish an image (publish.yml is gated on release:published/workflow_dispatch) — the release-cut and deploy remain Wei/crimson-spider-gated.

🤖 Generated with Claude Code

…99, setup probe (0.21.135)

Three prod log-hygiene findings (kth-k8s:zippy-goat nightly review 2026-08-13).
None is a real fault; each restates a benign/expected condition at a level that
floods the retained kubectl-logs window or poisons the ERROR metric health
thresholds key on. Same family as #8/#19/#20/#21.

1c. hypha/artifact.py::search — 6 query-planner narration lines
    (`search: stage parameter value`, `Adding stage filter condition`,
    `Adding condition to return ONLY …`) demoted INFO→DEBUG. search() is a hot
    listing path (~300k lines/24h); these are internal planner breadcrumbs with
    no steady-state operator value.

2.  hypha/http_rpc.py::_handle_rpc_post — catch starlette ClientDisconnect
    specifically, BEFORE the broad `except Exception`. A client hanging up
    before/while its body arrives is a normal network event, not a server error;
    the broad arm logged it ERROR+traceback+500 (618/24h). Now DEBUG (no
    traceback) + 499 (no client left to answer).

3.  hypha/core/workspace.py — default-service `setup` probe. The remote-service
    proxy (Munch/ObjectProxy) RAISES AttributeError('setup') when a default
    service defines no setup(); the old `if svc.setup:` inside a broad except
    logged a per-client ERROR whose detail interpolated to the useless bare
    string "setup" (~20/24h, unactionable). Now: getattr(svc,"setup",None) +
    callable() guard (no-setup → silent no-op), a narrower except around the
    resolve step, and every caught detail formatted `type(e).__name__: {e!r}` so
    a setup() that actually raises is diagnosable (names the real failure, not
    the attribute). workers/__init__.py:38 has the same shape but is left
    intentionally (worker services define setup; out of scope).

Tests: tests/test_log_hygiene_0044.py (5, real, Docker-free/FakeRedis, no mocks).
Reproduce-before-fix verified: stashing the 3 fixes makes the 3 regression
guards fail, with the captured log showing the exact prod signature
`ERROR workspace:… Failed to run setup for default service …: setup`.

Version bumped 0.21.134→0.21.135 across the 10 packaging files (CLAUDE.md and
.svamp/issues/0003.md mentions of 0.21.134 are #43 history, left as-is).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added bug Something isn't working documentation Improvements or additions to documentation labels Aug 13, 2026
@oeway
oeway merged commit 40dfa82 into main Aug 13, 2026
7 checks passed
@oeway
oeway deleted the fix/log-hygiene-0044-artifact-clientdisconnect-setup branch August 13, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant