[happy-v587]

Which issue or RFC does this PR close?

Closes #1982.
Closes #1983.

Rationale for this change

Supervisor wait spans cannot explain latency inside spawned workers, and a separately deployed background service needs its own protected metrics endpoint. This change adds child execution evidence and process-local health/progress observations while keeping durable acknowledgement distinct from child return.

What changes are included in this PR?

• Propagate W3C execution context into Server-composed spawned workers, instrument Memory/Topic Memory model calls and processing stages, and bound best-effort child tracing shutdown.
• Expose readiness/dependency state, processing freshness and invocation-duration histograms, plus model-usage queue pressure, dropped records and failed writes. Scrapes read local observations without running dependency probes.
• Add an authenticated, infrastructure-only background listener, with bounded labels and bilingual documentation for freshness, reset and coverage semantics.
• Preserve blocked classification in child execution spans. A real-spawn/local-OTLP regression verifies matching parent/child categories, non-error span status and an unacknowledged durable intent.

Supervisor execution trace
  -> spawn with W3C context
     -> child execution, model and processing spans
  -> verify durable acknowledgement

Are there any user-facing changes?

• With metrics enabled, the background role serves /health/live and /metrics at HTTP_HOST:POWERCONTEXT_SERVER_METRICS_BACKGROUND_PORT (default 8001). It uses the existing authentication, server.observe authorization and non-loopback bind policy. Set METRICS_ENABLED=false to disable this listener.
• Readiness metrics refresh at startup and every 30 seconds, retaining existing inference-probe caches. Processing measurements are local to each process/supervisor; scrape each deployed target. Missing observations must not be interpreted as zero failures.
• Tracing remains opt-in and requires tracing-otlp. No HTTP payload or persisted-format changes are introduced.

How was this change tested?

• python -m pytest tests/e2e/test_worker_otlp.py tests/e2e/test_worker_logging.py tests/test_server_tracing.py tests/test_observability_health.py tests/builtin/runtime/test_processing_composition.py tests/builtin/runtime/test_family_processing.py tests/builtin/runtime/test_model_usage_recorder.py tests/e2e/test_builtin_runtime.py::test_working_note_capture_does_not_promote_into_memory tests/e2e/test_runtime_server.py::test_working_note_http_and_sdk_capture_stays_out_of_memory tests/test_dream_review_transport.py::test_capabilities_remain_decodable_by_legacy_clients -q — 135 passed. Executed with the project Python environment from /tmp, using absolute test paths to avoid local environment-file discovery.
• python -m pytest tests/e2e/test_dream_usage.py -q — 6 passed.
• make check — passed, including lint/format checks, type checks, lock consistency and 50 integration-manifest tests.
• The blocked-worker regression failed before the fix (internal instead of blocked) and passes with the fix using a real spawned process and local protobuf OTLP/HTTP receiver. It verifies that the blocked intent remains unacknowledged.
• Validation limits: the OceanBase background CLI end-to-end test needs a configured backend and was skipped locally; Linux-only process metrics were also skipped on macOS. Local OTLP reception does not establish storage/query acceptance by an external tracing backend.

AI usage statement

Codex (GPT-6) assisted code review, the worker-block classification fix, regression testing, merge conflict resolution and PR preparation.

[happy-v587]

Addressed both review threads in 7263394:

• Spawned family/topic workers now accumulate recorder loss deltas through the worker context and return one bounded summary on completion or failure; the supervisor merges it into ServerMetrics, with model_usage_worker_reports_total{outcome} distinguishing received and missing reports. Covered by a real-spawn e2e with controlled recorder failures (including crash and tracing on/off).
• The metrics test now compares only invocation histogram samples across scrapes.

Both threads are resolved. @AlexStocks @hireview001 could you take another look?

[happy-v587]

The only remaining red check is tests (3.12, unit), which failed in tests/codex_plugin/test_token_savings.py::test_chunked_http_error_body_respects_the_absolute_budget (periods == [], a timing assertion). That test is unrelated to this change, and the same job passed on 3.11/3.13/3.14 in the same run. This fork account lacks permission to rerun jobs; could a maintainer rerun that job?

[happy-v587]

Addressed the new review threads in 74085fa:

• P1: model instrumentation now disables OpenTelemetry automatic exception events and status descriptions, then records only a bounded error.type and error status. This covers family and Topic Memory worker model calls; a unit regression verifies a ModelHTTPError body is not exported, and the real-spawn OTLP test verifies worker model spans carry no exception events or status messages.
• P2: child execution spans now map failure types through the same bounded processing_error_code/error_category path as the supervisor, so InvalidInferenceOutputError is recorded as dependency instead of internal.

Both threads are resolved. @hireview001 could you take another look?

[happy-v587]

The only red check is controlled-cli (windows-latest), failing in integrations/kimi-code/plugins/powercontext/tests/plugin.test.mjs:298 ("overflowed history ... concurrent Hooks" expected empty output but received runtime_unavailable). It is unrelated to this change and passed on the previous run. This fork account cannot rerun jobs; could a maintainer rerun it?

[happy-v587]

@AlexStocks 请基于最新 head 802f5bb2 再次评审。两条讨论均已补充修复说明和真实 CI 证据，并已关闭。

当前 CI 为 38 项成功、1 项既有条件跳过，无失败或待完成项。worker-observability 为 20 项通过、0 skip；真实 OceanBase background 1 项、clean migration 42 项、long-query 8 项均通过且无 skip。进程日志已保留，并核对了启动、关闭和退出的完整顺序。

验证：Main · E2E harness。

[hidb4ai]

LGTM

[hidb4ai]

LGTM