[Pika-Yalei]

Which issue or RFC does this PR close?

Closes #1983.

Initial monitoring scope: protected standalone metrics, dependency observations, and stopped/stalled-processing evidence. Exact request creation age across restart or partial acknowledgement is deliberately out of scope.

Rationale for this change

Operators need to distinguish unobserved or stale dependencies, idle processing, absent acknowledgement progress, and failures without making metrics collection execute probes or processing work. Standalone background services need an authenticated metrics surface.

What changes are included in this PR?

• Protected metrics-only listener for standalone background processes, using the existing HTTP authentication/configuration contract and rejecting unauthenticated non-loopback binding unless explicitly allowed.
• Passive dependency/readiness snapshots with explicit states, observation timestamps, probe outcomes and bounded latency histograms.
• Verified acknowledgement counters/timestamps; distinct no-op, hold, retry, error and supervisor lifecycle/role observations. Acknowledgement demonstrates processing progress, not artifact production.
• Capacity-independent background pending snapshots over the existing intents table: count Scopes where requested_generation > handled_generation, per binding, with a 30-second default cadence and 2-second observation deadline. Failures preserve the previous coherent snapshot.
• Bounded local oldest_observed_pending_age_seconds as an approximate lower bound over retained locally observed pending work. History resets on restart or leadership change and is not an independent stall signal.
• No request-timestamp table, per-generation timestamp writes/pruning, new age migration, exact durable oldest_pending_* metrics, or legacy-age-unknown metrics.
• Bounded labels without execution identities/raw keys; collection and owned log-delivery failure accounting. Unsupported OTLP queue occupancy/exporter-drop measurements remain unavailable, not zero.
• English/Chinese operational documentation and alert guidance: pending-work evidence plus snapshot freshness, verified acknowledgement progress, running/leader state and startup grace. Stopped, idle, standby, policy holds, retries and failures remain distinct.

Are there any user-facing changes?

New operational metric families and a standalone background metrics endpoint are available when metrics are enabled. Existing processing count/counter meanings are preserved; pending counts refresh independently of admission capacity.

This PR adds no schema table or migration. Pending observation reads existing binding intents in the background; its database cost scales with those intents and remains bounded by the observation deadline. Metric collection only copies bounded in-memory snapshots and makes no database/model calls; normal endpoint authorization retains its existing reads.

The local pending-age lower bound does not reconstruct durable request creation times. A large age alone does not imply a stall, zero does not prove an empty backlog, and restart/failover resets local history. See the operational documentation for freshness, reset and aggregation rules.

How was this change tested?

Published head: 95e91f4f0a3f5c4075407c4c3270b1c7b6b87b2f. The checks below ran against the identical tree 634824fc22e660b49a32550584f85eaa84763b71. Full local checks below are complete. All seven new-head CI workflows are terminal: six succeeded and E2E failed on the independently reproduced upstream issue below; downstream acceptance coverage remains incomplete. Prior-head CI is not evidence for this head.

• Main CI passed all 24 jobs, including all eight Python 3.11–3.14 unit/e2e jobs and the complete website export. Kimi Code, ZCode, Native personal service, Desktop validation and License workflows also succeeded; the optional desktop installed-package job was skipped.
• Current CI blocker: E2E harness validation has 2 failed, 269 passed. Both Kimi ARM plugin-installation comparisons also fail on untouched upstream master e4efb39bd6b8303a508c83fd5fcde5f99290b785 (2 failed, 106 deselected): the harness leaves __POWERCONTEXT_SCOPE_SCRIPT_JSON__ unresolved while normal setup renders an absolute script path. Those upstream tests are absent from this PR branch; CI runs their synthetic merge with master. Both downstream Acceptance (sqlite/oceanbase) jobs are skipped. The three live OceanBase metrics cases run inside Acceptance (oceanbase), so they remain unverified on this head. All eight Python 3.11–3.14 unit/e2e matrix jobs and the website job have passed. No unrelated Kimi fix is included here.
• Clean Python 3.12 unit/doctest suite: 4,971 passed, 199 skipped (403.06s). Full e2e suite: 849 passed, 329 skipped (448.00s). Optional backend/host/live-provider skips are separate from the required live OceanBase CI gate.
• Local test processes use always-on tracing and writable XDG/npm cache directories; these are test-environment overrides, not repository or production configuration changes. Initial environment-sensitive failures were rechecked on the unchanged base and all failed cases passed before the clean aggregate reruns.
• Final focused persistence/runtime/metrics/authorization/process suite: 82 passed, 3 skipped; the three skips require a dedicated real OceanBase tenant.
• make check passed, including all pre-commit hooks, both type-check passes, and 50 integration-manifest tests.
• SQLite regressions cover reopening, partial/duplicate/concurrent acknowledgement, rollback, existing counters, binding isolation and 2,000-Scope observation, with no request-timestamp table created.
• Real combined SQLite API lifecycle tests cover idle, barrier-stalled worker, stopped supervisor, raised observation errors and actual observation timeouts with stale-snapshot preservation/recovery. Leased standby and persisted lease takeover/reacquisition cover unobserved state and local-age reset.
• Backlogs of 1 and 1,200 Scopes: 50 collections each issue zero SQL, preserve durable rows, and expose the same 54 metric series. This is bounded collection evidence, not a production-scale OceanBase benchmark.
• The live OceanBase CLI authorization/passivity/shutdown and reopen/partial-ack count tests are retained and updated, but their Acceptance (oceanbase) job was skipped because harness validation failed. Do not count optional local skips or prior-head results as passing evidence for these three cases.
• The independent Atomic Memory OceanBase job passed 42 clean-migration and 8 long-query cases without skips. It does not run the three metrics acceptance cases. The independent Atomic Memory SeekDB job also passed 42 clean-migration and 8 long-query cases without skips.
• Independent review found no blocking defect or new privacy exposure; background endpoint/authentication regressions passed independently (30 tests).
• Documentation lint, 13 content tests, 2 link tests, and links across 273 public pages/434 repository files passed. The local production build was terminated with SIGKILL, but the new-head CI website check passed the complete build/export: 866 static pages generated and 946 public pages/internal links verified.

AI usage statement

Implemented, reviewed, and tested with assistance from OpenAI Codex. AI assistance covered code, regression tests, documentation, and this PR description.

[hidb4ai]

One reproduced snapshot-consistency issue remains in the shared metrics endpoint: processing counts and their freshness metadata need to remain one coherent snapshot when collection fails.

[hidb4ai]

Design note (non-blocking): prefer existing processing state for the initial monitoring scope

My preference is to deliver stopped/stalled-processing detection without adding pc_artifact_processing_requests in this PR. We already have a fairly large database schema and are cautious about adding tables. Each addition should be justified by a concrete requirement whose benefit warrants the migration, write, retention, and maintenance costs.

The proposed alternative is:

1. Keep backlog observation on the existing intents table. Count Scopes where requested_generation > handled_generation, grouped by binding. Retain periodic background observation, its timeout and snapshot timestamp, and observation while worker capacity is full. Scrapes should only copy the published in-memory snapshot; normal endpoint authorization retains its existing reads.

2. Detect stopped/stalled processing from backlog and acknowledgement progress. With sufficiently fresh pending-work evidence, a stopped Supervisor indicates stopped processing. For a running Leader, sustained pending work with no increase in verified acknowledgements over an operator-defined interval indicates a possible stall. Use the existing completed_total and/or last-acknowledgement timestamp, with a startup grace period when no acknowledgement has yet been observed. Keep idle, standby, policy holds, retries, failures, and stale/unobserved data distinct. Acknowledgement proves processing progress, including handled holds/no-ops; it does not prove an Artifact was produced.

3. Keep any waiting-age metric explicitly local and approximate. The existing bounded oldest_observed_pending_age_seconds can remain a lower bound for locally observed pending work, with documented resets on restart or leadership changes and bounded membership. Do not present it as the exact creation age of the oldest durable request, or use it alone to decide that processing has stalled.

4. Remove the persistence added solely for exact age. Drop the new request-timestamp table, timestamp insertion/pruning on request/acknowledgement, and the exact durable oldest_pending_* / legacy-age-unknown metrics. Preserve the protected standalone metrics listener, dependency observations, progress/error/retry metrics, capacity-independent pending observation, and coherent snapshot publication.

The important scope decision is that the initial release would detect stopped/stalled processing, but would not promise exact request age across restarts and partial acknowledgements. This is a deliberate contract change: update the documentation, alert examples, and acceptance tests accordingly rather than substituting approximate values under exact-age names.

The current table addresses that stronger age requirement coherently, but stores one row per unacknowledged generation. A single stalled Scope can therefore accumulate many rows even though the backlog metric counts it as one Scope; these writes also happen when the metrics endpoint is disabled. Its shared binding/Scope/generation structure is reusable across Artifact Families, but reuse alone does not establish that this storage is needed now.

This alternative still has the background read cost of counting pending intents, which should remain measured and bounded by an observation deadline. If exact durable age later becomes a firm operational requirement, we can evaluate timestamp/range storage at the shared processing boundary, with explicit workload and retention expectations. The existing generation counters alone cannot reconstruct those historical times.

[hireview001]

• CI failed: validate (failure).
• CI failed: E2E harness (failure). Failed steps: validate / Validate harness and Compose definitions.

[hidb4ai]

LGTM

[AlexStocks]

@Pika-Yalei please fix the github ci action failed process.

[Pika-Yalei]

@Pika-Yalei please fix the github ci action failed process.

fixing it...

[hireview001]

• Resolve merge conflicts with the target branch.