Skip to content

Trace writer lifetimes and yield replication verdict pages - #6323

Draft
schickling-assistant wants to merge 2 commits into
mainfrom
schickling-assistant/2026-11-10-writer-verdict-priority
Draft

schickling-assistant wants to merge 2 commits into
mainfrom
schickling-assistant/2026-11-10-writer-verdict-priority

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Interactive terminal setup performs foreground store writes. Existing traces did not expose writer-acquire versus hold time for individual operations, and replication verdict pages still borrowed the foreground writer even though the scheduler already supports background loans.

Goal

Make writer contention attributable and let interactive foreground work overtake queued replication verdict pages without changing authority checks or introducing another scheduler.

Decisions

Two signed commits: request-parented writer lifetime instrumentation, then verdict-page classification and regression tests.

  • Add st.db.writer child spans with writer_wait_ms, writer_hold_ms, queue, and static caller location. These use a sampled request context without the opt-in storage-phase flag. Existing metrics and their cardinality are unchanged.
  • Include shared finalization/commit/observer work before acknowledgment in batched hold time, and connection return in loan hold time.
  • Use the existing background writer lane for replication verdict pages. Keep per-acquisition trust-root revalidation, bounded chunks, and release points unchanged.

Verification

One final-head gate-slot ticket is in progress: cargo test --locked -p smallclaims and cargo clippy --locked -p smallclaims --all-targets -- -D warnings. Results pending; no pass claimed.
New deterministic coverage checks foreground batched progress during a real long verdict loop, trust-root changes between bounded loans before activation, and request-parented span fields/lifetime through finalization. No daemon restart or provider PTY input was used. Local pre-push evidence execution was disabled to prevent duplicate Rust admission; no CI dispatch requested.

Complexity

Reuse the existing scheduler, writer wait state, and context bridge. No second scheduler, busy-rejection path, or cancellation protocol.

Concerns

Priority is a dispatch-turn bound, not a latency guarantee: an active envelope, commit, and finalization remain non-preemptible. Concurrent long replication holds were observed, but the exact terminal timeout's writer wait remains unproven because its span had no writer attributes. The observed 3081 ms admission-loan hold from the 21:35Z trace is NOT fixed by this PR: that admission loan is already background. This PR makes the next reproduction name the holder site and distinguish writer wait from hold time. It does not claim to eliminate every attach stall.

Friction & bottlenecks

Existing telemetry lacked per-operation writer wait/hold attribution unless optional profiling paths were active. Instrumentation is trace-only and caller locations are not new metric labels.

Follow-ups

Use the new hold/caller attribution to identify any remaining non-preemptible long operation before changing chunking. No chunking changes are included here.

References

Related diagnostics-only work: #6320. This PR is independently based on main.

Posted on behalf of @schickling
field value
agent_identity unknown
session unknown
agent_persona unknown
agent_supervisor unavailable
agent_tool unknown
agent_tool_version unknown
agent_runtime unknown
agent_model unknown
worktree fractal/2026-11-10-writer-verdict-priority

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant