feat(content): VSS-snapshot content-ingest pipeline for Docenta (UFI.0-UFI.6) - #563
Merged
Conversation
Captures why the 0.12.28 pin still holds (0.13 replaced the rustls-tls-native-roots feature with a rustls-platform-verifier + pluggable-crypto model), the recommended rustls-no-provider + ring path that avoids the aws-lc-sys C build, the xwin/musl cross-compile risk, and a full validation + rollout + rollback plan. Planning only, no code change.
`just winget-av-submit <tag>` builds the WDSI false-positive submission archive (password-protected, from the release's actual uffs-windows-x64.zip contents so it never drifts from the real bin set) and prints the portal URL, SHA-256s, and every form field to paste. A non-blocking Defender scan in release.yml warns at release time when the Windows binaries will likely trip the recurring winget Validation-Defender-Error, pointing at the helper. README documents the drill. Stopgap until Authenticode signing lands.
…UTF-16 anti-pattern gaps resolve_ext_ids zipped an unbounded 0_u16.. range against ext_names.iter(); RangeFrom<u16>::next() computes its next state eagerly, so once a drive's ext_names table reached its legitimate u16::MAX ceiling, a query extension absent from that drive drove the range one step past u16::MAX and panicked with "attempt to add with overflow" - taking the whole daemon down. Switched to .enumerate(), which cannot overflow. Also closes two from_utf16_lossy anti-pattern gaps: $ATTRIBUTE_LIST/$UsnJrnl:$J name decoding now routes through the crate's shared malformed-name-safe decoder (WI-4.1) instead of a redundant lossy implementation, and the two genuinely non-filename Win32 sysinfo decodes (volume label, registry string) are marked AUDIT-OK. Also documents the --agg <PRESET> top-N cap and its terms:FIELD,top=N escape hatch in --agg --help, and splits args.rs's static help text into args_help.rs to stay under the workspace's 800-LOC file policy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A production report showed `uffs --daemon start` failing with "Daemon did not become ready in time / request timed out" on a 7-drive / 25M-record machine whose load legitimately took 2m27s - 27s past the (fixed) 2-minute client timeout - even though the daemon was healthy and fully loaded moments later per `--daemon status`. await_ready's `timeout` is now an idle budget instead of a hard wall-clock cutoff: every DaemonStatus::Loading response whose drives_loaded advances resets the deadline, so a daemon under heavy system load that keeps visibly making progress is never killed by an arbitrary fixed cutoff - only one that stalls for a full timeout window is. A hard 5x outer ceiling still bounds the total wait. Applied to both the sync client (hit by the CLI) and its async sibling, with new tests proving both the extend-on-progress and still-times-out-on-stall behavior. connect.rs split connect/auto-start/retry into connect_autostart.rs to stay under the workspace's 800-LOC file policy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bare-bones Layer 0 (uffs-content-protocol: shared wire types) and Layer 4 (uffs-content: unprivileged coordinator binary) crates for the new VSS-snapshot-scoped content-export tool. No job intake, VSS, MFT, or streaming logic yet - placement and naming only, built against Docenta's uffs-ingest-protocol-v2-vss.md contract. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
paths, and manifest wire types (UFI.0) Implements the first slice of the UFFS Content-Ingest protocol per uffs-ingest-implementation-plan.md UFI.0: - codec: bounds-checked LE Reader (every length-prefixed field validated before allocation, per the enterprise-review's Finding H10), BLAKE3 digest/checksum32 helpers. Digest is locked as plain unkeyed BLAKE3-256 so Docenta can use content_digest directly as its content ID. - error: full stable ErrorCode taxonomy (design-doc S16) with round-trip tested as_str()/FromStr. - state: CandidateOutcome (4-way, unchanged from the addendum) and a new JobState lifecycle with a proptested-shape legal-transition graph, mirroring uffs-daemon's ShardState pattern. - path_encoding: lossless UTF-16LE WindowsPath (handles unpaired surrogates, which a String-based representation cannot even hold). - manifest: ManifestHeader/CandidateRecord/ManifestTrailer per design-doc S11, with self-describing length fields and BLAKE3-truncated checksums, full round-trip + mutation + proptest coverage. 83 tests, clean under lint-prod + lint-tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…UFI.0)
Implements the framed content/control stream per design-doc S12:
FrameEnvelope (magic/version/type/flags/checksums, payload bounds-checked
before allocation) plus JobBegin, FileBegin, ContentChunk, FileEnd,
FileFailed, FileDeferred, FileAck, JobEnd, Progress, Heartbeat, JobCancel,
and WindowUpdate. Split into crates/uffs-content-protocol/src/frame/{mod,
job_begin,file_begin,content_chunk,file_end,file_failed,file_deferred,
file_ack,job_end,control}.rs plus a sibling tests.rs (matching the
compact_cache.rs / tests.rs convention) — a single frame.rs file would
have tripped the 800-LOC file-size gate.
Incorporates one deliberate protocol extension beyond the base spec,
driven by concrete downstream (Docenta) consumer feedback: a two-tier
query/delivery model. Candidate-match filters (ext/date/size-min, same
shape as existing UFFS CLI filters) determine the full candidate set,
independent of a separate content-delivery ceiling
(JobBegin.max_content_delivery_bytes) that controls which already-matched
candidates get a body streamed. Modeled as ReadMode::MetadataOnly +
FileEnd.content_digest: Option<Digest> rather than a new outcome variant,
so the addendum's already-reviewed 4-way completeness formula
(candidate_count = succeeded + failed_retryable + failed_terminal +
deferred_manual) is untouched — a MetadataOnly file is still Succeeded,
just without a delivered body. This keeps large-file metadata available
for a consumer's reap/tombstone reconciliation without transferring bytes
the consumer would only record as metadata anyway.
Adds codec::Reader::read_bytes_exact for payload reads whose length was
already validated via a separately-encoded field (frame envelope), as
opposed to a wire length-prefix.
31 new tests (108 total in the crate), clean under lint-prod + lint-tests
+ the file-size policy gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds crates/uffs-content-protocol/tests/golden_fixtures.rs plus 10 frozen binary fixtures under tests/fixtures/*.bin: manifest header, candidate record, manifest trailer, and framed JOB_BEGIN / FILE_END (success) / FILE_END (metadata-only) / FILE_FAILED / JOB_END, plus two deliberately invalid fixtures (corrupt checksum, truncated frame) that must be rejected. These decode frozen, committed bytes rather than regenerating them from the current encoder each run — a test that reproduced its own expected bytes every time would never catch an accidental wire-format regression, since it would just compare new (wrong) output against itself. This is also the cross-language conformance surface the addendum (§5.5) calls for: "a future implementation in another language is supported only after passing the same conformance corpus." Regeneration is deliberately gated: `#[ignore]`d tests only write a fixture when UFFS_REGENERATE_FIXTURES=1 is set, so a routine `cargo test` run can never silently overwrite the frozen contract. Added a .gitignore carve-out for these fixtures (blanket *.bin is otherwise gitignored workspace-wide), following the existing upcase-tables / MFT-capture carve-out pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… (UFI.0) New Layer 0 crate implementing the private wire protocol between uffs-content (unprivileged Coordinator) and the privileged Snapshot Reader process, per the addendum's corrected architecture (S2.1-S2.4): ReadRequest/ReadResponse, VolumeIdentity, StreamKind, RequestedReadMode, ActualReadMode, and a narrow ReaderErrorCode subset. Deliberately does NOT depend on uffs-content-protocol even though both are Layer 0 - per crate-graph.md, Layer-0-to-Layer-0 internal deps are disallowed so each stays independently buildable. Duplicates a small (~300-line) bounds-checked LE codec rather than sharing uffs-content-protocol's, which is the direct cost of that independence rule - documented in the crate's Cargo.toml. The Reader MUST revalidate every ReadRequest field against the snapshot itself (identity, EOF/VDL, stream) rather than trusting the Coordinator's claimed range - documented on ReadRequest per addendum S2.3's "resolves the unnamed stream itself" requirement. 38 tests, clean under lint-prod + lint-tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends uffs-broker-protocol with the Broker's VSS-lifecycle API per
addendum S4.3: CreateSnapshotLease(Result), DuplicateSnapshotHandle,
RenewSnapshotLease, ReleaseSnapshotLease, QuerySnapshotLease, wrapped in
tagged SnapshotManagerRequest/Response envelopes, over a new
SNAPSHOT_PIPE_NAME distinct from the existing daemon<->Broker PIPE_NAME.
DuplicateSnapshotHandle only carries the reader PID over the wire - the
actual DuplicateHandle call and identity verification happen Broker-side
via the existing check_client_identity pattern, never trusted from the
wire (documented on the type).
Split into snapshot_manager/{mod,codec,messages}.rs + a sibling tests.rs
(same pattern as uffs-content-protocol/src/frame/) since one flat file
would have exceeded the 800-LOC file-size gate. codec.rs duplicates the
same small bounds-checked LE primitives as the two content-protocol
crates rather than sharing them, for the same Layer-0-independence
reason.
Existing HandleRequest/HandleResponse (daemon<->Broker MFT handle
protocol) are unchanged.
19 new tests (52 total in the crate), clean under lint-prod + lint-tests
+ file-size policy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds crates/uffs-content/src/db/{mod,schema,queries}.rs implementing the
addendum §6 job database: uffs-content-jobs.sqlite3, WAL + foreign_keys +
synchronous=FULL pragmas, and the six-table schema (jobs, candidates,
attempts, consumer_acks, snapshot_leases, job_events) from addendum §6.4.
queries.rs adds a narrow typed API (create_job, insert_candidate,
record_terminal_outcome, completeness_summary) - just enough to prove two
things with real SQLite, not mocks:
- the completeness invariant (design-doc §2.2/§21.7): candidate_count ==
succeeded + failed_retryable + failed_terminal + deferred_manual, with
both a passing and a still-incomplete case;
- crash-recovery durability (design-doc §19.2): write a job with a mix of
terminal and still-pending candidates, drop the connection without a
clean job-complete step, reopen from the same file, and confirm
completed candidates stay completed.
record_terminal_outcome takes a new TerminalCandidateState enum (the
four terminal variants only, no Pending) rather than accepting the full
CandidateState and asserting/panicking on a non-terminal value - passing
Pending is now a compile error, so there is no runtime check to get
wrong and no panic in production code.
New workspace dependency: rusqlite (bundled feature, so the Windows
cross-compile via cargo-xwin doesn't need a system sqlite3 import
library). This has not yet been through cargo-vet - flagging for a
follow-up audit pass before this branch ships.
10 new tests, clean under lint-prod + lint-tests + file-size policy.
Workspace-wide `cargo check` confirmed no regressions from the new dep.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drop the durable rusqlite-backed job/candidate/attempt ledger in favor of a restart-from-zero model: an immutable manifest, an append-only JSONL failure log, in-memory counters, and an atomically finalized run summary. A run either completes (a valid final summary exists) or it didn't, and is retried from a fresh VSS snapshot — no mid-job transactions, leases, or crash reconciliation to get right. Docenta's own content-hash deduplication makes re-streaming after a crash a no-op on the consumer side, so restarting from zero costs nothing. rusqlite is no longer a dependency anywhere in the workspace.
Implements the real Coordinator workflow (job intake, candidate enumeration, manifest construction, protocol framing) against swappable CandidateSource/ContentSource backends, per the implementation plan's §9.5 "fast" test strategy. DirWalkCandidateSource/FsContentSource are real, correct, std::fs-backed stand-ins for the VSS-snapshot and privileged-Reader-backed implementations that land in UFI.1/UFI.2. Adds the end-to-end dir-walk parity harness: a deterministic fixture tree (size classes, non-BMP filename, hard links), an independent plain_walk oracle sharing no code with the pipeline, a minimal protocol-decoding test consumer, and a parity test that decodes real wire bytes and recomputes content digests independently rather than trusting the producer's self-reported ones.
Adds SnapshotLeaseManager, the Broker's in-memory VSS-lease bookkeeping behind a swappable VssProvider trait, per the implementation plan §4.2. Deliberately cross-platform (unlike the rest of this Windows-only crate) so the lease lifecycle — create/renew/release/expire/reconcile — is unit-tested against a fake provider on every host, not just Windows. Lease state is purely in-memory: there's no durable table surviving a Broker restart, so reconcile_at_startup can unconditionally delete every VSS snapshot the real backend still reports at startup — a freshly constructed manager holds no lease for anything, so all of them are definitionally orphaned. Adds the Win32_Storage_Vss windows-rs feature for the real VSS requestor COM implementation landing next.
Adds uffs-vss-requestor, a tiny per-run Windows helper the Broker will spawn once per volume scan, per docs/dev/architecture/uffs-vss-rust-cpp-shim-implementation-guide.md. windows-rs does not generate bindings for IVssBackupComponents (the VSS requestor interface) — verified directly against the crate's own generated source, not assumed. Rather than hand-roll the COM vtable from memory (unverifiable, high risk of silent memory corruption), the actual requestor sequence lives in a narrow native C++ shim compiled against the official Windows SDK headers (already present in the cargo-xwin SDK cache) via build.rs + the cc crate. Everything else — the private control-pipe protocol, process lifecycle, parent-death watchdog — is ordinary Rust. Uses VSS_CTX_FILE_SHARE_BACKUP: ephemeral, auto-release, no writer participation. The session (and therefore the snapshot) lives exactly as long as this helper process does; on crash, releasing the last IVssBackupComponents reference is what deletes it — no orphan reconciliation or persistent snapshot tracking needed, matching the job model's "rerun from a fresh snapshot" recovery story. Compile- and link-verified end to end through cargo xwin build (clang-cl against the real vsbackup.h/vswriter.h/vss.h headers, linked against vssapi.lib/ole32.lib) and cargo xwin clippy at the same strictness as this workspace's lint-prod/lint-tests/lint-ci gates — not just a syntax probe, a real produced PE executable. Not yet wired into uffs-broker (spawning, Job Object assignment, pipe creation) — that's the next slice.
Adds the Coordinator-facing Snapshot Manager pipe server (broker/snapshot_manager/mod.rs): a separate named pipe from the daemon's MFT-handle channel, verifying the connected client is uffs-content before dispatching CreateSnapshotLease/ DuplicateSnapshotHandle/RenewSnapshotLease/ReleaseSnapshotLease/ QuerySnapshotLease requests to the lease manager, and handling DuplicateSnapshotHandle's separate reader-identity check + DuplicateHandle call for the approved uffs-content-reader process. Spawned from serve_pipe_requests so it runs under both --run and the SCM-dispatched service path. Adds WindowsVssProvider (broker/snapshot_manager/vss_helper.rs): the real VssProvider backend, spawning one uffs-vss-requestor helper process per snapshot, suspended, assigned to a JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE Job Object before it ever runs, then resumed — so a dead Broker's handle-table teardown kills every live helper automatically. Drives each helper over a private JSON-lines control pipe (a deliberate, documented duplicate of uffs-vss-requestor::protocol, since that crate has no library target and the protocol is tiny/internal-only). list_existing_snapshots is now a no-op: VSS_CTX_FILE_SHARE_BACKUP is auto-release, so there is nothing to reconcile at startup — the OS itself cleans up a crashed Broker's helpers. Compile- and link-verified end to end via cargo xwin build/clippy at lint-prod/lint-tests/lint-ci strictness, producing a real linked uffs-broker.exe.
Adds an #[ignore]d integration test exercising the whole pipeline built so far (native shim -> uffs-vss-requestor helper -> Broker lease manager -> Job Object cleanup) at runtime for the first time, rather than only compile/link-verified. Fixes two real compile errors this uncovered on the actual Windows target: file_identity()'s unstable file_index() had no feature-gate, and handle_connection tripped clippy's cognitive-complexity limit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extracts the VSS create/read/delete round-trip logic (previously only in an #[ignore]d test) into a production self_test_round_trip function shared by `uffs-broker --self-test-vss <dir>` and the test itself, so the two paths can never drift apart. Adds scripts/windows/vss-snapshot-validation.rs, a thin rust-script wrapper matching the existing scripts/windows/*.rs convention, so the whole VSS pipeline (native shim -> uffs-vss-requestor -> Broker lease manager) can be smoke-tested standalone on a real Windows box without a running daemon or Coordinator. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pre-existing links using unqualified item names that aren't in scope from their defining module: SnapshotManagerResponse and ReadMode are defined in the parent module (need super::), and digest32 was a stale name for the digest function. Caught by the pre-push rustdoc gate, unrelated to this session's VSS/broker work but blocking the push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…estor version
Root cause of the reported hang: build_command_line wrapped
volume_path in a bare quoted string ("C:\"), but a single trailing
backslash immediately before a closing quote escapes the quote instead
of terminating the argument under Windows command-line parsing rules.
Every argument after volume_path (including --parent-pid) was silently
swallowed into it, so uffs-vss-requestor failed to parse its own args
and exited before ever connecting to the control pipe -- leaving the
Broker blocked forever in ConnectNamedPipe waiting for a connection
that would never come, with the process already gone by the time
anyone checked tasklist. Added quote_windows_arg implementing the real
escaping rule (double any backslash run before an embedded/closing
quote) and applied it to every string argument in the command line,
with unit tests covering the trailing-backslash case plus the other
standard pitfalls.
Also wires uffs-vss-requestor's build.rs to call
uffs_version::emit_build_env(), matching every other UFFS binary --
its --version output was silently missing the git-sha/commit-date
fingerprint entirely, making exactly this kind of stale-vs-fresh-binary
mismatch invisible. The validation script now prints `--version -v`
for both uffs-broker and uffs-vss-requestor up front, and defaults to
target\release\ over the installed ~\bin\ copy so it exercises the
just-built dev binary for this not-yet-released flag.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every stage of --self-test-vss was previously silent between "Running:" and the final PASS/FAIL line, so a hang (like the arg-quoting bug just fixed) gave zero indication of where it was stuck. Adds tracing::info! checkpoints at every step: pipe creation, helper spawn (with pid), waiting for the helper to connect, waiting for its Ready/Failed event, snapshot creation, marker verification, and deletion/release. Split create_snapshot into wait_for_helper_ready + finish_create_snapshot to keep its cognitive complexity under the lint ceiling with the new logging in place, and split the self-test round-trip logic out of vss_helper.rs into a new sibling file (vss_self_test.rs) since the added instrumentation pushed vss_helper.rs over the workspace's 800-LOC file-size policy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Command::output() waits for the child to exit before returning anything, which silently swallowed every tracing::info! progress line the Broker's self-test now emits -- exactly the visibility the logging was added for. Switched to Stdio::inherit() + .status() so the Broker's stdout/stderr stream straight to the terminal in real time, letting a hang show which step it's actually stuck on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The wrapper had no timeout at all -- a hung helper connection blocked it indefinitely with no way to tell it apart from "still working". Switched to spawn() + a poll loop (default 120s, --timeout-secs override) that kills the child and reports a clear timeout instead of hanging forever; the last "vss: ..."/"self-test: ..." line streamed to the terminal before the kill now pinpoints exactly which step it was stuck on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Folds the manual "open a second terminal and run tasklist" check we kept doing by hand into the script itself: a background thread polls for uffs-vss-requestor.exe every second and logs RUNNING/NOT RUNNING whenever its liveness changes, so a hang shows immediately whether the helper is still alive or already gone -- no more asking the operator to check by hand. Also drops the default timeout from 120s to 30s per observed round-trip timings (snapshot creation and deletion both take low single-digit seconds in practice), so a genuine hang is reported much sooner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… too Real hardware testing (via the new self-test-vss watchdog) proved create/read fully work end to end, but every Release command hung indefinitely with uffs-vss-requestor.exe confirmed still alive the whole time -- IVssBackupComponents::DeleteSnapshots was the one call that never returned. VSS_CTX_FILE_SHARE_BACKUP is documented as an auto-release context; the intended teardown is releasing the last IVssBackupComponents reference, which this shim already did correctly on every other exit path (Cancel/PipeClosed/ParentDied) and which the successful create path already proved COM itself is responsive for. Removed uffs_vss_delete_snapshot_set (native + FFI + Rust wrapper) entirely rather than leave a call known to hang, and Release now uses the same drop-based teardown as the other three paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The DeleteSnapshots removal didn't fix the hang -- it persisted identically with a plain IVssBackupComponents::Release() call, and restarting the machine's stuck VSS writers (System/MSSearch/WMI, found via vssadmin list writers) didn't clear it either. The Broker's own tracing has zero visibility past "waiting for Released confirmation"; we don't know if the helper is stuck in Rust dispatch, the mpsc channel, or inside the native Release()/CoUninitialize() call itself. Adds a best-effort append-only log at %TEMP%\uffs-vss-requestor-debug.log bracketing every step: pipe connect, snapshot creation, entering the main loop, each received event, and explicitly before/after drop(session) and the Released write. If "session dropped" never appears, that pins the hang inside destroy_session's native COM calls; if it does appear but "wrote Released event" doesn't, the hang is in the pipe write instead. Diagnostic-only -- not a permanent feature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…vent The debug log proved drop(session) (native VSS release) completes in ~11ms -- the hang is entirely in the final pipe write, not VSS/COM. All writers already confirmed Stable (post-reboot) with the hang still identical, ruling that out too. Splits write_event's single "writing Released event" checkpoint into three: about to writeln!, writeln! returned, and flush returned, so the log pinpoints whether the raw WriteFile call itself never returns or whether it's the flush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… just Release) Root cause found via the debug log: it's neither VSS/COM (drop(session) took ~11ms), AV (disabled, hang identical), nor stuck VSS writers (rebooted, all Stable, hang identical). It's Windows serializing synchronous I/O across duplicate handles of one file object from different threads: the reader thread's next blocking read (waiting for a message that will never arrive) sat pending on a clone of the same non-overlapped pipe handle the main thread tried to write the final reply on, deadlocking that write forever. The prior commit's fix only covered Release/Cancel; Ping needed the identical fix, since it's meant to support repeated liveness checks over a lease's lifetime and would have hit the exact same deadlock on its first Pong reply once a real Broker used it. Ping/Pong is now handled entirely inside the reader thread (no session access needed, no cross-thread handle interleaving at all); only Release/Cancel are forwarded to the main thread, which stops the reader immediately after so its read is never concurrently pending with the main thread's teardown-and-final-write. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ping had no caller anywhere -- the deadlock fix for it was reviewed but never actually exercised end to end. Adds WindowsVssProvider::ping_lease, sending a real BrokerCommand::Ping to the live helper session and waiting for HelperEvent::Pong (not part of the VssProvider trait -- FakeVssProvider has no use for it yet, so it stays a WindowsVssProvider-only capability until a real feature needs it). --self-test-vss now calls it between marker verification and deletion, so the create/ping/delete round trip is proven for real, not just reviewed. A ping_lease failure surfaces as an overall self-test failure without skipping the cleanup delete. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A real deployment spawns one helper per content-scan job; unconditional logging would grow an unbounded file over millions of runs. debug_log is now opt-in via UFFS_VSS_DEBUG_LOG (unset = production-safe no-op) with a 10MB truncate-before-append cap as a safety net for an extended troubleshooting session. The validation script sets the env var automatically for the Broker it spawns, since that script only ever runs for diagnostics -- no manual step needed to get the log back. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nnel Real-hardware full-system content-read scans showed ~40% of candidates on one drive failing OpenFileById with ERROR_INVALID_PARAMETER, all sharing file_reference == 0. Bisected against a plain ad-hoc CLI search for one of the affected files (small result set, delivered inline) which reported the correct nonzero file_reference — proving the MFT parse and compact-index build were never at fault. Root cause: uffs-client's shmem transport (used once a search's result set crosses SHMEM_THRESHOLD, which any full multi-drive scan does) never carried file_reference in its compact ShmemRecord — the reader hardcoded 0 under the assumption that no shmem-path consumer needed it. That assumption predates uffs-content, which now needs file_reference on every candidate to OpenFileById against a VSS snapshot. 0 is never a valid NTFS file reference (it's the reserved $MFT record), so the reader correctly rejected every one of these reads. Add file_reference to ShmemRecord (88 -> 96 bytes, format version 3 -> 4), populate it on write and consume it on read instead of hardcoding 0, and change the round-trip test's fixture to a nonzero value so a regression here is actually caught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Step 1 of validating the "reads candidates in ascending FRS order == near-sequential physical disk access" assumption for uffs-content's read scheduling, before investing in an FRS-sort read-order change. Takes uffs --format json output (path + file_reference), samples a subset, and cross-references each file's on-disk starting LCN via `fsutil file queryextents` to report the Spearman rank correlation between FRS order and physical order. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same diagnostic as the previous check_frs_vs_lcn.ps1 (sample uffs --format json rows, cross-reference each file's on-disk LCN via fsutil file queryextents, report the Spearman correlation between FRS order and physical order) -- rewritten in Rust via rust-script per project preference over PowerShell tooling. Also fixes a bug in the original: the CSV output path construction failed when json_path was a bare filename with no directory component (Join-Path on an empty parent). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends check_frs_vs_lcn.rs beyond a single correlation coefficient: now also reports the total seek distance (sum of |delta LCN| touring the sample once) under three orderings -- the search response's natural order (what the read pipeline processes today), ascending-FRS order, and the oracle (true ascending-LCN order, the unbeatable lower bound). This turns "FRS correlates weakly/moderately with LCN" into a concrete, actionable number: what fraction of the achievable seek-distance reduction does cheap FRS-sorting actually capture, versus how much is only reachable by a real LCN-resolution pass. Also converts to MiB of head travel via `fsutil fsinfo ntfsinfo` when available. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a 3rd/4th arg (mode + optional offset): "random" (default, unchanged -- true uniform draw across every row) or "block" (sample_size *consecutive* rows in search-response order, from a given or random offset). Block mode tests a different, more operationally relevant question than global random sampling: are candidates as the read pipeline's bounded sliding window would actually encounter them together physically clustered on disk, versus whether FRS order tracks LCN order across the whole result set. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_bytes_per_cluster's digit extraction collected every digit character anywhere on the matching fsutil line, not just the value -- fragile to whatever exact formatting fsutil uses. Real-hardware run produced an implied ~40972 bytes/cluster (not a valid NTFS cluster size, which must always be a power of two between 512 B and 2 MiB), which silently inflated the printed MiB-of-head-travel numbers ~10x. Rewrite to only parse what follows the last ':' on the matching line (handling 0x-hex values too), and add a hard sanity check: reject any parsed value that isn't a power-of-two NTFS cluster size instead of silently reporting a nonsense conversion. Falls back to "unknown" if the format still doesn't match on a given Windows version, same as before. The bug only affected the MiB display -- the underlying cluster counts, Spearman correlation, and seek-distance-reduction percentage were always correct. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware benchmarking (check_frs_vs_lcn.rs) found ascending-FRS read order only weakly-to-moderately correlates with actual on-disk physical layout (Spearman 0.56-0.68 across independent random samples) and captures only ~28-29% of the achievable seek-distance reduction on a volume reorganized over years. The rest requires resolving true LCN. Adds an opt-in daemon-side post-search step instead of teaching uffs-content-reader to parse the MFT itself: uffsd already performs a full MFT parse to build its search index (an already-open, already-broker-authorized volume handle), so grafting one more targeted-record-read pass onto that is far cheaper than giving the intentionally narrow, non-elevated uffs-content-reader a whole new MFT-parsing capability it would otherwise never need, and avoids re-deriving everything from scratch in a second process. - uffs-mft: new `lcn_resolve` module. `resolve_frs_to_lcn` (Windows-only) opens the MFT's own extents and does targeted per-FRS record reads (ascending order, so the reads stay close to sequential within $MFT itself) to find each file's first non-sparse $DATA run's LCN. The underlying `first_data_lcn` byte-parser is pure and cross-platform, kept separately testable without Windows. - uffs-client: new `SearchParams::resolve_lcn_order` flag (opt-in, off by default -- interactive searches never set it). - uffs-daemon: new `IndexManager::device_paths` map (tracks which drives were loaded from a VSS snapshot device vs. a live volume, since VolumeHandle::get_mft_extents needs the same one reopened) and a new `physical_order` module: a plain post-processing pass over the already-built row list, run only when the flag is set, so every other search request's code path is completely unaffected. Grouped per drive (LCN only means something within one volume) and never fails the search -- a drive whose volume can't be reopened just keeps its rows in original order with a warning logged. - uffs-content: sets the flag when building its per-drive enumeration SearchParams. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SearchParams::resolve_lcn_order was only ever set by uffs-content's own job code (bypassing the CLI-args constructor entirely), so there was no way to exercise the daemon's physical-location-ordering feature through the ordinary uffs.exe CLI -- regenerating a JSON dump with the plain CLI always showed unsorted (natural) order regardless of the daemon-side feature, which was confusing during manual verification. Adds --resolve-lcn-order as an explicit, diagnostic-only CLI flag (off by default, same as before) so the read-order optimization can be exercised and measured directly (e.g. via scripts/windows/check_frs_vs_lcn.rs) without running a full uffs-content job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A real-hardware full-drive-set job showed drives were processed strictly one at a time in read_and_emit_all_candidates: a slow HDD-backed lease held up every other drive's candidates, including a fast SSD-backed lease sitting fully idle in queue, even though the two share no connection pool, no volume handle, and no physical device. Each lease run now gets its own thread, coordinated through a Mutex<EmitState> so two drives' candidates can never interleave their frames on the wire -- the protocol only requires per-candidate contiguity (FrameOrdering::None), which this preserves exactly. Also splits the now-826-line workflow.rs into workflow.rs + a new workflow/emit.rs sibling module (mirroring the file's existing workflow/pipeline.rs split), keeping both files under the workspace's 800-line file-size policy without trimming any documentation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware runs kept showing single-digit MiB/s content-read throughput on some drives even after LCN-sorted ordering, with no way to tell whether that's the drive's actual physical ceiling or still a software problem. measure_raw_throughput.rs streams raw sequential reads straight off a volume/VSS device (dd-style, bypassing the whole read pipeline) at three zones (outer/middle/inner) to measure the real floor to compare pipeline throughput against. check_frs_vs_lcn.rs now also reports fragmentation: how many sampled files span more than one on-disk extent, and the intra-file seek distance that costs even under perfect oracle LCN ordering, since ordering candidates by first-extent LCN says nothing about a file that's itself scattered across the disk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ileSizeEx Real-hardware run against E:/M: failed immediately with ERROR_INVALID_PARAMETER: GetFileSizeEx does not work on raw volume/device handles, only regular file handles. Volume size now comes from FSCTL_GET_NTFS_VOLUME_DATA's TotalClusters * BytesPerCluster, the same ioctl uffs-mft's own VolumeHandle::get_ntfs_volume_data already uses against these exact handles for this exact reason. Verified compiling clean against the Windows target (cargo xwin check + clippy); still no live-hardware run available from this session. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware run against C:/D: showed the outer-edge zone (2% offset) failing with ERROR_INVALID_PARAMETER while middle/inner zones (0.5/0.9 fractions) happened to succeed: volume-handle I/O requires a sector-aligned offset even for buffered reads, and an arbitrary fraction of the volume size essentially never lands on one by chance. Rounds every zone's offset down to a 1 MiB boundary before seeking, and changes read_zone to only ever issue full chunk_bytes-sized reads (dropping a less-than-a-full-chunk remainder) so a ragged final read can't hit the same alignment failure from the length side. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware runs against E:/M: produced an absurd "72 quadrillion MiB" intra-file seek distance -- traced to fsutil file queryextents reporting sparse-file holes with Lcn = 0xFFFFFFFFFFFFFFFF (the standard "no on-disk allocation" sentinel, not a real physical location). Any file with such a hole swamped the whole sample's total since u64::MAX dwarfs every real LCN by many orders of magnitude. query_all_extents now drops extents at that sentinel entirely (they carry no real seek cost, since there's nothing to seek to), and intra_file_seek_distance uses saturating arithmetic as defense in depth against any other unexpected value. Verified with a standalone rustc test simulating a real-extent/hole/real-extent sequence. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware benchmarking against E:/M: showed pipeline throughput far below their own raw sequential floor even with zero fragmentation, pointing at per-file open/close overhead as the likely cause -- but that was still a theory, not a measurement. read_logical now times each phase of a cache-miss (open_volume_hint, open_file_by_id, file_size) plus the read itself, and logs one structured debug-level line per call for later aggregation. That data was previously unreachable: uffs-content-reader's own tracing subscriber writes to stderr with no level cap, but its stderr was piped to Stdio::null() by the Coordinator, discarding every event including this new timing. reader_client.rs now redirects it to a discoverable temp log file instead (mirroring ephemeral_daemon's --log-file for uffsd), and logs that path in the Coordinator's own log. Diagnostic instrumentation only -- no behavior change to the read path itself. Full gate (host + xwin check/clippy plain and pedantic/nursery, fmt, file-size, workspace tests, rustdoc, doctests, xwin test --no-run) passes clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
open_volume_hint's CreateFileW was being opened and closed fresh for every single candidate, even though it only identifies the volume (never the file) and every candidate on one connection is read from the same volume -- one physical connection is only ever drawn from one lease's pool, so device_path never actually changes mid-connection. ReadHandleCache now caches this handle the same way it already caches the per-file handle, reopening only if device_path ever changed (a belt-and-suspenders check, not something that happens in practice). Removes one whole CreateFileW+CloseHandle cycle per candidate at no correctness cost -- real-hardware benchmarking against small-file-heavy drives found this handle was pure per-file waste today. Full gate (host + xwin check/clippy plain and pedantic/nursery, xwin doc for the cfg(windows)-gated intra-doc links, fmt, file-size, workspace tests, doctests, xwin test --no-run) passes clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware benchmarking against small-file-heavy drives found GetFileSizeEx a real fraction of per-candidate time even after the volume-hint and file-handle caching already in place. The manifest size the real VSS Coordinator already knows for each candidate (read from the exact same frozen snapshot the Reader opens the file against) makes that re-query redundant for the one real production caller. ReadRequest gains known_logical_size: Option<u64> -- opt-in per request, never a blanket trust-model change to read_logical itself. Only VssCandidateSource/ContentReader::begin_read populates it (via CandidateEntry::logical_size); any other/future caller that leaves it None gets the original always-re-verify GetFileSizeEx behavior with no code changes needed on its part. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RUSTDOCFLAGS="-Dwarnings" cargo xwin doc --workspace --document-private-items had apparently never been run against this workspace before -- every one of these was a real, pre-existing latent bug across uffs-mft, uffs-security, uffs-broker, uffs-client, uffs-daemon, uffs-cli, and uffs-content, invisible on the host (macOS) doc build because the affected code is #[cfg(windows)]-gated. Three distinct root causes, fixed accordingly: - Bare-name intra-doc links inside a MODULE-level (//!) doc comment resolve relative to something other than that module's own scope in this rustdoc version -- even for items defined in the very same file. Fixed by fully-qualifying via the item's real crate-rooted path (crate::module::Item), matching how other already-working links in the same files were already written. - Links to items that are genuinely private (module-private or pub(crate)) from public-facing doc comments -- de-linked to plain backtick text, since the private item can't be a real clickable destination in a normal (non---document-private-items) doc build. - A handful of genuinely stale references: stream::run (renamed to stream::spawn), MftReader::new_for_volume (renamed to MftReader::open), and DirWalkCandidateSource (referenced via a bare name never imported into that file's scope) -- fixed to the current real names/paths, not just silenced. cargo xwin doc --workspace --all-features --no-deps --document-private-items now passes clean end to end with -D warnings. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real-hardware run confirmed the running binary had the per-phase timing instrumentation built in (version=... c596e5c-dirty) and processed 118,847 candidates, yet the reader's own log file had zero "read_logical: per-phase timing" lines -- only INFO-level connection events. tracing_subscriber::fmt()'s default level caps below debug! when no max level is explicitly configured, silently dropping every one of the new timing events regardless of how the code was built. with_max_level(DEBUG) makes the instrumentation this crate already ships actually reach its own per-job temp log file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run_job's enumeration loop processed request.roots strictly one at a time -- each root's whole search-and-collect cycle blocking the next root from even starting, even though each enumerate() call opens its own independent connection to the daemon and shares no mutable state with any other call. Real-hardware benchmarking on a two-drive job showed ~15s + ~13s back to back (~28s total) for enumeration alone; running both concurrently cuts that to ~max(15s, 13s), and the effect compounds with every additional root a job touches. CandidateSource now requires Sync (mirroring ContentSource's own bound, already proven safe for exactly this kind of cross-thread sharing), and enumerate_all_roots_concurrently spawns one thread per root via std::thread::scope, joining and concatenating results back in roots' own order -- same shape as the earlier concurrent-lease-runs fix for content reads. Verified with a dedicated test (SlowEnumerateCandidateSource, 4 roots at 100ms each) proving wall-clock time reflects the slowest single root, not the sum -- 8/8 consecutive runs at ~140ms, comfortably under the ~300ms sequential-detection threshold. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documents the full filter surface (SearchFilters/SearchFilterParams, CLI flags, the narrower content-export JobRequest subset), the two independent extension/type-group taxonomies and how they combine, and the lack of magic-byte classification — the concrete questions raised while scoping Docenta's consumer-side extension handling.
githubrobbi
enabled auto-merge
July 19, 2026 18:36
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 19, 2026
…image
is_uffs_content_image's `starts_with("uffs-content")` fallback also
matched uffs-content-reader.exe, so verify_coordinator_identity would
wrongly accept the Reader process as a legitimate Coordinator. Drop
the starts_with fallback on both this and is_uffs_content_reader_image
(same risk class) in favor of exact-name matching; the two literal
forms already cover every real binary/exe pair.
Caught by the merge-queue run of PR #563 failing
recognizes_coordinator_image_names on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships the full UFFS-side half of the Docenta content-ingest protocol: a
VSS-snapshot-backed, privileged content-export pipeline that streams file
content for a UFFS query result over a framed wire protocol, without
exposing a raw volume/snapshot handle to the downstream consumer.
New crates:
uffs-content— the Coordinator: VSS snapshot leasing, ephemeraluffsdspawn + candidate enumeration, job workflow, transport server (
--serve).uffs-content-protocol— Coordinator<->consumer wire protocol (frameenvelope, all 12 frame types, manifest format, codec, error taxonomy).
uffs-content-reader/uffs-content-reader-protocol— the privilegedReader process and its Coordinator<->Reader IPC protocol
(
OpenFileByIdagainst the snapshot device).uffs-vss-requestor,--self-test-vss).Key milestones (UFI.0-UFI.6, see
docs/dev/architecture/uffs-ingest-protocol-v2-vss.mdfor the full spec):connection-blip resume
seek thrashing) and sliding-window pipelining
protocol_versionnegotiation/rejection, JOB_BEGIN snapshot provenance
JobRequestdefaulting to all local NTFS drives, skipVSS-unsupported (removable) drives instead of aborting
(real-hardware benchmarking confirmed both were sequential bottlenecks)
known_logical_sizeto skipredundant
CreateFileW/GetFileSizeExper candidatedaemon log surfacing
docs/architecture/filtering-reference.md— filter/type-group referencefor Docenta's consumer-side extension handling
Real-hardware validated: 721,532-candidate / 66 GiB run across 6 drives
(SSD + HDD), 4 transient read races on actively-changing files correctly
classified retryable (not terminal), 0 unexpected failures.
Test plan
cargo xwin test --no-run)lint-prod/lint-tests, plain + pedantic + nursery clippycargo xwin doc --workspace --document-private-items -Dwarnings