Prevent partial chain state from escaping host recovery - #14
Closed
lr00rl wants to merge 25 commits into
Closed
Conversation
* fix: raise the task file-size cap from 8 MiB to 256 MiB The 8 MiB RLIMIT_FSIZE capped every file a task script could write — including the agent's own release binary during a self-update. The 2026-08-12 fleet upgrade died to SIGXFSZ (exit 153, 'File size limit exceeded') on 18 of 20 attempted nodes. The server-side update script now lifts the inherited cap itself (the only fix that reaches agents already deployed); this raise is the belt-and-braces for non-update tasks and for future agents once their update has run. 256 MiB still bounds runaway writes. * fix: bump grpc to v1.82.1 (GO-2026-6061) govulncheck flags the sing-box stats gRPC client path under the pinned v1.72.2. Unrelated to the rlimit change; the vuln DB entry postdates the last release. protobuf rides to v1.36.11. Full suite green.
The E3 transaction path depends on the task execution and 256 MiB output baseline already present on main, while the approved feature branch must retain the exact integration ancestry. Merge that prerequisite before adding E3 behavior. Constraint: Feature work starts at exact integration c783354 and requires main d2666e9 Confidence: high Scope-risk: narrow Reversibility: clean Tested: ancestry and exact prerequisite OID verified Not-tested: E3 behavior is introduced by the following commit
Apply the E3 fragment and source-side declaration through one private, crash-recoverable transaction manager. Recovery now precedes observable agent traffic, transfers one stable terminal result into the durable outbox, and retains generic task behavior. Installer and compatibility wiring make the private transaction root and alpha contract explicit, while the real sing-box harness proves client-to-source-to-target traffic. Constraint: E3 uses the existing opaque shell task channel and adds no dependency Constraint: Host state must be exactly the old pair or desired pair before result publication Rejected: Shell traps alone | cannot recover helper SIGKILL or power loss Rejected: Generic interrupted-task synthesis | could publish an unknown result before host recovery Confidence: high Scope-risk: moderate Reversibility: clean Directive: Keep linechain journal cleanup after confirmed outbox completion and before acknowledged outbox removal Tested: go test ./...; go test -race -cover ./...; go vet ./...; installer integrity; real sing-box 1.13.18 client-B-A-origin E2E Not-tested: Full server-to-agent E3 payload integration awaits the server lane exact contract
The initial E3 lane left authority and crash-boundary gaps around lock files, runtime layout, result retention, and artifact binding. This follow-up refuses hostile lock state, resolves layout locally, retains only E3 results during cleanup retries, binds combined digests to raw artifact bytes, and makes transaction roots durable and private. Constraint: Server documents cannot choose arbitrary host paths Constraint: Generic task result behavior must remain unchanged Rejected: Repairing hostile lock modes in place | could mutate another owner’s state Confidence: high Scope-risk: moderate Directive: Keep cleanup retryable without retaining unrelated generic results Tested: full normal/race-cover/vet/gofmt/diff/install gates Not-tested: Full live server payload replay remains cross-lane integration work
The follow-up touched existing tests and runtime layout discovery while closing the crash and authority gaps. Normalize those files so CI applies the same formatting gate as local verification. Constraint: CI rejects any gofmt drift Confidence: high Scope-risk: narrow Tested: gofmt and targeted Go tests Not-tested: none
Generic durable shell tasks must not be classified as replayable linechain operations merely because their script mentions a helper string. Require the server-authored E3 marker, helper invocation, and transaction-root binding together. Constraint: Generic result endpoints are not replay-safe Confidence: high Scope-risk: narrow Directive: Server task rendering must emit lattice-linechain-e3-v1 exactly Tested: go test, race-cover, vet, gofmt, diff Not-tested: full live server payload replay
Use explicit leased protocol metadata for live classification and derive artifact paths from the agent-resolved sing-box layout while retaining a narrow legacy recovery marker.\n\nConstraint: Existing pending journals do not carry leased protocol metadata\nRejected: Trust server-supplied absolute paths | permits host redirection\nConfidence: medium\nScope-risk: moderate\nDirective: Remove legacy marker fallback once all persisted outbox entries carry durable_protocol\nTested: go test ./internal/linechain ./cmd/lattice-agent\nNot-tested: authentic external sing-box E2E harness
Require explicit linechain-e3-v1 metadata for durable E3 execution and remove script-marker classification.\n\nConstraint: pending outbox records do not yet persist protocol metadata\nRejected: Parse script comments | spoofable and conflates generic delivery\nConfidence: medium\nScope-risk: moderate\nTested: gofmt; targeted go test (legacy cleanup regression remains)\nNot-tested: full contract matrix and real-binary E2E
Add an explicit protocol field and protocol-aware Begin seam so recovery can classify pending deliveries without parsing scripts.\n\nConstraint: existing Begin callers and persisted entries must remain readable\nRejected: infer protocol from task script | spoofable\nConfidence: medium\nScope-risk: narrow\nTested: go test ./internal/taskoutbox\nNot-tested: agent integration wiring and real E2E
Recovery failure now gates readiness and polling without flushing pending results, preventing arbitrary durable entries from crossing an unresolved linechain authority boundary.\n\nConstraint: recovery must precede every observable delivery\nRejected: flush retained results during blocked recovery | journal/outbox identity may be unresolved\nConfidence: high\nScope-risk: moderate\nTested: focused linechain tests (remaining cleanup retention assertion fails)\nNot-tested: full matrix and real E2E
Use the protocol-aware outbox seam when available while preserving compatibility with existing outbox implementations.\n\nConstraint: mocks and older stores expose only Begin(task)\nRejected: change interface abruptly | breaks generic and NetGuard implementations\nConfidence: medium\nScope-risk: narrow\nTested: focused linechain test (cleanup retention remains red)\nNot-tested: full matrix and real E2E
Treat protocol and durability as one leased-task contract, admit generic, NetGuard, and E3 pairs only, and bind protocol to duplicate journal identity. E3 tasks are rejected before journaling when linechain is unavailable.\n\nConstraint: protocol metadata is server-authored and security-sensitive\nRejected: accept durable=true with missing protocol | ambiguous execution authority\nConfidence: high\nScope-risk: moderate\nTested: internal/taskoutbox passes; existing agent tests require fixture updates for explicit protocol pairs\nNot-tested: full contract matrix and real E2E
Accept the v2 document envelope while preserving v1 journal compatibility, and require the agent-owned fragment basename before local path derivation.\n\nConstraint: existing journals and tests remain version 1\nRejected: trust server absolute paths in v2 | host path redirection\nConfidence: medium\nScope-risk: narrow\nTested: go test ./internal/linechain\nNot-tested: semantic sidecar merge and real E2E
Perform raw decode-time rejection of legacy absolute path fields before deriving agent-owned artifact locations.\n\nConstraint: v1 documents and journals retain compatibility\nRejected: validate after normalization | derived local paths obscure hostile input\nConfidence: high\nScope-risk: narrow\nTested: go test ./internal/linechain\nNot-tested: semantic sidecar overlay and real E2E
Require the v2 fragment basename to match the exact lowercase hexadecimal namespace contract.\n\nConstraint: server must bind deterministic artifact identity\nRejected: broad prefix and extension checks | permits ambiguous names\nConfidence: high\nScope-risk: narrow\nTested: go test ./internal/linechain\nNot-tested: sidecar overlay and real E2E
Replace the file-write smoke test with a loopback-only sing-box 1.13.x harness that observes the B-to-A hop, drives basename-only v2 Manager documents through create/resync/remove, validates runtime discovery, and exercises process-group crash recovery while the supervisor remains alive. Keep the binary external and mandatory in the script lane. Constraint: E2E owns only the agent test and invocation script; Manager, main loop, and outbox integration remain worker-2 owned Constraint: The official sing-box binary is a local test tool, never a Go or repository dependency Rejected: Stub restart/check commands | they cannot prove that B loads the published fragment Rejected: Starting B pre-chained | it cannot prove the apply operation changed live routing Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep the v2 fixture basename-only and preserve real process restart plus observer assertions Tested: go test ./cmd/lattice-agent -run '^TestLinechainE2E(Apply|Restart|Active)Helper$' -count=1 Not-tested: Full real-binary lane pending integration with worker-2 final v2 Manager head
Update the cleanup-failure regression to exercise the explicit leased E3 protocol contract rather than obsolete durable-result-only payloads.\n\nConstraint: generic and NetGuard deliveries must remain distinct\nRejected: infer E3 from script marker | spoofable\nConfidence: high\nScope-risk: narrow\nTested: focused test remains red, exposing outbox completion protocol propagation gap\nNot-tested: full matrix and real E2E
The integration regression must model a node that resolved its local sing-box layout; otherwise the protocol guard correctly rejects the lease before exercising retention.\n\nConstraint: E3 execution is forbidden when readiness is false\nRejected: weaken production guard | would permit unsafe host execution\nConfidence: high\nScope-risk: narrow\nTested: go test ./cmd/lattice-agent -run TestRunTasksLinechainCompletesHandoffWithoutReplay -count=1\nNot-tested: full matrix and real E2E
Replace the file-write smoke test with a loopback-only sing-box 1.13.x harness that observes the B-to-A hop, drives basename-only v2 Manager documents through create/resync/remove, validates runtime discovery, and exercises process-group crash recovery while the supervisor remains alive. Keep the binary external and mandatory in the script lane. Constraint: E2E owns only the agent test and invocation script; Manager, main loop, and outbox integration remain worker-2 owned Constraint: The official sing-box binary is a local test tool, never a Go or repository dependency Rejected: Stub restart/check commands | they cannot prove that B loads the published fragment Rejected: Starting B pre-chained | it cannot prove the apply operation changed live routing Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep the v2 fixture basename-only and preserve real process restart plus observer assertions Tested: go test ./cmd/lattice-agent -run '^TestLinechainE2E(Apply|Restart|Active)Helper$' -count=1 Not-tested: Full real-binary lane pending integration with worker-2 final v2 Manager head
A leased linechain task must be resolved by the linechain journal manager before generic interrupted-result synthesis; generic and NetGuard entries retain legacy recovery behavior.\n\nConstraint: E3 execution outcome is owned by the linechain transaction journal\nRejected: synthesize unknown result in generic outbox path | can conflict with terminal journal\nConfidence: high\nScope-risk: moderate\nTested: go test ./internal/taskoutbox\nNot-tested: full cross-check matrix and real E2E
Decode only the basename-bound v2 wire contract, derive all host paths from the configured runtime layout, and keep ordinary sidecar rewrites outside the fragment predecessor CAS. Normalize absent artifact digests consistently so create recovery can restore the exact old pair. Constraint: E3 v2 wire documents cannot carry host paths or previous sidecar digests Constraint: Existing v1 test fixtures are obsolete and migrate with the strict decoder Rejected: Preserve v1 wire compatibility | the approved E3 contract requires strict v2-only decode Confidence: high Scope-risk: moderate Reversibility: clean Directive: Never reintroduce server-controlled artifact paths or sidecar predecessor CAS into v2 Tested: go test ./internal/linechain ./internal/taskoutbox ./cmd/lattice-agent -count=1 Not-tested: Real sing-box lane until accepted E2E increment is integrated
Exercise the exact VLESS+REALITY transport shape, make the crash boundary signal-driven, and model ordinary sidecar resync outside the E3 transaction so removal proves it tolerates unrelated metadata drift. Constraint: E3 v2 artifact authority binds only the fragment predecessor; ordinary metadata writers may change sidecar bytes independently Rejected: Manager-driven metadata resync | it hides the required independent-writer drift case Rejected: Polling published files as the crash trigger | it races a fast restart and can miss the intended journal phase Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep the crash marker inside the restart helper so pair_published is deterministic Tested: tagged helper compilation and go vet for cmd/lattice-agent Not-tested: Unsimulated real-binary lane pending worker-2 Manager contract repairs
Canonicalize trusted runtime roots, reject trailing and legacy wire data, merge server-owned sidecar keys over ordinary metadata, and require exact task/lease/protocol agreement between bounded outbox and linechain snapshots before recovery can mutate either domain. Constraint: Completed E3 outbox entries may outlive cleaned linechain journals; leased E3 entries may not lack them Constraint: Ordinary sidecar top-level fields are outside E3 ownership and must survive create, replace, and remove Rejected: Byte-level sidecar predecessor CAS | independent metadata writers make it stale by design Confidence: high Scope-risk: moderate Reversibility: clean Directive: Run cross-contract validation before either generic outbox or linechain recovery Tested: go test ./internal/linechain ./internal/taskoutbox ./cmd/lattice-agent -count=1 Not-tested: Full/race/real-binary gates follow this commit
The managed linechain transaction now binds server-approved task scripts to canonical local artifacts, validates complete journal authority across the outbox handoff, preserves semantic sidecar ownership, and proves runtime state before recording terminal results. Trusted task execution no longer exposes linechain paths to ordinary or NetGuard tasks, while nofollow bounded reads and fault-matrix coverage close recovery and rollback ambiguity. Constraint: E3 recovery must suppress observable networking until exact outbox, journal, artifact, and runtime authority agree Constraint: Existing ordinary sidecar fields remain locally owned and survive managed overlay updates Rejected: Trust task and lease identifiers alone | they do not bind the approved script or desired artifacts Rejected: Include command output in durable failures | checker output may contain credentials Confidence: high Scope-risk: moderate Reversibility: clean Directive: Do not bypass RunLinechain protocol selection or weaken full-journal authority capture before recovery Tested: go test ./... -count=1; go vet ./...; go test -race -cover ./... -count=1; exact unsimulated sing-box 1.13.18 E2E; release workflow and install integrity scripts; version and compatibility output; secret scan Not-tested: GitHub CI on this unpushed commit
The cmd package test binary now enters the same rlimit child shim as the production agent before testing starts. This prevents a trusted E3 shell test from recursively running the entire test suite when Runner re-execs /proc/self/exe on Linux. Constraint: Linux task execution re-execs the current binary before invoking the interpreter Rejected: Increase the helper timeout | it would mask recursive test execution and keep CI flaky Confidence: high Scope-risk: narrow Reversibility: clean Directive: Keep cmd TestMain aligned with production taskexec.MaybeRunChildShim behavior Tested: helper race test x10; go test -race -cover ./...; go test ./...; go vet ./...; exact sing-box 1.13.18 E2E with zero residual helpers; installer and release checks Not-tested: GitHub CI for this new commit
Contributor
Author
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome
Verification
go test ./... -count=1go test -race -cover ./... -count=1go vet ./...sh scripts/test-install-integrity.shLATTICE_SINGBOX_E2E_BIN=/absolute/path/to/sing-box sh scripts/test-linechain-e2e.sh(sing-box 1.13.18)git diff --checkBoundaries
integration