Repository navigation
SPE-5052 [S1]: tee the run log to Agent Substrate under one Graph ID - #5
Merged
Merged
Conversation
ADR 0001 phase S1, swarm side (criteria 3 to 6 and 8). - swarm/runlog.py: emit() mints an additive msg_id into each record and tees it to substrate after the local write. No SUBSTRATE_URL, no network. - swarm/substrate_tee.py: explicit swarm-type to EVENT_KINDS table, A01..A15 to swarm-aNN-xxx surfaces, session_id <AGENT>@<replica>:<graph_id>, dedupe on (surface, msg_id) over 24 h, a durable correlation to Graph ID cache, owner/name repo slug from the origin remote, Graph ID minting. - swarm/substrate_client.py: stdlib REST and stateless MCP client, 1.5 s timeout, back-off after an outage, never raises. - scripts/orch_plan.py: a new plan resolves or mints a Graph ID, binds it with graph_bind (adopting the server's winner) and registers the node set; --graph-id added. Re-plans still reuse and nothing time-varying enters the hashed brief. The tee fails open: with the substrate unreachable every swarm flow completes as before. Refs SPE-5052 Refs SPE-7629
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile on #5, all seven findings: - the bearer token is an unredirected header, so a redirect never carries it - a per-surface token override (SUBSTRATE_TOKEN_<SURFACE>) lets one orchestrating process that holds several tokens speak as each agent; S4 still provisions one SUBSTRATE_TOKEN per agent process - the whole request (connect and read) is bounded by the timeout plus a small slack and the body is capped at 1 MiB; a hit starts the back-off - --dry-run runs write the local record but never tee - the correlation to Graph ID cache is keyed by the substrate base URL - dedupe distinguishes pending from sent, so a killed send can be retried after 60 s while a sent record stays deduped for 24 h - the 24 h prune has an index and runs at most once a minute per process Each fix has a test. No change to the event contract. Refs SPE-5052
Greptile on #5: the deadline ran urlopen on a worker thread and armed the close timer only after urlopen returned, so a server trickling response headers kept the thread and its connection alive past the deadline, and later calls could pile up more. Every request now goes through one seam, _open, built on http.client in the calling thread. A timer armed before the request shuts the socket down at the deadline, which ends a blocked connect, header read or body read at once; no worker thread exists. Redirects are not followed at all, so the bearer is never re-sent, and the body stays capped at 1 MiB. Real-socket tests: a trickling-headers server is cut at the deadline with no lingering thread (stable over repeated runs), a 302 is not followed and the second server sees nothing, and an oversized body fails. Refs SPE-5052
Greptile on #5: connection setup (DNS, connect) was not bounded, and the http.client transport had dropped HTTP_PROXY / HTTPS_PROXY / NO_PROXY. Each request now goes through a per-call urllib opener (ProxyHandler, a handler that never follows redirects, and connection classes that register their socket the moment TCP is up and again after TLS setup) on a daemon worker. The caller waits at most TIMEOUT_S + DEADLINE_SLACK_S over DNS, connect, headers and body; at the deadline it shuts the socket down, and a connect that completes later is killed on arrival. Only a worker stuck in DNS can outlive its deadline, and at most four workers exist at once; with none free a call fails at once. Real-socket tests: stuck DNS returns within the bound and caps lingering workers, a late connect is killed, HTTP_PROXY is used and NO_PROXY bypasses it, plus the existing trickling-headers, redirect and body-cap tests. Refs SPE-5052
Greptile on #5: a non-2xx reply whose body timed out, failed or exceeded MAX_BODY_BYTES became an empty answer, so the client never backed off and each later write paid the timeout again. Such a body now raises in the worker and in _request's error branch, so it takes the same back-off path as a network failure. An ordinary non-2xx (normal body or none) is still returned as (status, text) without back-off. Tests cover an oversized 500 body, a trickling 500 body and the fake-level error branch. Refs SPE-5052
greptile-apps
Bot
dismissed
their stale review
October 7, 2026 15:50
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
SomeRandmGuyy
added a commit
that referenced
this pull request
Oct 7, 2026
…rite (stacked on #5) (#6) * SPE-5052 [S1]: tee the run log to Agent Substrate under one Graph ID ADR 0001 phase S1, swarm side (criteria 3 to 6 and 8). - swarm/runlog.py: emit() mints an additive msg_id into each record and tees it to substrate after the local write. No SUBSTRATE_URL, no network. - swarm/substrate_tee.py: explicit swarm-type to EVENT_KINDS table, A01..A15 to swarm-aNN-xxx surfaces, session_id <AGENT>@<replica>:<graph_id>, dedupe on (surface, msg_id) over 24 h, a durable correlation to Graph ID cache, owner/name repo slug from the origin remote, Graph ID minting. - swarm/substrate_client.py: stdlib REST and stateless MCP client, 1.5 s timeout, back-off after an outage, never raises. - scripts/orch_plan.py: a new plan resolves or mints a Graph ID, binds it with graph_bind (adopting the server's winner) and registers the node set; --graph-id added. Re-plans still reuse and nothing time-varying enters the hashed brief. The tee fails open: with the substrate unreachable every swarm flow completes as before. Refs SPE-5052 Refs SPE-7629 * SPE-5054 [S3]: swarm memory goes through the substrate's governed memory.write ADR 0001 phase S3, swarm side. The swarm had no memory module (its vector memory is design only), so this is a thin client, not a migration. - swarm/memory.py: memory_write maps decision/retro/pattern to decision/fact/fact and graph/repo/agent/global to graph:<id>, repo:<o/n>, agent:<surface>, global; derives an idempotency key from correlation, task, attempt, kind, scope and text and reuses it on every retry. Outcomes: denied raises E-POLICY, quarantined and review.required are returned and never retried, version conflicts re-read and re-submit with expected_version, an unverified write retries with the same key, and ok is true only for accepted and verified_persisted. An unreachable substrate is "unavailable", never a pretend write. memory_query and run_start_context read through memory_search and /brief, fail-open. - scripts/swarm_run.py: the assignment prompt gains a Substrate memory section when the substrate answers; unchanged byte for byte otherwise. - swarm/substrate_client.py: mcp_call_json for tools that answer a list. - tests: every handler, the key, and an AST scan proving no swarm-side store feeds an agent's context (the substrate's store-lock cannot). Refs SPE-5054 Refs SPE-7629 * fix(S1): harden the substrate tee after review Greptile on #5, all seven findings: - the bearer token is an unredirected header, so a redirect never carries it - a per-surface token override (SUBSTRATE_TOKEN_<SURFACE>) lets one orchestrating process that holds several tokens speak as each agent; S4 still provisions one SUBSTRATE_TOKEN per agent process - the whole request (connect and read) is bounded by the timeout plus a small slack and the body is capped at 1 MiB; a hit starts the back-off - --dry-run runs write the local record but never tee - the correlation to Graph ID cache is keyed by the substrate base URL - dedupe distinguishes pending from sent, so a killed send can be retried after 60 s while a sent record stays deduped for 24 h - the 24 h prune has an index and runs at most once a minute per process Each fix has a test. No change to the event contract. Refs SPE-5052 * fix(S3): unambiguous write keys, targeted version re-read, fuller store scan Greptile on #6, all five findings: - the idempotency key hashes a JSON array of correlation, task, attempt, kind, scope, normalized subject and text, so subject-only differences and '|' inside a field no longer collide; expected_version stays out of it - a version conflict re-reads the standing decision with a targeted search by subject, falls back to the scope-wide read, and stops with conflict instead of dropping expected_version when it cannot find it - the criterion-7 scan walks nested Python modules and catches side-effect, dynamic and re-export TypeScript imports - unavailable is documented as an unknown result: recover by retrying the same logical write, which reuses its key Each change is tested. Refs SPE-5054 * fix(S1): bound the network work itself, not the wait for it Greptile on #5: the deadline ran urlopen on a worker thread and armed the close timer only after urlopen returned, so a server trickling response headers kept the thread and its connection alive past the deadline, and later calls could pile up more. Every request now goes through one seam, _open, built on http.client in the calling thread. A timer armed before the request shuts the socket down at the deadline, which ends a blocked connect, header read or body read at once; no worker thread exists. Redirects are not followed at all, so the bearer is never re-sent, and the body stays capped at 1 MiB. Real-socket tests: a trickling-headers server is cut at the deadline with no lingering thread (stable over repeated runs), a 302 is not followed and the second server sees nothing, and an oversized body fails. Refs SPE-5052 * test(S3): intercept the substrate_client._open seam, not urlopen The client no longer calls urllib.request.urlopen (S1 moved every request to http.client behind _open), so the memory tests patch _open with the same fakes and keep every assertion. Refs SPE-5054 * address greptile review feedback Greptile on #5: connection setup (DNS, connect) was not bounded, and the http.client transport had dropped HTTP_PROXY / HTTPS_PROXY / NO_PROXY. Each request now goes through a per-call urllib opener (ProxyHandler, a handler that never follows redirects, and connection classes that register their socket the moment TCP is up and again after TLS setup) on a daemon worker. The caller waits at most TIMEOUT_S + DEADLINE_SLACK_S over DNS, connect, headers and body; at the deadline it shuts the socket down, and a connect that completes later is killed on arrival. Only a worker stuck in DNS can outlive its deadline, and at most four workers exist at once; with none free a call fails at once. Real-socket tests: stuck DNS returns within the bound and caps lingering workers, a late connect is killed, HTTP_PROXY is used and NO_PROXY bypasses it, plus the existing trickling-headers, redirect and body-cap tests. Refs SPE-5052 * address greptile review feedback Greptile on #5: a non-2xx reply whose body timed out, failed or exceeded MAX_BODY_BYTES became an empty answer, so the client never backed off and each later write paid the timeout again. Such a body now raises in the worker and in _request's error branch, so it takes the same back-off path as a network failure. An ordinary non-2xx (normal body or none) is still returned as (status, text) without back-off. Tests cover an oversized 500 body, a trickling 500 body and the fake-level error branch. Refs SPE-5052
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ADR 0001 phase S1, swarm side (SPE-5052): criteria 3 to 6 and 8. Pairs with the substrate PR https://github.com/swcstudiospace/agent-substrate/pull/11, which adds the 15 surfaces, the token rule and
graph_bindthis calls.swarm/runlog.py:emit()mints an additivemsg_idinto every record and tees it to the substrate after the local write. With noSUBSTRATE_URLnothing is sent and no file is created.swarm/substrate_tee.py: an explicit swarm-type toEVENT_KINDStable (the enum is not widened;swarm_typerides in the payload), A01..A15 toswarm-aNN-xxx,session_id = <AGENT>@<replica>:<graph_id>, dedupe on(surface, msg_id)over 24 h, a durable correlation to Graph ID cache, theowner/namerepo slug from the origin remote, and Graph ID minting.swarm/substrate_client.py: stdlib REST and stateless MCP client, 1.5 s timeout, back-off after an outage, never raises.scripts/orch_plan.py: a new plan resolves or mints a Graph ID, binds it withgraph_bind(adopting the server's winner on conflict) and registers the node set;--graph-idadded. Re-plans still reuse, and nothing time-varying enters the hashed brief.Cut from
origin/main:orch_plan.py,runlog.pyandscript_base.pyare byte-identical there and on #4's branch.Not in this PR
S2 lease bridge, S3 swarm memory client, S4 projector, S5 gate verdicts. Signed envelope files (
gate.verdict,task.assign) are files, not run-log records, so they are not tee'd; the gate script's own exit record is.Verification
pytest: whole suite passes, including 65 new tests (map exhaustiveness is checked by AST over every emit site, double-publish sends one request,conflict: trueadopts the server's winner, fail-open withurlopenraising on every call, re-plan still reuses).scripts/build_agents.py --check,build_trae_agents.py --check,bun test tests/ts: pass.ruff: no findings in the files touched. Two E401 findings intests/test_uplift_harness.pyare onorigin/mainand untouched.orch_planbinds a Graph ID, registers 13 nodes underacme/scratch, and its events are in the ledger asswarm-a01-orchwith sessionA01@r0:<graph>;events_verifyis intact; a re-plan reuses and leaves one binding row; with the substrate stoppedorch_planexits 0 in under a second.Refs SPE-5052
Refs SPE-7629
The PR appears safe to merge; no blocking issue remains.
What we checked:
_exchangepasses body-read failures and oversized error bodies back as exceptions._requeststarts the backoff, so later calls return before opening a socket.Summary
Adds an optional tee from the local run log to Agent Substrate. Events carry an agent identity and the run's Graph ID.
Greptile automatically discovered a related ticket that helped explain the purpose of this PR: send swarm events to Substrate under a shared Graph ID while letting local work continue during an outage.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD Plan["New plan"] --> Bind["Bind Graph ID; adopt server's answer"] Bind --> Register["Register task nodes"] Register --> Emit["Emit run-log record"] Emit --> Local["Write local JSONL with msg_id"] Local --> Enabled{"Tee enabled and not a dry run?"} Enabled -- No --> Continue["Continue local work"] Enabled -- Yes --> Claim["Find Graph ID and claim message"] Claim --> Send["POST /events"] Send --> Result{"Reply"} Result -- Accepted --> Sent["Mark message sent"] Result -- Refused --> Release["Release claim"] Result -- Unreadable or oversized --> Backoff["Return None; pause requests for 30 seconds"] Backoff --> Release Sent --> Continue Release --> ContinueReviews (5) · Last reviewed commit: "address greptile review feedback" · Reviewed by Greptile