feat(simulator): run container societies on Kubernetes, validated at a hundred agents - #932
Merged
Conversation
14 tasks
[gstack-context] Decisions: Main v1 uses one schema-bound RunSpec and Kubernetes executor with durable start-or-attach binding, aggregate admission, generations, at-most-once dispatch, and non-event-appending Temporal finalization. Remaining: Pass the exact candidate through docs checks and the mandatory isolated blind review, then begin the public contract/fake-kernel batch. Skill: /plan-eng-review [/gstack-context]
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…merge) Snapshot for review, committed with --no-verify at the user's request while a fix round was still writing to the tree. NOT the atomic commit; it will be squashed. Some files are mid-fix, and formatting/docs-drift that the pre-commit hook normally repairs has not been applied. Directories now name behaviors rather than vendors: kernel/ -> run/, runtime/ -> agents/, platform/ -> cluster/, with the Kubernetes and Temporal SDKs confined to named adapters. The render seam no longer round-trips platform-owned data back for validation, so Readiness, Attachment, and Bootstrap stop existing rather than being renamed. RunSpec carries a Symbol.for brand so one identity gate replaces a structural check that disagreed with a nominal one. Run reclamation gains a heartbeat and a durable in-cluster worker, because cleanup previously died with the process that started the run. Known red at the time of the review that prompted this snapshot: - lint: 7 errors in newly authored code - prettier and docs drift: unrepaired, hook bypassed - coverage: 81.63% -> 81.37% against a meet-or-beat requirement - vendor import boundary violated in 6 files and not yet lint-enforced - worker direct-invocation guard is not symlink-safe Tests were green at 37 files / 201 tests when the snapshot was staged. The ADR amendment still owes its blind teammate review gate, and the two OpenClaw fixes still have no live evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
chughtapan
marked this pull request as draft
August 5, 2026 06:18
chughtapan
changed the title
Symlink-safe worker entry detection: isEntryModule canonicalizes both the invoked path and the module URL, so a worker reached through a symlinked path still serves its task queue instead of exiting silently. NanoClaw regains the gateway-disconnect termination signal the render-seam collapse dropped. Application.attach takes a report sink, so a runtime that can observe its own bridge loss reports it while its Sandbox still runs. OpenClaw needs no sink; its gateway exposes no post-attach failure channel. An acquired Sandbox that stops being observable now fails the run instead of retrying forever, restored inside the existing per-agent read rather than by reinstating the discarded poll. The bootstrap tests assert the typed failure channel only, so they can detect a regression to throwing. Kubernetes and Temporal SDK imports are confined to their adapters, with the workflow surface carved out explicitly because a Temporal workflow module cannot avoid importing the SDK that defines it. Tests: 38 files, 218 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
chughtapan
changed the title
… path Five copies of "was this module the process entry point" had drifted into three semantics. The two profile launchers compared an uncanonicalized argv path against import.meta.url, so reaching them through the controller image's symlinked dist directory made a direct invocation look like an import: the process exited successfully having submitted nothing. cluster/entry.ts now owns the comparison and the duplicate exported guard is gone. The evaluation submitter spawned dist/platform/<profile>/main.js, a path no build produces since the profiles moved. Every cell submission failed with a generic submitter error. The path is derived in one place and pinned against both the source module it compiles from and the simulator's own scripts. Also confines the Kubernetes and Temporal SDKs to their adapters, leaving the workflow surface carved out because a Temporal workflow module cannot avoid importing the SDK that defines it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
Temporal forces a Promise at the activity boundary. That shape had propagated inward until the Kubernetes adapter carried a second Promise-native API beside its Effect one, and four behavior modules were written around it. Conversion now happens once, in runAtPromiseBoundary, and everything it calls is Effect-native. The boundary helper rejects with the run's own failure rather than a fiber wrapper, so Temporal records ControllerAttemptFailed or KubernetesCallFailed instead of an opaque FiberFailure. The init container is an Effect program over @effect/platform FileSystem. Validation failures are typed failures with the same single sanitized stderr line and nonzero exit; ENOENT is no longer control flow. Suppressions for async-keyword and promise-type fall from 56 to 12, and the survivors are all signatures the Temporal SDK owns: worker and connection construction, workflow execution, and the activity types proxyActivities requires. Raw node:fs survives only where @effect/platform has no lstat, since the containment checks must reject a symlink rather than follow it. Entry detection in the init container gains the existence guard the shared helper has, so a missing argv[1] is a plain negative instead of a thrown ENOENT. It stays a local copy because its own regression test executes this file as TypeScript through a symlink, where a .js specifier does not resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
…uster refactor) Main gained a directed-link fault-injection subsystem after this branch diverged: a policy interpreter, drop/delay/hold verbs, and the LinkDriver implementation whose absence left LinkController.disable with an unsatisfiable requirement here. The run now stands the fabric up and provides the driver, so that capability works again. The fabric shapes what this process can observe, which is the customer's own controlled endpoints. A roster agent runs in its own container and its agent-to-agent traffic never crosses that stream, so the fabric does not register it and link control over a containerized agent fails rather than silently passing traffic it claims to police. Carrying directed-link policy across the container bridge is not attempted here. Main also removed app principals. The conversation-create payload no longer carries appId, and the router process no longer sends one. Nothing flagged that file as conflicting; only the type checker found it. Resolution favored this branch wherever the cutover changed a contract and favored main wherever the change was additive. The agent runtime keeps its container shape and gains main's optional inbound-link stage; run execution keeps the cluster session and gains the fabric. Simulator: 40 files, 243 tests. Evals: 12 files, 88 tests. Both typecheck. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ Adds src/test-utils, the package's first, following the convention already in packages/server and packages/protocol: a curated index over one fixture module, exporting only what the run regressions consume. The kernel harness moved there so the execute suite is a suite rather than a suite plus its scaffolding. Declares the simulator's layer order in the architecture config rather than waiving the imports it implies: cluster over run over agents over network over events over ledger. Twenty-seven cross-domain findings were the check asking what the layering was, not reporting a violation. The in-cluster controller now reaches the run outcome and ledger error types through the package facades.
Main moved the repository scripts into domain folders and split verification across commit, push, and CI. The simulator profile checks keep their place in the workflow, now naming the relocated package test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
The merge kept this branch's package.json entries, which still pointed at the pre-reorganization script paths. pnpm install runs prepare, so every CI job failed before it could install: the tsgo exec-bit restore no longer lived where the script said. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wp7vy3DXqmDMg485Z3rhQ
… one experiment The reachability gate failed because compute-next-version.test.sh landed in the tree without the package.json entry that reaches it. Size the GKE profile for a single experiment rather than a standing fleet: one zonal cluster, one agent node, and one system node. The workload identity principal names the cluster's own location, which is now a zone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ter lifecycle The agent pool now idles at zero and provisions nodes when Kueue admits a cohort, so an idle profile costs only its resident controller. Scaling from zero requires the pool's label and taint in the pool declaration, because the autoscaler decides whether a node that does not exist yet would accept the pending pods. Size the ClusterQueue quota against one agent node's measured allocatable capacity. The previous quota described three e2-standard-8 nodes, which no longer exist; a quota larger than the pool can deliver admits a cohort that is then unschedulable, so the run hangs on pending pods rather than failing. Pin Agent Sandbox to the commit its v0.5.4 tag points at. The pin named the annotated tag object, and checking out FETCH_HEAD lands on the commit, so the installer's own verification could never succeed. Terraform no longer restates the chart's quota, which gave one number two owners and let the copies drift apart unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…KE profile one run path Server-side apply forces field ownership. An earlier submission's create owns these fields under Update, which conflicts with an Apply even from the same manager, so installing the run worker onto a cluster that had already hosted one failed with a 409 and no run could be submitted twice. Log the cause of a failed Temporal run instead of discarding it. The submitted detail stays sanitized because it is operator output and the connection it carries can hold a credential, so the cause belongs in the log rather than the message. cluster.sh gains run, which builds the controller image, pushes it, and references the digest the registry reports. Assembling that reference by hand is how a run ends up pulling an image that does not exist. Setup installs the experiment-grade Temporal the profile has always required but never deployed. Size the queue and the agent pool ceiling for a hundred-agent soak. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…invalid inputs unrepresentable Both agent runtimes carried the same workspace, MCP, and digest machinery. One agents/workspace module now owns it, and the redaction boundary is unchanged: the MCP projection still hashes environment key names without their values. Images and workspace paths become branded schemas validated at construction. The endpoint a runtime attaches to is a host and port rather than a URL, which removes seven predicates that only existed because the type could express a protocol, path, query, fragment, credentials, or a mismatched port. Drop the Distributed prefix from identifiers whose module no longer needs it to disambiguate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rdown A hundred-agent run surfaced four faults that a two-agent run cannot. Wait for the installed worker revision rather than any available replica. Every submission installs the image it just built, so every submission rolls the Deployment; counting the outgoing replica as ready handed the workflow to a Pod the rollout then deleted, and the activity stopped heartbeating mid-run. Heartbeat on a schedule for the whole attempt. Preparing a run costs one admission per agent, so a large cohort takes longer to prepare than the heartbeat deadline allows, and proof of life cannot wait for the observation loop. The signal binds to the activity where the SDK still owns the ambient execution context, because a fiber resuming after a timer does not. Carry the cohort's startup budget from the submitter to the controller. The controller read MOLTZAP_STARTUP_TIMEOUT_MS but nothing ever set it, so its two minute default was the only reachable value and a cold cohort could not finish becoming ready. Report a cluster failure's detail when it is stringified. Without it the ledger recorded a cluster error naming neither the operation nor its cause. Supervise the Temporal port-forward. A run outlives one, and losing it reported a run that was still going, and later succeeded, as failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Parking sets the system pool to zero, which the variable's own validation rejected, so the down verb could not run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ociety # Conflicts: # package.json
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The port-forward supervisor never supervised. errexit is inherited by the subshell, so the first dropped forward ended the loop that exists to replace it, and the readiness wait watched the supervisor rather than the forward, so a parked controller was indistinguishable from a working one forever. The liveness signal now survives its own failure. One throw ended the loop and starved the deadline exactly as a missing signal did. Binding it is required rather than optional, so an implementation that resolves the activity context per call, and therefore throws off the worker, cannot compile. --delete-artifacts now discards the objects it promised to, and refusing to destroy no longer depends on a wildcard that exits non-zero precisely when the bucket is empty. Pin the Agent Sandbox commit in one place. The profile and its test restated it, and updating the installer alone left the profile check red. Cover what the live run proved: the rollout predicate, the signal arriving before a cohort is admitted, the startup budget's path to the controller, and a cluster failure that names its operation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
chughtapan
changed the title
The ignore entries answered a partially installed worktree, where the tsx binary was present without its package, rather than anything about the repo. A complete install resolves it, and the entries then read as unnecessary configuration and fail the root lint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fourth case tipped the scope past the generative-test requirement. These pin the contract's fixed shapes rather than an invariant over generated input, so they carry the same reason the other regression-only scopes state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e the soak a home The blind teammate review of candidate 78ff2f9 returns FAIL. A post-review amendment changed binding text in an admitted record with no changelog receipt, no trajectory correction, and no cited source event, while the retained events say the opposite, and the profile documentation and tooling still enforce the pre-amendment gate. Reconciling that is a maintainer call, so the record states the blockers rather than resolving them. The hundred-agent soak was reachable from nothing. Document what it proves, which is that a cohort that size comes up and is reclaimed rather than that a gate passes, and check it alongside the other profile modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One end-to-end module replaces the two-, four-, ten-, and hundred-agent modules. The path is the same at two agents and at a hundred, so the count is an input rather than four near-copies, and the contradiction between a record that said four, a ledger that said ten, and tooling that enforced ten stops being a question about which file is right. MOLTZAP_COHORT_SIZE carries it along the path the startup budget already takes: the submitter refuses what could never be a count, the controller validates the bound, and the experiment reads it through the same validated configuration rather than reaching for the process. Correct the decision in place, with the receipt its own rule requires: the renamed field, the acceptance gate, the snippet's export, and an autoscaling non-goal that never meant a profile's node pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The documented hundred-agent recipe could not have worked. Cohort readiness is bounded by the controller's budget, whose two-minute default no longer covers provisioning nodes and pulling an image onto each one, and the variable that raises it appeared in no document. The module's own timeout never covered that wait either, so it is gone rather than misattributed. Attribute the newest correction the way this ledger already attributes its two other unlocated exchanges: retain the literal reply, state what was searched and when, and mark what no event states. The scale-claim non-goals go back unchanged, because nothing sourced their removal. Test the controller's cohort validator, exclude the first size the record defers, and stop the profile documentation naming a smoke that no longer ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ew gate An override is a decision the log should carry rather than an absence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
chughtapan
marked this pull request as ready for review
August 6, 2026 06:48
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The simulator runs a society of container agents on Kubernetes through one
RunSpec, on a local cluster or on GKE. A hundred-agent run on GKE finished green and is what most of this branch's late commits come from.What landed
kernel/→run/, andcluster/,agents/,events/,ledger/,network/named for behaviour rather than implementation.agents/workspaceowns the workspace, MCP, and digest machinery both runtimes had copies of. Images and workspace paths are branded and validated at construction; the endpoint a runtime attaches to is a host and port, which deleted seven predicates that only existed because aURLcould carry a protocol, path, query, fragment, or credentials.gke/cluster.shcoverssetup,up,run,down,delete. Agents autoscale from zero toagent_max_nodes, so an idle profile costs only its resident controller.The hundred-agent run
local/hundred-agent-soak.mjsbrings up 100 agents, holds them 10 minutes, and tears down. On GKE it finishedexitCode: 0/ProgramFinished, having scaled to 8 nodes and 101 pods, and its exported ledger holds 100agent-runtime-readyrecords and zero failures. The artifacts are in the profile's bucket asmanifest.json,records.ndjson, andcompletion.json.Getting there surfaced five faults a two-agent run cannot reach. Each is fixed here:
Update, which a laterApplyfrom the same manager rejects with 409, so a profile could serve exactly one run.MOLTZAP_STARTUP_TIMEOUT_MSbut nothing ever set it, leaving its two-minute default as the only achievable value.An error has occurred; it now records which call failed and why, which is how the last of these was found.Review
A pre-landing review ran three reviewers over the merged diff: testing, maintainability, and adversarial. All three returned HOLD; every finding above P2 is fixed in
24ae7343.The three that mattered most were mine and were not theoretical:
errexitis inherited by the subshell, so the first dropped forward ended the loop meant to replace it. Its readiness check watched the supervisor rather than the forward, so it could never fail and a parked controller hung forever. Proven with a harness, fixed, and re-proven.--delete-artifactsskipped the check but deleted nothing, andgcloud storage lsexits non-zero precisely when the bucket is empty, so underset -euo pipefaildeletewas bricked on the common case.toContainwas satisfied by a single upfront beat, so reverting the fix still passed. It now asserts the signal precedes cohort admission, and I verified a revert fails it.Left deliberately:
force: trueremoves the loud 409 that used to stop a second concurrent submission from rolling the worker under an in-flight run. Single-run-at-a-time is the profile's real contract and is not yet enforced — worth an issue, not a blocker for an experiment profile.Verification
build,typecheck:tests,lint,test,arch:check, andgke-profile-checkpass acrossworkspace,@moltzap/simulator, and@moltzap/evals; all 9 projects' tests pass.docs:check:driftis clean.gke-profile-checkwas red on this branch before this review — it runs in CI but is outside the usual gate set, and the Agent Sandbox pin fix had left the profile and its test asserting the old tag SHA. The pin now has one owner and the test cross-checks it.