Skip to content

feat: consolidate Meridian orchestration workspace - #139

Open
yzsnstotz wants to merge 981 commits into
mainfrom
feat/meridian-monorepo-portability
Open

yzsnstotz wants to merge 981 commits into
mainfrom
feat/meridian-monorepo-portability

Conversation

@yzsnstotz

Copy link
Copy Markdown
Owner

Summary

  • Consolidate Runtime, Orchestrator, Supervisor, CLI, contracts, and the independent Gateway in one portable Meridian workspace
  • Remove hardcoded developer paths and add native/Clawso service discovery
  • Add MIT licensing, migration tooling, clean-home lifecycle coverage, and safe dependency pins

Test plan

  • npm audit --audit-level=high (0 high/critical; one low esbuild development-server advisory remains)
  • npm run build
  • npm run typecheck
  • npm run lint
  • npm test
  • npm run test:orchestrator (125 files, 1520 tests)
  • npm run test:integration (4 tests)

Delivery status

  • PR only: this branch is pushed for Clawso Marketplace source pinning; merge requires separate approval.

yzsnstotz and others added 30 commits May 21, 2026 06:58
[BATCH-3-GATE] — Background-trigger end-to-end against synthetic fixture
[B1] — config/projects/mumu.json file
[B2] — config/projects/mumu/manifest.json file
[B3] — short_drama seed templates
[B4] — seeds/prompts/create_from_template.md
[B5] — seeds/prompts/style_observe.md
[B6] — seeds/prompts/style_user_write.md
[BATCH-4-GATE] — validate mumu manifest gate
…indow prune

Root-cause for a 2026-05-21 system-wide slow-spawn storm where syspolicyd
held ~80% CPU and trustd ~20% with a tight fcntl OFD_SETLK loop, stalling
every exec on the machine. The meridian-roles GUI hub (PID 31837) sustained
~2000 page-faults/sec for 13h; sample profile showed the main thread inside
SyncProcessRunner::Spawn under an AsyncFunctionAwaitResolveClosure. fs_usage
traced the spawnSync churn back to defaultGetProcessAttrs (execFileSync ps
+ lsof) plus a burst of open()+pread(64KB) on every rollout in
~/.codex/sessions/<today>/ — 89 files on the incident day.

The cost model the existing learning prescribed (≤10 unresolved rows × 3s
poll cadence = negligible) broke under three drifts:
- K: 6 ESTABLISHED inbound HTTP connections from Chrome on :7701 (multi-tab
  System Monitor). No per-connection rate limit on /api/agentapi-processes.
- M: orphan PIDs from a C6 validator-respawn loop in mumu/taskspec stayed
  unresolved indefinitely. Negative pidToFile entries were NOT cached at all,
  so every poll re-paid the full ps+lsof+N-rollout-scan per orphan PID.
- N: 89 rollouts in today's day directory; collector pread'd 64KB head from
  every single one on every unresolved lookup.

Per-tick cost ≈ K × M × (2 spawns + 2 × N × open/pread/close) — for K=6,
M=5, N=89 that's ~60 spawns + ~5340 file-head reads every poll cycle,
sustained over 13 hours. The validator-respawn loop is the supply line, not
the cause; resolving M-growth at the resolver is sufficient to break the
amplification regardless of upstream Pathology-1 surfaces.

The previous learning (codex-claude-session-file-resolution-and-process-shape.md
§"Pitfalls hit on first ship" §B) explicitly predicted this and prescribed
the fix: "If that ever shows up in profiling, the next step is a per-PID
short TTL (≈30s), NOT reintroducing permanent negative caching." This PR
finally ships that.

L1 — negative-resolution TTL (NEGATIVE_RESOLUTION_TTL_MS = 30_000).
pidToFile is now Map<number, { filePath: string | null; attemptedAtMs }>.
Positive entries are kept for the PID's lifetime (unchanged). Negative
entries are kept for 30s, then re-resolved. The rollout-warmup retry
contract is preserved (codex creates the rollout ~14s after spawn, so the
first poll still legitimately misses and re-tries after the TTL).

L2 — filename-window prune in listCodexRolloutFiles. Codex rollout
filenames encode session start as rollout-YYYY-MM-DDTHH-mm-ss-<uuid>.jsonl
(UTC). Parsing the filename ts lets us drop out-of-window candidates BEFORE
any open()/readSync. The window mirrors the existing
[pidStartMs - 1s, pidStartMs + CODEX_SESSION_OPEN_WINDOW_MS] check that
peekCodexSessionMeta performs against the session_meta.timestamp, so no
authoritative filter changes — just a fast-path that drops the I/O.
Non-conforming filenames (test fixtures like rollout-A.jsonl) fall through
to peek.

Tests: existing "does NOT cache negative resolution" test updated to assert
new TTL semantics (cached within TTL, re-resolved after). 4 new tests for
the filename prune: before-window rejection, after-window rejection,
non-conforming filename pass-through, and a scale test (87 out-of-window +
2 in-window) asserting listDir is the only I/O.

All 31 token-usage tests pass. Full 164 server-scope tests pass. The 4
pre-existing flaky test files (a2a TCP race, validator-orchestrator,
scheduler/archiver, scheduler-engine) fail identically on clean origin/main
and are not touched by this change.

Companion learning: Docs/Projects/Meridian-roles/learnings/
token-usage-orphan-pid-rollout-fanout-storm.md captures the failure mode,
diagnostic chain (sample + fs_usage), and the K×M×N cost model so the next
agent facing "syspolicyd 80% / trustd OFD_SETLK / hub PID page-faults" can
short-circuit straight to the resolver rather than chasing the macOS-side
red herrings the prior storm learnings document.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rm/V-01

fix(token-usage): negative-resolution TTL + filename-window prune
[X1] add ads-prod HMAC caller registration
[V-01-A] — E2E §5.3 automated harness
[V-01-A] — address validator E2E coverage
The chatter-create.html form and the chatter-init CLI wizard both
target POST /api/role to create roles (with role_type on the body),
but the route was missing from matchRoleRoute/the dispatch switch.
Every request returned 404 even though handlers.createRole was
already implemented and exercised by 81 unit tests — so on a fresh
meridian-roles deploy there was no HTTP path to create a chatter
without using the legacy POST /api/agent-dispatcher/start endpoint
(which forces role_type=agent-dispatcher).

Wires the existing handler symmetrically with the other /api/role
routes (GET /api/role/<id>, PATCH .../config, DELETE) — three small
additions in role-handlers.ts:

- add "create-role" to RoleRouteMatch union
- add POST /api/role -> { kind: "create-role" } in matchRoleRoute
- add case "create-role" -> handlers.createRole(body), status 201

Adds one integration test exercising the new route end-to-end via
the existing test harness; 82/82 unit tests now pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(roles): wire POST /api/role to handlers.createRole
yzsnstotz added a commit that referenced this pull request Aug 14, 2026
The final CI step runs `npm run test -w @meridian/orchestrator`, but main has
no `workspaces` key and no such package -- that layout exists only on
feat/meridian-monorepo-portability (#139). On main the step fails immediately
with "No workspaces found: --workspace=@meridian/orchestrator".

It arrived on main in a60fcef, which created ci.yml already written against
the post-#139 layout. Nobody noticed because the step is last and the suite
before it failed first, so every PR was already red for an unrelated reason
and this second failure sat behind it. Fixing the test in the previous commit
is what surfaced it.

Guard it on the workspace actually existing instead of deleting it. One
workflow file then stays correct in both layouts -- skipped on main, running
automatically once #139 lands -- so the two branches do not have to carry
divergent copies of ci.yml and merge them later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
yzsnstotz added a commit that referenced this pull request Aug 14, 2026
The final CI step runs `npm run test -w @meridian/orchestrator`, but main has
no `workspaces` key and no such package -- that layout exists only on
feat/meridian-monorepo-portability (#139). On main the step fails immediately
with "No workspaces found: --workspace=@meridian/orchestrator".

It arrived on main in a60fcef, which created ci.yml already written against
the post-#139 layout. Nobody noticed because the step is last and the suite
before it failed first, so every PR was already red for an unrelated reason
and this second failure sat behind it. Fixing the test in the previous commit
is what surfaced it.

Guard it on the workspace actually existing instead of deleting it. One
workflow file then stays correct in both layouts -- skipped on main, running
automatically once #139 lands -- so the two branches do not have to carry
divergent copies of ci.yml and merge them later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leo and others added 4 commits August 16, 2026 01:43
# Conflicts:
#	.github/workflows/ci.yml
#	package-lock.json
#	package.json
#	packages/runtime/src/hub/router.initialize-order.test.ts
…tials

Asking meridian who it is required a fully configured environment. With no
TELEGRAM_BOT_TOKEN, `meridian --help` and `meridian service doctor` both
died at startup — so anything that inspects an installation before a user
has onboarded secrets could not observe this tool at all, not even enough
to confirm it is installed.

Three separate places all made the same assumption: demand configuration
before looking at what the caller actually asked for.

- config.ts parsed at *import*. Nearly everything transitively imports it,
  so the process exited before main() ever read argv. It now parses on
  first read, behind a proxy so all 29 existing `config.x` call sites are
  untouched. Anything that genuinely needs configuration still fails, with
  the same message, at the moment it reads it.
- interface/index.ts and hub/instance-manager.ts each captured
  config.AGENT_WORKDIR into a module-level constant; both now read on use.
- meridian-cli.ts built the provider model catalog at module scope, and its
  constructor reads configuration.

requiresHubCallerIdentity defaulted to *true*, so every command it did not
recognise — including --version and --help — was treated as a hub call and
required a derived caller key. Local questions are now listed explicitly.

Also adds --version/-v, which did not exist: a tool that cannot state its
own version cannot be recognised as already installed, only reinstalled.

Verified with an empty environment (env -i, fresh HOME):
  --version        exit 0  meridian 1.0.0
  --help           exit 0  usage
  service doctor   exit 0  {"ok":true,...}
  health           exit 1  unchanged — it needs a running service

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ials

Let the tool identify itself before it has been given any credentials
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants