Skip to content

fix(core)!: apply the service manifest to any plugin list and pick job queues per process - #298

Merged
zaxovaiko merged 27 commits into
devfrom
feat/service-split-seams
Nov 5, 2026
Merged

zaxovaiko merged 27 commits into
devfrom
feat/service-split-seams

Conversation

@mp-blurify

@mp-blurify mp-blurify commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

createApp applies SERVICE_MANIFEST to whatever plugin list it is given and names everything a manifest leaves out in one error with one manifest line that boots, refuses to boot a split that would skip a money, compliance or authorization control, logs each other optional port it leaves unbound, and a new JOB_WORKERS env var picks the job queues a process consumes. Also fixes the create:service scaffold and records a follow-up ADR.

Why

  • The manifest only worked through loadExtensions(). createApp read SERVICE_MANIFEST just to require SERVICE_NAME, so a host that passes its own plugin list (the downstream guide shows exactly that) booted every module under the split service's own consumer group: every broker event was handled once more across the cluster. Verified on a real consumer: the "split" process loaded every module and created its group on ~150 streams.
  • What a manifest missed surfaced one kind per boot, and the rule that a manifest lists its modules' dependencies had been dropped from the docs. Dependencies, the modules that bind required ports, the modules behind controls and AdminGuard's modules now fail the boot once, in one error naming each, with one manifest line that also adds what those modules need in turn (transitively, infra overlays' dependencies included). SERVICE_MANIFEST=wallet gets a line that boots on its first boot.
  • A split service lost controls silently. Modules resolve some ports with c.has(), so wallet,identity,mail,audit took deposits and bets past the RG limits, signed players in without the country check and sent regulatory mails without an audit record, and its admin routes fell back to static grants without iam - an admin whose role was revoked could still approve a withdrawal. The boot said nothing. Now a manifest boot fails closed: a plugin lists a port that backs a money, compliance, authorization or audit control in controlPorts, with the module that binds it, and the boot refuses while one is unbound and the plugin list has that module, printing the manifest line to paste. AdminGuard holds ADMIN_PERMISSION_RESOLVER (iam) and ADMIN_SECURITY_POLICY (identity) to the same rule on every manifest boot, admin routes or not, and a manifest boot builds the guard before it serves. Declared: RG_LIMITS (wallet, gaming), GAME_GEO_CHECK (gaming), GEO_CHECK_COMMANDS (identity, bonus, notifications), MCP_TOKEN_REVOCATION (identity), AUDIT_WRITER (mail). A full boot, and an install whose list lacks the binding module, only log them, as before. A port whose absence only degrades a feature or already fails closed (no KYC policy, no risk tags) is not a control port.
  • Every process consumed every queue. A job that runs for minutes and mostly waits on an outside service (an agent run waiting on a model) took a share of every API replica. It cannot move to a manifest service: that gets its own copy of every event, and the in-process MCP kernel needs every module that contributes tools. JOB_WORKERS (names, prefix*, ! exclusions; unset = all) lets one more full instance with the same SERVICE_NAME take such queues while the replicas skip them. A malformed value or an inclusion that matches no queue fails the boot; the boot log lists consumed and skipped queues. ADR-0046 (Proposed).
  • pnpm create:service generated a host that could not boot: no SERVICE_NAME, and an example manifest that left out what its modules need. Its example is now the line the first boot of SERVICE_MANIFEST=wallet suggests, which an E2E boots. It now also warns that a module two services load handles each event in both, until ADR-0047 lands.
  • ADR-0047 (Proposed, not implemented): modules a split service loads only as dependencies still wire their handlers under the service's own group (audit, for one, would write each row once per service); the proposal is that only selected modules consume. It is a follow-up outside this PR.

Alternatives considered

  • Consumers call loadExtensions(): it rewrites relative paths, so it breaks hosts that list bare package specifiers.
  • A process role (SERVICE_ROLE=api|worker): fixes one topology and cannot name queues. Role metadata per worker registration: a module cannot know how it is deployed. A worker-only process without routers: router factories are boot hooks workers rely on (ADR-0038). Details in ADR-0046.

Risks

  • Behaviour change (!): a process that sets SERVICE_MANIFEST with a static plugin list now boots that subset, or refuses to boot over missing dependencies or a skipped control. With the shipped modules every split carries iam, identity, audit, mail, compliance, wallet, gaming, player-management, exchange-rate and chat, and until ADR-0047 lands their handlers run in the split and again in the rest of the install. When a manifest boot fails, each listed plugin the boot had not registered runs its register() once, into a container of its own that is then dropped, to learn which module binds which port; register() must only declare (documented on Plugin.register), and the failed boot disposes its container. Consumer groups the old full boot created under a split SERVICE_NAME are stale (XGROUP DESTROY). Merges to dev publish a canary immediately.
  • JOB_WORKERS needs operator care: every queue needs a consumer somewhere, and processes that share a SERVICE_NAME must load the same modules; core cannot check either across processes. Unset changes nothing.
  • Overlapped with feat(core): opt-out anonymous usage telemetry #295 and fix(core): deliver domain events at least once #297 in create-app.ts, ADR-0017 and the messaging rule. Both have merged and dev is merged here; they took ADR-0044 and ADR-0045, so this PR's ADRs are ADR-0046 (job worker selection) and ADR-0047 (dependency-only modules).

Comment thread tools/create/create-service.ts Resolved

@zaxovaiko zaxovaiko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Traced every createApp caller and every ctx.jobs.worker registration (core plus the betfeel overlays), all of them go through the new selection, and schedules still land after it. Minor:

  • create:service accounts identity,mail still double-handles: the monolith boots identity too, so its 3 session handlers run twice, and the suggested audit override writes every audit row twice. Maybe worth a warning in the scaffold output until ADR-0045 lands?
  • Let's give the agent-hub queues one shared prefix, so the split is JOB_WORKERS=agent-hub* on the worker and !agent-hub* on the API without listing each queue.
  • Nothing checks across processes, so a dead worker means a queue that silently backs up. We'll need a queue-depth alert next to the deploy change. But that's for later.
  • ADR-0044 collides with #295 and #297.

Mateusz Paś and others added 7 commits October 14, 2026 10:08
The generated host set SERVICE_MANIFEST but not SERVICE_NAME, which createApp requires
with a manifest, so it never booted; its example manifest (identity,wallet) also left out
wallet's dependencies. It now bakes the service name as SERVICE_NAME, passes the identity
tables as authSchema like the consumer template, uses identity,mail as the example, and
drops dependencies its main.ts never imports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
…ssing dependencies

SERVICE_MANIFEST was applied only by loadExtensions(), while createApp read it just to
require SERVICE_NAME. A host that passes its own plugin list therefore booted every module
under the split service's own consumer group, so every broker event was handled once more
across the cluster.

createApp now selects the manifest's modules from whatever list it is given, and
loadExtensions() no longer filters. A manifest that leaves out a module the selected ones
depend on - directly or through another left-out module, infra overlays' dependencies
included - fails the boot with one error naming them all and a manifest line to paste. The
rule that a manifest lists its modules' dependencies and port providers is back in the
messaging rule.

BREAKING CHANGE: a process that sets SERVICE_MANIFEST and passes its own plugin list now
boots that subset, or refuses to boot over missing dependencies, instead of every module.
Consumer groups the old full boot created under the split SERVICE_NAME are stale; remove
them with XGROUP DESTROY.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
Every process consumed every queue, so a job that runs for minutes and mostly waits on an
outside service took a share of every API replica. A manifest service cannot take it over:
its own SERVICE_NAME gives it its own copy of every event, and a handler that needs the
in-process MCP kernel needs every module that contributes tools.

JOB_WORKERS picks the queues a process consumes: names, prefix* patterns and !exclusions;
unset is every queue, as before. createApp applies it when it registers workers, so it holds
for any queue driver, and logs the queues it consumes and skips. A malformed value, or an
inclusion that matches no registered queue, fails the boot; an exclusion that matches
nothing is allowed, since the module behind it may be absent. To keep a queue off the API
replicas, run one more full instance with the same SERVICE_NAME and JOB_WORKERS=<queues>,
and give the replicas JOB_WORKERS=!<queues>. ADR-0044 (Proposed) records the decision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
A split service loads the modules its own depend on, and those still wire their event
handlers, workers, schedules and routers - under the service's own consumer group, so
audit's handlers, for one, would run once per service. ADR-0045 (Proposed) records the
direction - only the modules a manifest selects consume - for a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
The JOB_WORKERS selection copied the registered workers before the router factories ran, so a
worker registered from a factory never started, even with JOB_WORKERS unset. The selection now
runs at the registration loop, after the factories; an inclusion that matches no queue still
fails the boot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
identity and mail resolve some ports with c.has() and run without them, so a split service of
identity and mail skipped the country check on sign-in and sent regulatory mails without an
audit record, and nothing at boot said so. A plugin now lists such ports in optionalPorts with
what it skips without each, and loadPlugins logs a warning for every one left unbound. identity
declares the geo check, player provisioning and MCP token revocation; mail the audit writer and
play eligibility.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
A generated service and the monolith both load identity, so identity's handlers ran in both,
and the override the scaffold suggested added audit, which would write every audit row twice.
The scaffold now says so after it generates the host, points at the boot log's optional-port
warnings, and no longer suggests adding audit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
@mp-blurify mp-blurify changed the title fix(core)!: apply the service manifest to any plugin list and pick job queues per process (BF-739) fix(core)!: apply the service manifest to any plugin list and pick job queues per process Oct 14, 2026
@mp-blurify
mp-blurify force-pushed the feat/service-split-seams branch from 3dedd4c to beea7f2 Compare October 14, 2026 08:37
@mp-blurify

Copy link
Copy Markdown
Collaborator Author

Thanks, @zaxovaiko.

  • Scaffold: it now warns after generating that a module two services load handles each event in both (audit writes every row twice), points at the boot's optional-port warnings, and no longer suggests adding audit (beea7f2).
  • Shared prefix: agreed. That's a rename in the overlay that registers those queues; JOB_WORKERS already takes prefix*.
  • Queue-depth alert: agreed, with the deploy change. ADR-0044 lists it.
  • ADR-0044: feat(core): opt-out anonymous usage telemetry #295 and fix(core): deliver domain events at least once #297 are still open, so whichever lands later renumbers.

@zaxovaiko

Copy link
Copy Markdown
Member

/codex all

@blurify-codex blurify-codex Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: a service manifest can boot money and admin routes without their compliance and permission controls. Blocker: split services can duplicate audit records when they load shared dependency modules.

Reviewed security, compliance, conventions, performance, reliability, rollout, spec, the whole change.

zaxovaiko
zaxovaiko previously approved these changes Oct 15, 2026

@zaxovaiko zaxovaiko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Under SERVICE_MANIFEST a split service could boot without the module behind
a money, compliance or authorization control and skip it in silence: wallet
without compliance took deposits and bets past the RG limits, and admin
routes without iam fell back to the static role table, so an admin whose
role was revoked in the database could still approve a withdrawal.

A plugin now lists such ports in controlPorts, with the module that binds
each. A full boot logs an unbound one, as before. A manifest boot refuses to
start while one is unbound and the plugin list has its module, naming the
port, what its module skips and a manifest line that adds the binding
modules and what they need in turn. AdminGuard holds ADMIN_PERMISSION_RESOLVER
(iam) and ADMIN_SECURITY_POLICY (identity) to the same rule once admin routes
resolve it. An install whose plugin list lacks the module has the control
nowhere and boots as before.

Declared: RG_LIMITS (wallet, gaming), GAME_GEO_CHECK (gaming),
GEO_CHECK_COMMANDS (identity, bonus, notifications), MCP_TOKEN_REVOCATION
(identity) and AUDIT_WRITER (mail). Ports whose absence already fails closed
- no KYC policy, no risk tags, no bonus grants, no play eligibility - are
left as they are.

The container now clears a token's resolving mark when its factory throws,
so a refusing factory refuses again with its own error rather than as a
circular dependency.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>

@zaxovaiko zaxovaiko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite yet - the scaffold example still doesn't boot, and fixing a manifest takes up to three boots. The AI side looks right to me: core gets no runtime or model code, and JOB_WORKERS is a generic seam the hub can use. Minor:

  • The MCP_TOKEN_REVOCATION control also refuses a split while agents.mcp.enabled is off. Admin routes need iam anyway, so that's for later.

Comment thread tools/create/create-service.ts Resolved
Comment thread docs/guides/downstream-consumer.md Resolved
Mateusz Paś and others added 2 commits October 16, 2026 12:28
A manifest boot found what it missed in three places: dependsOn before
loading, control ports after configure(), and AdminGuard's ports only when a
router factory resolved the guard. SERVICE_MANIFEST=wallet took three boots
to fix. A split that served no admin routes booted without iam and then
failed every request that resolved the guard lazily (MCP authorize, for
one), since the container does not cache a factory that throws.

A manifest boot now checks dependencies before loading and the rest once
its modules registered and configure() ran: required ports, the selected
modules' controlPorts, and AdminGuard's ports on every manifest boot,
admin routes or not. Whatever it finds fails the boot with one error that
names each missing module and why, and one "Add them" line that also adds
what those modules need in turn - their dependencies, the binders of their
required ports and the modules behind their controls - so one edit fixes
the manifest. The guard is then built once, at boot.

Which module binds a required port is learned only after the boot has
failed, by registering each plugin of the list into a container of its own.
A plugin whose register() throws there - a module this process does not
configure - counts with what it bound before the throw, and the error names
it when that leaves a required port without a known binder.

loadPlugins() takes requiredPorts: 'defer' so a manifest boot can fold
required ports into that error; a full boot asserts them as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
The usage example, identity,mail, did not boot: identity's controls pull in
compliance and iam, mail's pulls in audit, and compliance depends on wallet,
gaming, player-management and exchange-rate. The example now lives in
tools/create/service-example.json, which the scaffold prints, and is the
manifest the first boot of SERVICE_MANIFEST=wallet tells to use.

An E2E boots it against the real plugin list and checks that the wallet
boot still suggests exactly it, so a module that gains a dependency or a
control fails the test instead of leaving the example stale. It keeps its
app open across its tests, as the other suites do: BullMQ disconnects a
connection still initialising when its queue closes, rejecting commands it
never awaited, so an app closed straight after boot leaks unhandled
rejections.

The scaffold's warning now says every split loads iam and identity, and
what follows from them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
@mp-blurify
mp-blurify requested a review from zaxovaiko October 16, 2026 10:29
# Conflicts:
#	packages/core/src/server/plugin-host/define-plugin.ts
#	packages/core/src/server/runtime/__tests__/create-app.test.ts
#	packages/core/src/server/runtime/create-app.ts

@jakubfilinger-b jakubfilinger-b left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall the manifest/JOB_WORKERS logic reads correct to me (parser edge cases look handled). Two notes inline/below: the double register() and the ADR-0044 number clash with #297.

@jakubfilinger-b

Copy link
Copy Markdown
Collaborator

Heads-up: this and #297 both claim ADR-0044 (as the description notes), so whichever merges second needs to renumber its ADRs.

…ose what it opened

A failed SERVICE_MANIFEST boot registered every listed plugin a second time, each into a
container of its own, to learn which module binds which port, so a register() with an effect
outside the registry would act differently on that path. No core plugin's does: each only
declares into the registry. mail and identity read a secret and throw, which the boot error
already reports. audit, chat and notifications built their logger inside register(); it now
sits at module scope, as in the other modules. A downstream plugin can still open a connection
or start a timer there, and the contract was written down nowhere.

- The boot error no longer registers the plugins the boot already registered: what they bound
  is in the process's container, which the check reads. Every other plugin registers once, so
  no register() runs twice in one process.
- createApp disposes its container when the boot fails. The caller gets no app to close, so a
  refused manifest boot left the Redis client createApp opened, and any connection configure()
  bound, connected and reconnecting, which keeps alive a host that does not exit on the error.
- Plugin.register states the contract: declare only, and open a resource in a provide() or
  router factory that closes it with onDispose(). ADR-0044, the plugin-author rule, the
  scaffold commands and template say the same, and the error-tracking guide's example, which
  initialised the vendor SDK in register(), now does it in the factory.
- A unit test runs every core plugin through the refused boot's registration pass and fails on
  a timer, an open handle, a process listener, a global or a process.env change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
@mp-blurify

Copy link
Copy Markdown
Collaborator Author

@jakubfilinger-b on the ADR numbers: agreed. #297 is still open and dev's latest is 0043, so I'm leaving them as they are for now. Whichever merges second renumbers. If #297 lands first, this PR's 0044 and 0045 become 0045 and 0046, and every cross-reference moves with them: the ADRs themselves, the messaging rule and AGENTS.md, the docs, and the plugin-host and scaffold notes that cite ADR-0044.

…and 0046

dev took ADR-0044 for usage telemetry, so check:refs failed on the
duplicate number after the merge.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mp-blurify and others added 3 commits October 20, 2026 13:55
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
…elivery adr

dev took ADR-0045 for at-least-once event delivery, so the job worker
selection ADR becomes ADR-0046 and the dependency-only modules ADR
becomes ADR-0047. References that mean the delivery ADR keep 0045.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
@mp-blurify

Copy link
Copy Markdown
Collaborator Author

Merged dev into this branch, so it now builds on #297's at-least-once event delivery. #297 took ADR-0045, so this PR's ADRs are renumbered to ADR-0046 (per-process job worker selection) and ADR-0047 (dependency-only modules provide but do not consume), with every cross-reference updated. The only conflicts were ADR-0017's update notes (both kept) and the imports in create-app.test.ts; the merge also left the audit plugin's logger unused, so it is removed.

Mateusz Paś and others added 8 commits October 26, 2026 13:17
# Conflicts:
#	tools/create/create-service.ts
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>

# Conflicts:
#	.rulesync/rules/messaging-and-microservices.md
#	AGENTS.md
#	packages/core/src/server/plugin-host/load-extensions.ts
…e sportsbook ones

dev took ADR numbers 0046 and 0047 for the sportsbook ledger port and provider promotions; this branch's two ADRs become 0048 and 0049.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
dev's SERVICE_MANIFEST entries carry a no-routes flag and a split now needs REALTIME_NAMESPACE: the boot check takes the entries, keeps the flags in the manifest line it suggests, and the tests set the namespace beside the service name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
Dev took ADR-0048 for claimable rakeback, so this branch's ADRs move to
ADR-0049 (per-process job worker selection) and ADR-0050 (dependency-only
modules), with every cross-reference updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Mateusz Paś <mateusz.pas@studiosoftware.com>
@jakubfilinger-b

Copy link
Copy Markdown
Collaborator

Heads-up: this PR and #372 both introduce an ADR numbered 0049. Whichever merges second will need to renumber its ADR and update its references.

@jakubfilinger-b jakubfilinger-b left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed. The only change of its own since my approval is 8bd1962 (boot check reads the flagged manifest entries, tests set REALTIME_NAMESPACE), which looks right; the rest is merges of dev. The ADR numbers are settled now.

@zaxovaiko
zaxovaiko added this pull request to the merge queue Nov 5, 2026
Merged via the queue into dev with commit e19e8ce Nov 5, 2026
7 checks passed
@zaxovaiko
zaxovaiko deleted the feat/service-split-seams branch November 5, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants