feat(executor): parallel/MPI jobs + real SLURM sbatch + connector-as-dispatch (IBM PE codified) - #24
Merged
Merged
Conversation
…dispatch (IBM PE codified)
Answers "what job types", "how do we leverage SLURM", and "the MCP-connector pattern" — one
Grant-gated executor now dispatches batch, parallel, stateful, and connector work alike.
- Parallel / MPI (the IBM Parallel Environment POE pattern): K8sAdapter emits an Indexed Job
(completions/parallelism=N, completionMode=Indexed) — each task gets JOB_COMPLETION_INDEX as
its rank. `sourceosctl run --parallelism N --nodes M`.
- Real SLURM: SlurmAdapter emits a real sbatch script (#SBATCH --ntasks/--nodes/--gres, `srun`
to launch MPI ranks, grant-bound via --comment), and submits it via `ssh $SOURCEOS_SLURM_LOGIN
sbatch` when configured (else emits the script). Replaces the hpc-slurm descriptor stub.
- Connector-as-dispatch: ConnectorAdapter + a `connector` backend (external/untrusted, so
sensitive data won't auto-route to a vendor). A Gemini/OpenAI/Claude Files-API / MCP call is
the SAME materialize->handle->dispatch->result shape as a compute job, gated by the same Grant.
Also fixes a latent bug: `place` reused cmd_run but its parser omits the run-only flags
(--cost/--inception/--parallelism) -> AttributeError; cmd_run now reads them defensively.
Tests: +1 indexed-job +1 sbatch +1 connector +1 connector-plane +1 place-regression = 110 green.
docs/PATTERN_INTEGRATION.md updated (these three frontiers now shipped; volunteer-mesh is next).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One Grant-gated executor now dispatches batch, parallel/MPI, stateful, and connector work alike.
JOB_COMPLETION_INDEX=rank);sourceosctl run --parallelism N --nodes M.sbatchscript (--ntasks/--nodes/--gres,srunfor MPI ranks, grant-bound) and submits viassh $SOURCEOS_SLURM_LOGIN sbatchwhen configured.connectorbackend (external/untrusted). A Gemini/OpenAI/Claude Files-API / MCP call is the same materialize→handle→dispatch→result shape as a compute job, gated by the same Grant.placebug (cmd_run read run-only flags → AttributeError; now defensive).110 tools tests green.
docs/PATTERN_INTEGRATION.mdupdated.