Skip to content

feat(jobs): run jobs side by side when the extension allows parallel jobs - #3

Merged
Nagi-ovo merged 13 commits into
voyager-crew:mainfrom
ccch1mneyyy:feat/parallel-jobs
Oct 24, 2026
Merged

Nagi-ovo merged 13 commits into
voyager-crew:mainfrom
ccch1mneyyy:feat/parallel-jobs

Conversation

@ccch1mneyyy

Copy link
Copy Markdown
Contributor

Why

voyager serve ran one job at a time, so several agents asking ChatGPT Pro through one browser queued behind each other's hour-long replies. With the companion Voyager change, the person can turn on Run jobs in parallel in the popup's Local bridge card; serve then runs that many jobs at once (up to 10, or fewer with "maxParallelJobs" in ~/.voyager/config.json). Switch off, or an older extension: nothing changes, one job at a time and busy (exit 5) for the next.

Companion PR (extension + @voyager-crew/bridge-protocol 0.5.0): voyager-crew/voyager#1067

What changes

submit ── admission.ts ──▶ busy?  capacity = min(extension maxTasks, maxParallelJobs)
                                   conflict key: same conversation / any project change
                                   voyager call runs only alone (it borrows a free tab)
job ── OpsClient.call(op)
        ├─ page op (open, select, attach, send, approve, project.*, close)
        │     └─ PageTurns.take(job) … slot.run … slot.idle(job) → give back
        └─ waitReply / read / download → slot.run directly (overlaps other jobs)
      ask: ops.exclusive(model.select → composer.attach → composer.send)
TaskSlot: one entry per job; a poll settles every entry, then sends one message
          (an abort first, then the oldest task due)
  • JobRunner keeps a map of active jobs; voyager status lists activeJobs and maxJobs ("parallel jobs: up to 10 at once" / "off").
  • PluginState.update() serializes read-modify-write per plugin, so project-upload fingerprints of jobs side by side no longer overwrite each other.
  • contrib/web-pro-kit: drive_request.py and drive_mirror.py push take a lock on this computer ($VOYAGER_HOME/locks/, never in Drive) for what they publish; two agents could otherwise lose a request or leave a mirror whose MIRROR.json names the wrong archives. drive_wait.py already watches one task folder each; the runbook now says one task ID and one waiter per job, and one mirror per commit.
  • README, README_ZH, the agent skill and CONTRIBUTING describe the switch, the cap and the conflict rule.

Before merge

This depends on @voyager-crew/bridge-protocol 0.5.0 (maxTasks, activeJobs, maxJobs), which is staged by the Voyager PR. package.json already asks for ^0.5.0, and bun.lock is left at 0.4.0 until 0.5.0 is on npm, so CI's bun install --frozen-lockfile fails until then: after publishing, run bun install and commit the lockfile. Kept as a draft for that reason.

Testing

  • bun run verify (format, lint, typecheck, 333 tests) and bun run build, against the unreleased protocol package via a local file: link.
  • New tests: test/jobs/parallel.test.ts (capacity from maxTasks, maxParallelJobs, conflict keys, voyager call alone, page ops never overlap across jobs while replies do; mutation-checked by disabling the page turns), test/jobs/pageTurns.test.ts, slot tests for several jobs, serialized plugin-state updates. The fake extension now runs several tasks at once and records a timeline.
  • Kit: every script's --selftest passes; drive_request.py's new two-writer check fails without the lock (Version: 2) and passes with it.
  • Live, Chrome 155 with the Voyager branch build and this serve: three voyager ask --model extra-high at once all succeeded, awaited together, each reply carrying only its own token; a second ask into the same conversation was busy with error.details.conflict; with the switch off the second of two asks was busy (maxJobs: 1). Not tried live: GPT-6 Pro itself (quota) and more than three jobs at once.

ccch1mneyyy and others added 2 commits October 22, 2026 17:06
…jobs

voyager serve ran one job at a time, so several agents asking ChatGPT Pro
through one browser queued behind each other's hour-long replies. When the
person turns on "Run jobs in parallel" in the Voyager popup, the extension
reports maxTasks above one with each poll, and serve now runs that many
jobs at once (up to 10), or fewer with "maxParallelJobs" in config.json.
With the switch off, or an older extension, nothing changes.

- Each job holds its own task slot; a poll still carries one message, so
  the tasks of several jobs go out on successive polls.
- Ops that type, click, open or close a tab take turns (PageTurns), since
  the extension runs one of those at a time across all its tabs; replies
  are awaited at once. An ask keeps its turn from the model choice to the
  send, so another job's model choice cannot slip in between.
- Two asks into one conversation, or two project changes, never run
  together: the later one is busy with error.details.conflict. A raw
  voyager call runs only alone, as it borrows whichever tab is free.
- Plugin state gets a serialized update(), so project upload fingerprints
  of jobs side by side no longer overwrite each other.
- voyager status lists every active job and maxJobs.

Needs @voyager-crew/bridge-protocol 0.5.0 (maxTasks, activeJobs, maxJobs);
bun.lock follows once it is published.

Co-Authored-By: Claude <noreply@anthropic.com>
…e by side

With parallel jobs, two agents can steer the same task or publish the same
mirror at once. drive_request.py read the version and wrote N+1, so two
writers lost a request; two drive_mirror.py pushes of one mirror each
deleted the volumes the other wrote and left a MIRROR.json for the wrong
archives. Both now hold a lock on this computer for the file or mirror
they publish (local_lock.py, under $VOYAGER_HOME/locks, never in Drive;
the system frees it when the process ends). The runbook says how to run
tasks side by side: one task ID and one waiter per job, and one mirror
per commit.

Co-Authored-By: Claude <noreply@anthropic.com>
Nagi-ovo and others added 10 commits October 23, 2026 01:16
A job waiting for another job's turn at the page waited for as long as
that job held it, and a stuck holder kept every sibling waiting forever.
PageTurns.take() now refuses a job that is already cancelled, drops its
abort listener once the turn is granted, and fails a wait of more than
five minutes as busy before the step goes out, naming the job that holds
the turn (error.details.pageTurn). Closing the tab at the end of a job
waits one minute at most: the job's outcome is settled by then, so a
holder that does not let go leaves only a tab open.

voyager status names the job that has the page turn and the jobs waiting
for it (activeJobs[].pageTurn in JSON).

Co-Authored-By: Claude <noreply@anthropic.com>
Conflict keys came only from a job's input at submit, so a new ask (no
--chat) held no key, and once it had sent and returned its conversation
id, an `ask --chat <that id>` was admitted beside it and typed into the
same conversation from a second tab. A job now also holds its
conversation's key from its first checkpoint that names a conversationId
until it ends; the key is built in one place (conversationConflictKey).

Co-Authored-By: Claude <noreply@anthropic.com>
A maxParallelJobs of 0, a negative or fractional number, or a string was
dropped as if unset, so a person who meant to limit parallel jobs got as
many as the extension allows. Such a value now counts as 1, and serve
logs it once (again only after it changes).

Co-Authored-By: Claude <noreply@anthropic.com>
serve stored an absent maxTasks as 1, so voyager status told the person
of an extension without parallel jobs to turn them on in the popup. An
absent maxTasks now stays null (jobs still run one at a time, busy and
exit 5 as before), status reports it as extension.maxTasks, and voyager
status says which case it is: not in this Voyager version, off, or up to
N at once (and when maxParallelJobs holds it below the popup's number).
Unpaired, or with the extension away, it says nothing about it.

Co-Authored-By: Claude <noreply@anthropic.com>
With jobs side by side, busy has three causes: a conflict on the same
conversation or project, every slot taken, or a voyager call that runs
only alone. The note now gives the server's reason and the job to wait
for; with one job at a time it keeps its earlier words.

Co-Authored-By: Claude <noreply@anthropic.com>
2.2.1 does not have the parallel switch, so "versions after 2.2.0" sent
people looking for it. The popup also lets the person choose how many
jobs run at once (2 to 10) rather than always ten; a job that waits more
than five minutes for its turn at the page stops as busy; and voyager
status names the job acting on its page.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
drive_mirror.py --selftest pushed test mirrors under whatever
VOYAGER_HOME was set, so a plain run left lock files in the person's
~/.voyager/locks. It now runs with its own temporary VOYAGER_HOME. The
lock folders are created 0700 like ~/.voyager itself (and tightened when
an earlier kit created them with the default mode), since their file
names come from local paths.

Co-Authored-By: Claude <noreply@anthropic.com>
A flat five-minute wait failed a job whose sibling held the turn for a
step that may legitimately run longer, such as an --attach upload of up
to 10 minutes. The limit now tracks the holder: while it runs an op, the
jobs waiting behind it wait for that op's own deadline (defaultOpTimeout
or the plugin's timeoutMs, the same deadline the task slot enforces)
plus 60 s of grace; between its ops it has the grace alone, and each new
holder starts afresh. Only a holder past that fails the waiting jobs as
busy before their op goes out, naming it and the op it is stuck in
(error.details.holderOp). Cancelling a waiting job and the one-minute
wait for closing the tab stay as they were.

Co-Authored-By: Claude <noreply@anthropic.com>
… step may take

The five-minute limit is gone: a job now stops as busy only when the job
holding the page is stuck past the time its step may take.

Co-Authored-By: Claude <noreply@anthropic.com>
@Nagi-ovo
Nagi-ovo marked this pull request as ready for review October 24, 2026 16:03
Co-Authored-By: Claude <noreply@anthropic.com>
@Nagi-ovo
Nagi-ovo merged commit fa4b710 into voyager-crew:main Oct 24, 2026
1 check passed
@Nagi-ovo

Copy link
Copy Markdown
Member

Thanks so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants