Skip to content

feat(workflow): add HITL steps and manual run controls - #1205

Merged
jeff-r2026 merged 1 commit into
Tencent:mainfrom
SaulMoro:feat/workflow-run-controls
Oct 19, 2026
Merged

jeff-r2026 merged 1 commit into
Tencent:mainfrom
SaulMoro:feat/workflow-run-controls

Conversation

@SaulMoro

@SaulMoro SaulMoro commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Workflows can now put a person in the loop (HITL), and a person can fix a run by hand. Closes #1081.

schemaVersion: v1alpha1
name: release
description: Pick a plan, approve the deploy, write the notes together.
steps:
  - id: plan
    choose: { message: Which plan?, options: [small, big], default: small }
  - id: deploy
    needs: [plan]
    approve: { message: "Deploy the {{ steps.plan.choice }} plan?", required: true }
    command: ./deploy.sh
  - id: notes
    needs: [deploy]
    interactive: true
    prompt: Write the release notes with me.
 teamai workflow
   start <name> [--driver local] [--inputs]
+               [--auto-approve]
   next <run> --json {driver, running, canTakeLease}
+                    driver.can.hitl          -> data.hitl {kind, message, options, default, required, respond}
   report <run> --json {...}
+  respond <run> <step> <choice> [--note] [--pass] [--json {driver}]
+  retry <run> <step>
+  cancel <run> [step]
+  resume <run> [--tools] [--auto-approve]

How next treats a HITL request:

Driver Optional approve Required approve Choose interactive
driver.can.hitl hitl request hitl request hitl request leased
without HITL, auto_approve yes fails hitl_required default, else hitl_required fails hitl_required
without HITL no, step skipped fails hitl_required fails hitl_required fails hitl_required

Who can answer a required request: respond from a TTY (channel tty), or a driver's own UI with respond --json (channel driver). Without a TTY, respond is channel cli and answers optional requests only.

src/workflow/
├── parse-hitl.ts     # approve / choose / interactive -> StepHitl
├── run-controls.ts   # respondToHitl, cancelRun, retryStep (new)
├── engine.ts         # leaseNext: hitl answers, table for drivers without HITL, run.driver
├── step-state.ts     # waiting status, answer per pass
├── local-driver.ts   # TTY question while polling, interactive foreground, driver lock
├── process-stop.ts   # one stopper for commands and tools
└── records.ts        # cancelled and hitl_required, answer provenance, retried ends replace stats

What a manual control changes:

retry <run> <step>
  free its lease or archive its pass, start a new pass
  re-pend its dependency cone and the skips in its containers
  a HITL request in the cone that waits, or was answered and has not run, is asked again at a new pass
  reopen enclosing workflow steps and a settled run
  remove a queued end, replace a published end and its stats
  revision += 1

cancel <run> [step]
  stop pending / waiting / leased / running work under it
  list invalidated attempts in next's cancelled
  whole run: status cancelled

The local driver asks HITL requests on its terminal and polls in the background, so an answer from another process closes the question. An interactive tool gets the terminal. Ctrl-C goes to the tool, as with teamai env exec. One local driver runs a run at a time.

Evidence

  • Before: on main, teamai workflow respond|retry|cancel|resume do not exist. A step can't ask a person, and a stopped run can only be started again from the beginning.
    After: the built CLI, run as separate processes for each command (npm run build && npm run test:e2e -- src/__tests__/e2e/workflow-run-controls.test.ts src/__tests__/e2e/workflow-conformance.test.ts src/__tests__/e2e/workflow-signals.test.ts):
✓ workflow-signals.test.ts (4 tests)
   gives Ctrl-C to an interactive tool and asks a HITL request only after the tool exits
✓ workflow-run-controls.test.ts (8 tests)
   opens a HITL request, answers from another process, and returns the guarded lease
   stops a long sleep from a second process and exits with cancelled
   retries a failed step, then resumes and runs its skipped dependent
   resumes in the original workspace after answering the first HITL request before any lease
   resumes from a subdirectory in the open local lease workspace
   supports process-only auto approval on start / on resume
   rejects invalid pass values and disallows auto approval in the JSON payload
✓ workflow-conformance.test.ts (32 tests, the same scenarios as the in-process client)
   HITL approve then respond and repeated response
   required approve refuses drivers without HITL and agent answers
   a driver UI answers a required HITL request
   interactive work requires a driver with HITL
   interactive hand-back refuses a driver that no longer has HITL
   retry failed work reopens its run and skipped dependents
   cancel leased work invalidates its report
   whole run cancel has its own outcome
   ...
Test Files  3 passed (3)
     Tests  44 passed (44)

Also passing: npx tsc --noEmit, npm run lint, and npx vitest run (11,191 passed, 21 skipped as on main). A table of retry and cancel from every step state (pending with an answer, waiting, leased, running, failed, cancelled, refused approve, answered choose) checks that no HITL answer outlives its pass, and that real commands at max_parallel 1 and 2 never run two attempts of one step at once, also when the shell cannot exec into the command (node work.js; true). A queued end that a retry replaces while an earlier end is being dequeued stays queued (the dequeue now holds the run lock). The advertised respond command quotes fan-out step paths such as 'items[0].ask' and runs as printed in zsh. retry of a reopened run returns resume: "teamai workflow resume <run>". A real-CLI cancel of sh -c 'sleep 30 & wait' at max_parallel: 1 leaves no sleep in ps, and the command can still read /dev/tty. commands.md was regenerated with npx vitest run commands-reference -u.

The new terminal test drives the built CLI in a pseudo-terminal with a fake tool that survives Ctrl-C. It goes red when the driver handles the terminal's Ctrl-C itself during an interactive step (the driver exits), and when the driver opens the HITL question while the tool runs.

A manual run of the same workflow with Claude Code as the interactive tool showed Ctrl-C reaching Claude ("Press Ctrl-C again to exit") with the driver still running. Codex and OpenCode TUIs, Windows taskkill, and the extra providers and agents are left to CI.

Merge Danger

Door: two-way

The new YAML keys, commands, and the hitl answer are additive. Run files and records written before this change still read: every new field is optional. External drivers that know nothing about HITL keep working. A HITL step is answered for them by the table above, and the guides describe the contract for drivers that want HITL.

Blast Radius: workflows

  • Headless prompt tools now run in their own process group, so cancel can stop the whole tool. A terminal Ctrl-C reaches the tool through the driver, which stops it with SIGTERM.
  • A cancelled max_parallel: 1 command stays in the terminal, so it keeps Ctrl-C and /dev/tty. The driver stops it together with every process it started, taken from one ps snapshot. SIGKILL goes only to captured processes whose start time still matches, so a reused PID is spared; lstart has one-second resolution, so this narrows that window rather than closing it. A process that daemonizes out of that tree is not reached. Without ps, it stops the shell alone, as on main.
  • Prompt steps marked interactive keep the repository changes the person makes. Headless prompts keep the read-only check.
  • A run that is retried after it ended publishes a second end. The reports branch and stats/<user>.yaml replace the first one instead of counting another run.

@github-actions

Copy link
Copy Markdown
  • [P1 blocking] Invalidate dependent HITL answers and passes on retry — src/workflow/run-controls.ts:201
    With plan choosing small/big and deploy requiring approval, answer small, approve deploy, then retry plan before deploy is leased. The pending branch preserves deploy’s old answer, so choosing big executes deploy without fresh approval. The waiting branch also preserves its pass number, allowing an old respond --pass 1 to approve the replacement request. Reset dependent HITL answers and advance their passes when retry invalidates their context.

Testing evidence is sufficient: the description records a build and representative real-CLI E2E runs. I only read the scoped diff; nothing from the PR was run, built, or installed.

@SaulMoro

Copy link
Copy Markdown
Collaborator Author

Fixed in 5a48151. A HITL request in the retry cone that is waiting, or answered and not yet run, now starts a new pass: the dependent approval is asked again in the new context, and an answer to the old pass gets stale_response. Both cases from the comment are tests in workflow-run-controls.test.ts.

@github-actions

Copy link
Copy Markdown
  • [P1 blocking] Wait for the invalidated command to exit before starting its retry — src/workflow/local-driver.ts:434
    Retry a running command whose SIGTERM handler performs cleanup before exiting. The next poll receives its replacement lease, moves the old process from inFlight to dying, and starts the replacement immediately: neither room() nor command startup counts dying. Even with max_parallel: 1, both attempts run concurrently and can modify the same files. Wait for cancellation cleanup before starting the replacement.

Earlier finding resolved: dependent HITL answers are cleared and waiting requests advance their passes on retry.

Testing evidence is sufficient: the description records a build and representative real-CLI E2E runs. I only read the scoped diff; nothing from the PR was run, built, or installed.

@SaulMoro

Copy link
Copy Markdown
Collaborator Author

Fixed in f15b628. Commands that are still exiting after a cancel now take a slot in the local driver, so a retried step starts its new pass only after the cancelled attempt has exited, at any max_parallel. The case from the comment, a SIGTERM cleanup longer than a poll at max_parallel: 1, is a test in workflow-local-driver.test.ts.

@github-actions

Copy link
Copy Markdown
  • [P1 blocking] Wait for the cancelled attempt even when parallel slots remain — src/workflow/local-driver.ts:505
    With max_parallel: 2 and one running command, retry it while its SIGTERM handler performs cleanup. The next poll moves the old attempt into dying, but inFlight.size + dying.size < maxParallel still permits its replacement to start immediately. Both attempts can modify the same files concurrently. Block the replacement until its cancelled predecessor exits; counting occupied slots only fixes the max_parallel: 1 case.

Earlier HITL invalidation finding resolved: dependent answers are cleared and replacement requests advance their passes. The command-overlap finding is only partially fixed, as described above.

Testing evidence is sufficient: the description records a build and representative real-CLI E2E runs. I only read the scoped diff; nothing from the PR was run, built, or installed.

@SaulMoro

SaulMoro commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fixed in 9fe51cc. A held lease now waits while an attempt of its own step is still exiting, so a retried command never starts beside its cancelled attempt at any max_parallel; other steps still use free slots. The test runs at max_parallel 1 and 2.

The earlier Ubuntu failure was that test hitting its 15 s timeout. It now runs the command through exec, so a cancel at max_parallel 1 reaches node whatever /bin/sh is, and it has a 30 s timeout (9cb3e39).

@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Retry clears dependent HITL answers and advances replacement request passes.
  • Replacement commands wait for their cancelled predecessor to exit, even when parallel slots remain.

Testing evidence is sufficient: the PR description records a build and representative real-CLI E2E runs. Deferred provider and agent coverage is not blocking.

I only read the scoped diff; nothing from the PR was run, built, or installed.

@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Retry clears dependent HITL answers and advances replacement request passes.
  • Replacement commands wait for their cancelled predecessor to exit, even when parallel slots remain.

Testing evidence is sufficient: the PR description records a build and representative real-CLI E2E runs. Deferred provider and agent coverage is not blocking.

I only read the scoped diff and trusted base review rules; nothing from the PR was run, built, or installed.

@SaulMoro
SaulMoro force-pushed the feat/workflow-run-controls branch from d99cdd0 to 3659a18 Compare October 18, 2026 17:00
@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Retry clears dependent HITL answers and advances replacement request passes.
  • Replacement commands wait for their cancelled predecessor to exit, even when parallel slots remain.

Testing evidence is sufficient: the PR description records a build and representative real-CLI E2E runs. Deferred provider and agent coverage is not blocking.

I only read the scoped diff and trusted base review rules; nothing from the PR was run, built, or installed.

A workflow step can now ask a person: `approve` (yes or no, `required`
for answers that must come from a person), `choose` (options with an
optional default, readable as `steps.<id>.choice` and `.note`), or
`interactive: true` (a person works in the tool). `next` returns a
`hitl` request to a driver that declares `driver.can.hitl`; a driver
without HITL gets the auto-approve table or `hitl_required`.

New `teamai workflow respond`, `retry`, `cancel` and `resume` change a
run by hand. Retry reopens a step and what depends on it as a new pass,
reopens a settled run, replaces its published end and stats, and
prints the `resume` command when the run needs a driver again.
Cancel stops open work and invalidates its leases. A driver with its
own UI answers through `respond --json` with channel `driver`.

The local driver asks HITL requests on its terminal while it polls for
answers from other processes, gives the terminal to interactive tools,
stops cancelled tools and every process a cancelled command started,
starts a retried step only after its cancelled attempt has exited, and
holds one driver lock per run.

Closes Tencent#1081
@SaulMoro
SaulMoro force-pushed the feat/workflow-run-controls branch from 3659a18 to 1c183af Compare October 18, 2026 17:54
@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Retry clears dependent HITL answers and advances replacement request passes.
  • Replacement commands wait for their cancelled predecessor to exit, even when parallel slots remain.

Testing evidence is sufficient: the PR description records a build and representative real-CLI E2E runs. Provider and agent coverage deferred to CI is not blocking.

I only read the scoped diff and trusted base review rules; nothing from the PR was run, built, or installed.

@jeff-r2026
jeff-r2026 merged commit 8dc60b0 into Tencent:main Oct 19, 2026
14 checks passed
@SaulMoro
SaulMoro deleted the feat/workflow-run-controls branch October 19, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat] Add manual run controls and human gates

2 participants