Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/main/daemon/serialize-grid-transcript-replay.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ const KNOWN_PREEXISTING_I2_FAILURES: Record<string, number> = {
less: 6,
nano: 2,
opencode: 5,
// OpenCode boot captures: rows on its true-colour background, and the row a shrink leaves
// behind, restore inexactly after a resize. Pre-existing: they landed with no serializer change.
'opencode-1-18-32-timed-boot-hidden-pane': 16,
'opencode-1-18-32-timed-boot-slow': 16,
'opencode-1-18-32-timed-first-launch': 16,
'opencode-2-0-18-timed-boot-hidden-pane': 14,
Comment on lines +38 to +43

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

rg -n -C 10 'new-fail|KNOWN_PREEXISTING_I2_FAILURES' src/main/daemon/serialize-grid-transcript-replay.test.ts
rg -n -C 8 'I2|checkpoint|new-fail' src/main/daemon/serialize-grid-roundtrip.ts
sed -n '180,245p' src/main/daemon/serialize-grid-transcript-replay.test.ts

Repository: stablyai/orca

Length of output: 7757


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- replay test: setup and assertion path ---'
sed -n '1,80p' src/main/daemon/serialize-grid-transcript-replay.test.ts
sed -n '150,230p' src/main/daemon/serialize-grid-transcript-replay.test.ts
printf '%s\n' '--- roundtrip classification ---'
sed -n '296,345p' src/main/daemon/serialize-grid-roundtrip.ts
printf '%s\n' '--- PR diff for the reviewed file ---'
git diff --no-ext-diff --unified=25 8cc28545a2514cf14cd25187b9bd24a6d9e03ba1 5ff7d779449c6f97aedce9f3092bad2583eb7c23 -- src/main/daemon/serialize-grid-transcript-replay.test.ts

Repository: stablyai/orca

Length of output: 12143


Assert OpenCode I2 failure identities, not only their count.

For the default two-seed run, each non-null gridDiff.new adds one new-fail. The OpenCode entries compare only the aggregate count. A different I2 mismatch can therefore replace the known resize mismatch and still pass. Store and compare the expected failureSignatures for each OpenCode transcript.

// Shrink leaves the cursor one column short; also present in the pre-Qoder serializer.
'qoder-no-account': 2,
'qoder-ready': 2,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"capturedAt": "2026-09-30T08:40:33.073309Z",
"platform": "darwin",
"command": ["opencode"],
"cols": 120,
"rows": 40,
"note": "1.18.32 on macOS 26.6.2 arm64, 120x40, alternate screen. Why this run: answered like a hidden Orca pane (silent on OSC colour queries), as a background worker pane is. Recorded with a Python PTY driver rather than the repo recorder, because the driver stamps every read with its time and answers the terminal queries OpenCode sends; responder 'headless' = Orca's hidden-pane replies, silent on OSC colour queries. Isolated HOME and XDG dirs, OPENCODE_DISABLE_AUTOUPDATE=1, OPENCODE_CONFIG_CONTENT {\"autoupdate\":false}, no model configured, env -i (no ORCA_*); version and binary sha256 unchanged before and after. No input typed. Recording ran 42.8 s; no bytes after the last chunk. <name>.timing.json holds each PTY read as [ms since spawn, UTF-16 length] and recordedUntilMs.",
"exitCode": null
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"recordedUntilMs":42761.8,"chunks":[[785.0,205],[786.9,71],[791.9,8],[793.5,57],[793.9,32],[794.0,16],[794.0,16],[794.1,24],[794.2,24],[794.2,24],[794.3,36],[794.4,24],[794.5,24],[795.2,1024],[795.8,1024],[796.3,1024],[796.7,1024],[796.9,1024],[797.2,961],[811.9,1024],[812.5,1024],[813.1,1024],[813.6,1024],[814.1,1024],[814.7,937],[2523.9,1024],[2524.9,1024],[2525.2,1024],[2525.5,1024],[2525.8,1024],[2526.1,1024],[2526.3,353],[2541.6,13],[2609.5,934],[2610.0,938],[2610.6,871],[2611.3,584]]}

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"capturedAt": "2026-09-30T08:39:12.532885Z",
"platform": "darwin",
"command": ["opencode"],
"cols": 120,
"rows": 40,
"note": "1.18.32 on macOS 26.6.2 arm64, 120x40, alternate screen. Why this run: the slowest recorded boot: 3.9 s of silence after bracketed paste is enabled, and the longest gap (161 ms) between the bare OpenCode title and the input box. Recorded with a Python PTY driver rather than the repo recorder, because the driver stamps every read with its time and answers the terminal queries OpenCode sends; responder 'xterm' = xterm.js 6.0 replies incl. OSC 10/11/12/4 colours (a visible Orca pane). Isolated HOME and XDG dirs, OPENCODE_DISABLE_AUTOUPDATE=1, OPENCODE_CONFIG_CONTENT {\"autoupdate\":false}, no model configured, env -i (no ORCA_*); version and binary sha256 unchanged before and after. No input typed. Recording ran 76.9 s; no bytes after the last chunk. <name>.timing.json holds each PTY read as [ms since spawn, UTF-16 length] and recordedUntilMs.",
"exitCode": null
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"recordedUntilMs":76886.4,"chunks":[[2454.9,276],[2478.3,8],[2513.8,1024],[2518.9,1024],[2531.6,1024],[2539.1,1024],[2581.2,1024],[2583.9,1024],[2587.4,411],[2593.0,1024],[2602.3,1024],[2607.8,1024],[2621.8,1024],[2622.5,1024],[2623.3,937],[6479.2,1024],[6485.0,1024],[6487.5,1024],[6490.0,1024],[6499.8,1024],[6502.7,1024],[6509.0,353],[6538.4,13],[6685.0,934],[6686.4,938],[6697.9,871],[6699.1,584]]}

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"capturedAt": "2026-09-30T08:33:44.169427Z",
"platform": "darwin",
"command": ["opencode"],
"cols": 120,
"rows": 40,
"note": "1.18.32 on macOS 26.6.2 arm64, 120x40, alternate screen. Why this run: first launch on an empty home: no first-run dialog appears before the input box. Recorded with a Python PTY driver rather than the repo recorder, because the driver stamps every read with its time and answers the terminal queries OpenCode sends; responder 'xterm' = xterm.js 6.0 replies incl. OSC 10/11/12/4 colours (a visible Orca pane). Isolated HOME and XDG dirs, OPENCODE_DISABLE_AUTOUPDATE=1, OPENCODE_CONFIG_CONTENT {\"autoupdate\":false}, no model configured, env -i (no ORCA_*); version and binary sha256 unchanged before and after. No input typed. Recording ran 14.9 s; no bytes after the last chunk. <name>.timing.json holds each PTY read as [ms since spawn, UTF-16 length] and recordedUntilMs.",
"exitCode": null
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"recordedUntilMs":14932.3,"chunks":[[894.6,205],[897.3,71],[906.1,8],[909.0,57],[909.3,32],[909.4,16],[909.5,16],[909.6,24],[909.6,24],[909.8,24],[909.9,36],[910.0,24],[910.1,24],[911.8,134],[915.0,1024],[916.4,1024],[917.0,1024],[917.5,1024],[918.0,1024],[918.6,1024],[933.4,1024],[934.1,1024],[934.7,1024],[935.3,1024],[935.8,1024],[936.4,937],[2731.2,1024],[2731.8,1024],[2732.2,1024],[2732.7,1024],[2733.1,1024],[2734.0,1024],[2734.4,353],[2766.9,13],[2847.0,934],[2847.6,938],[2848.1,871],[2848.6,584]]}

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"capturedAt": "2026-09-30T08:50:47.405221Z",
"platform": "darwin",
"command": ["opencode", "--standalone"],
"cols": 120,
"rows": 40,
"note": "opencode v2.0.18 on macOS 26.6.2 arm64, 120x40, alternate screen. Why this run: OpenCode 2 answered like a hidden Orca pane; show-cursor repeats on every frame after the box. Recorded with a Python PTY driver rather than the repo recorder, because the driver stamps every read with its time and answers the terminal queries OpenCode sends; responder 'headless' = Orca's hidden-pane replies, silent on OSC colour queries. Isolated HOME and XDG dirs, OPENCODE_DISABLE_AUTOUPDATE=1, OPENCODE_CONFIG_CONTENT {\"autoupdate\":false}, no model configured, env -i (no ORCA_*); version and binary sha256 unchanged before and after. No input typed. Recording ran 41.5 s; no bytes after the last chunk. Run as `opencode --standalone` from a 2.0.18 binary; Orca's opencode2 row launches `opencode2 --standalone`. <name>.timing.json holds each PTY read as [ms since spawn, UTF-16 length] and recordedUntilMs.",
"exitCode": null
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"recordedUntilMs":41502.8,"chunks":[[307.5,280],[316.3,1024],[316.7,1024],[317.0,1024],[317.4,1024],[317.7,1024],[318.0,1022],[318.3,164],[318.3,24],[318.3,24],[326.8,1024],[327.5,1024],[328.0,1024],[328.6,1024],[329.1,1024],[329.7,937],[1339.7,1024],[1340.4,1024],[1341.1,1024],[1341.8,1024],[1342.5,1024],[1343.1,1024],[1343.8,353],[1348.0,13],[1387.9,934],[1388.4,938],[1388.7,832],[1418.2,483],[1436.2,483],[1453.8,488],[1471.8,497],[1489.7,498],[1508.0,504],[1525.8,504],[1543.2,504],[1560.9,504]]}

Large diffs are not rendered by default.

132 changes: 132 additions & 0 deletions src/main/runtime/agent-prompt-launch-submit-retry.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { resolveAgentPromptSubmitDelayForAgent } from '../../shared/agent-prompt-injection'
import type { TuiAgent } from '../../shared/tui-agent'
import { TUI_AGENT_CONFIG } from '../../shared/tui-agent-config'
import { createAgentPromptSubmissionRuntime } from './agent-prompt-submission-runtime-test-fixture'
import type { OrcaRuntimeService } from './orca-runtime'
import { dispatchPreambleSendOptions } from './orchestration/preamble'

vi.mock('../git/worktree', () => ({
listWorktrees: vi.fn().mockResolvedValue([
{
path: '/tmp/worktree-a',
head: 'abc',
branch: 'feature/launch-submit-retry',
isBare: false,
isMainWorktree: false
}
]),
listWorktreesStrict: vi.fn().mockResolvedValue([
{
path: '/tmp/worktree-a',
head: 'abc',
branch: 'feature/launch-submit-retry',
isBare: false,
isMainWorktree: false
}
])
}))

const PROMPT = 'Task: reply with OK'

async function sendDispatch(
agent: TuiAgent,
retrySubmitAfterLaunch: boolean,
afterFirstEnter?: (runtime: OrcaRuntimeService) => void
) {
let entered = false
const { runtime, handle, writes } = await createAgentPromptSubmissionRuntime((rt, data) => {
if (data === '\r' && !entered) {
entered = true
afterFirstEnter?.(rt)
}
}, agent)
const sent = runtime.sendTerminalAgentPrompt(handle, PROMPT, {
...dispatchPreambleSendOptions('request-1'),
retrySubmitAfterLaunch
})
return { runtime, sent, enters: () => writes.filter((data) => data === '\r').length }
}

// Why: OpenCode 2 drops an Enter sent before its mode row is drawn, so a worker's first dispatch
// gets one more Enter; every other send keeps exactly one.
describe('retry Enter on a launched worker first dispatch', () => {
afterEach(() => vi.useRealTimers())

it.each(['opencode', 'opencode2'] as const)(
'%s gets a second Enter after its row delay',
async (agent) => {
vi.useFakeTimers()
const submitDelayMs = resolveAgentPromptSubmitDelayForAgent(process.platform, PROMPT, agent)
const retryDelayMs = TUI_AGENT_CONFIG[agent].submitRetryDelayMs ?? 0
const { sent, enters } = await sendDispatch(agent, true)

await vi.advanceTimersByTimeAsync(submitDelayMs)
expect(enters()).toBe(1)
await vi.advanceTimersByTimeAsync(retryDelayMs - 1)
expect(enters()).toBe(1)
await vi.advanceTimersByTimeAsync(1)

await expect(sent).resolves.toMatchObject({
accepted: true,
bytesWritten: expect.any(Number),
prompt: { stages: ['input_accepted'] }
})
expect(enters()).toBe(2)
}
)

it('keeps one Enter for dispatches that did not launch the agent', async () => {
vi.useFakeTimers()
const { sent, enters } = await sendDispatch('opencode', false)

await vi.runAllTimersAsync()
await sent
expect(enters()).toBe(1)
})

it.each(['codex', 'claude', 'aider'] as const)(
'%s keeps one Enter on a launched worker first dispatch',
async (agent) => {
vi.useFakeTimers()
const { sent, enters } = await sendDispatch(agent, true)

await vi.runAllTimersAsync()
await sent
expect(enters()).toBe(1)
}
)

it('never answers a permission prompt drawn after the first Enter', async () => {
vi.useFakeTimers()
const { sent, enters } = await sendDispatch('opencode', true, (runtime) => {
runtime.onPtyData(
'pty-prompt',
'\x1b]9999;{"state":"waiting","agentType":"opencode"}\x07',
Date.now()
)
})

await vi.runAllTimersAsync()
await expect(sent).resolves.toMatchObject({ accepted: true })
expect(enters()).toBe(1)
})

it('leaves the first Enter standing when the pane exits before the retry', async () => {
vi.useFakeTimers()
const submitDelayMs = resolveAgentPromptSubmitDelayForAgent(
process.platform,
PROMPT,
'opencode'
)
const { runtime, sent, enters } = await sendDispatch('opencode', true)

await vi.advanceTimersByTimeAsync(submitDelayMs)
expect(enters()).toBe(1)
await runtime.onPtyExit('pty-prompt', 0)
await vi.runAllTimersAsync()

await expect(sent).resolves.toMatchObject({ accepted: true })
expect(enters()).toBe(1)
})
})
164 changes: 164 additions & 0 deletions src/main/runtime/agent-prompt-launch-turn-observation.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { TUI_AGENT_CONFIG } from '../../shared/tui-agent-config'
import { AGENT_PROMPT_TEST_WORKTREE_PATH } from './agent-prompt-submission-runtime-test-fixture'
import { OrcaRuntimeService } from './orca-runtime'
import { dispatchPreambleSendOptions } from './orchestration/preamble'
import { makeStore } from './runtime-rpc-worktree-store-fixtures'

vi.mock('../git/worktree', () => ({
listWorktrees: vi.fn().mockResolvedValue([
{
path: '/tmp/worktree-a',
head: 'abc',
branch: 'feature/launch-turn-observation',
isBare: false,
isMainWorktree: false
}
]),
listWorktreesStrict: vi.fn().mockResolvedValue([
{
path: '/tmp/worktree-a',
head: 'abc',
branch: 'feature/launch-turn-observation',
isBare: false,
isMainWorktree: false
}
])
}))

const RETRY_DELAY_MS = TUI_AGENT_CONFIG.opencode.submitRetryDelayMs ?? 0

type HookRow = { state: 'done' | 'working' | 'waiting'; stateStartedAt: number; prompt: string }

/**
* An OpenCode pane whose status plugin reports through the hook store. It has posted nothing
* before the first submit; `onEnter` runs on each Enter so a test can post what OpenCode would.
*/
async function createOpenCodePane(onEnter: (enter: number, post: (row: HookRow) => void) => void) {
let handle = ''
let hook: HookRow | null = null
const writes: string[] = []
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the runtime reads only repos, worktree meta and settings from this store fixture.
const runtime = new OrcaRuntimeService(makeStore() as never, undefined, {
getAgentStatusSnapshot: () =>
hook
? [
{
paneKey: 'prompt-pane',
terminalHandle: handle,
state: hook.state,
prompt: hook.prompt,
agentType: 'opencode',
connectionId: null,
receivedAt: Date.now(),
stateStartedAt: hook.stateStartedAt
}
]
: []
})
const post = (row: HookRow): void => {
hook = row
}
runtime.setPtyController({
spawn: vi.fn().mockResolvedValue({ id: 'pty-prompt' }),
write: (_ptyId, data) => {
writes.push(data)
if (data === '\r') {
onEnter(
writes.filter((write) => write === '\r').length,
(row) => void setTimeout(() => post(row), 50)
)
}
return true
},
kill: () => true,
getForegroundProcess: async () => null
})
handle = (
await runtime.createTerminal(`path:${AGENT_PROMPT_TEST_WORKTREE_PATH}`, {
launchAgent: 'opencode'
})
).handle
const sent = runtime.sendTerminalAgentPrompt(handle, 'Task: reply with OK', {
...dispatchPreambleSendOptions('request-1'),
retrySubmitAfterLaunch: true
})
return { runtime, sent, enters: () => writes.filter((data) => data === '\r').length }
}

const WORKING = (): HookRow => ({
state: 'working',
stateStartedAt: Date.now(),
prompt: 'Task: reply with OK'
})
const UNSUPPORTED_RECEIPT = { prompt: { provider: 'unsupported', observation: 'unsupported' } }

// Why: OpenCode's plugin posts `working` right after every real submit and nothing for a brief
// stuck in its box, so a turn this request started makes the retry Enter unnecessary. The receipt
// stays unsupported: with no post before the first submit, a plugin that never loaded looks the
// same as a stuck brief.
describe('retry Enter skip on a launched OpenCode worker first dispatch', () => {
afterEach(() => vi.useRealTimers())

it('skips the retry Enter when the first Enter started a turn', async () => {
vi.useFakeTimers()
const { sent, enters } = await createOpenCodePane((enter, post) => {
if (enter === 1) {
post(WORKING())
}
})

await vi.runAllTimersAsync()
await expect(sent).resolves.toMatchObject(UNSUPPORTED_RECEIPT)
expect(enters()).toBe(1)
})

it('retries once when no turn starts, with the same receipt', async () => {
vi.useFakeTimers()
const { sent, enters } = await createOpenCodePane(() => {})

await vi.advanceTimersByTimeAsync(RETRY_DELAY_MS + 1_000)
await expect(sent).resolves.toMatchObject(UNSUPPORTED_RECEIPT)
expect(enters()).toBe(2)
})

it("does not count the plugin's first `done` post as a turn start", async () => {
vi.useFakeTimers()
// The plugin's first post, `done` with an empty prompt, arrives just after the submit.
const { sent, enters } = await createOpenCodePane((enter, post) => {
if (enter === 1) {
post({ state: 'done', stateStartedAt: Date.now(), prompt: '' })
}
})

await vi.runAllTimersAsync()
await expect(sent).resolves.toMatchObject(UNSUPPORTED_RECEIPT)
expect(enters()).toBe(2)
})

it('skips the retry Enter when OpenCode asks for permission after the first Enter', async () => {
vi.useFakeTimers()
const { sent, enters } = await createOpenCodePane((enter, post) => {
if (enter === 1) {
post({ state: 'waiting', stateStartedAt: Date.now(), prompt: '' })
}
})

await vi.runAllTimersAsync()
await expect(sent).resolves.toMatchObject(UNSUPPORTED_RECEIPT)
expect(enters()).toBe(1)
})

it('leaves the first Enter standing when the pane exits during the wait', async () => {
vi.useFakeTimers()
const { runtime, sent, enters } = await createOpenCodePane((enter) => {
if (enter === 1) {
setTimeout(() => void runtime.onPtyExit('pty-prompt', 0), 500)
}
})

await vi.runAllTimersAsync()
await expect(sent).resolves.toMatchObject({ accepted: true, ...UNSUPPORTED_RECEIPT })
expect(enters()).toBe(1)
})
})
Loading
Loading