We currently provided a simulated plan mode profile. Our new architecture should be based entirely on dispatching agents through CLI commands, like claude - "run the eval case". These kind of dispatches should be able to be made in plan mode when given the right flag - this seems to be a pretty standard feature, and I think it exists in all our supported harnesses.
This should mean that we don't need any kind of plan mode simulation, and can simply rely on it being supported by the harness CLI. The main open question here is how this plays with multi-turn eval runs. Ideally, we'd be able to engineer and eval that starts in plan mode, then moves naturally to "act" mode, as a normal session starting in plan mode would go. This should get us as close as possible to simulating a real agent session.
We currently provided a simulated plan mode profile. Our new architecture should be based entirely on dispatching agents through CLI commands, like
claude - "run the eval case". These kind of dispatches should be able to be made in plan mode when given the right flag - this seems to be a pretty standard feature, and I think it exists in all our supported harnesses.This should mean that we don't need any kind of plan mode simulation, and can simply rely on it being supported by the harness CLI. The main open question here is how this plays with multi-turn eval runs. Ideally, we'd be able to engineer and eval that starts in plan mode, then moves naturally to "act" mode, as a normal session starting in plan mode would go. This should get us as close as possible to simulating a real agent session.