Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/premerge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -655,7 +655,7 @@ jobs:
continue-on-error: true
timeout-minutes: 45
shell: pwsh
run: ./scripts/ci/browser-windows-native.ps1
run: ./scripts/ci/browser-windows-native.ps1 -HostProfile Release
- name: Retain bounded owned Windows browser evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
Expand Down
30 changes: 21 additions & 9 deletions bundled-plugins/colossus/skills/computer-use/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,25 @@ not confine actions performed through another app's interface.

1. Call `computer.windows` and select an existing window from the returned list.
Use its opaque `window_id` with `computer.open`. Never invent window handles,
process IDs, element IDs, session IDs, or control generations.
2. Call `computer.snapshot` for accessible controls or `computer.screenshot` for
visual content. Screenshots return an image to image-capable models and an
inline preview to the user; a separate image upload is unnecessary.
3. Choose one action using the returned session, generation, and observation ID.
process IDs, element IDs, session IDs, or control generations. Opening control
does not activate the window.
2. Call `computer.snapshot` for accessible controls and current foreground metadata.
3. Find the entry in `observation.windows` whose `window_id` matches
`observation.window_id`. If its `foreground` is `false`, use `computer.focus`
with that observed window ID and the fresh observation. Then take a new
snapshot before any other input. Snapshots and screenshots can read background
windows; they do not activate them, and clicks do not focus them automatically.
4. Choose one action using the returned session, generation, and observation ID.
Coordinates are physical pixels local to the captured window, including its
frame, not coordinates on the full desktop. Prefer an observed element for a
click when accessible semantics identify it accurately.
4. Observe again after every action. Each input consumes its observation; moving
or resizing the window, switching apps, or changing control invalidates old
observations. A successful input receipt does not prove the intended result.
5. Observe again after every action. Each input, including focus, consumes its
observation; moving or resizing the window, switching apps, or changing control
invalidates old observations. A successful input receipt does not prove the
intended result.

Use `computer.screenshot` for visual content. It returns an image to image-capable
models and an inline preview to the user; a separate image upload is unnecessary.

Use only the observation from the latest `computer.snapshot` or
`computer.screenshot`; another snapshot or screenshot replaces the previous one.
Expand Down Expand Up @@ -59,7 +67,11 @@ control. Do not repeatedly poll or try other input routes. A new run starts with
no computer session: enumerate and open again rather than reusing a previous
run's session, window, or observation IDs.

On a stale observation, inspect the current session before choosing another action.
On a stale observation, inspect `computer.status` and take a fresh snapshot before
choosing another action. Check the selected window's `foreground` flag and follow
the explicit focus sequence above if it is in the background. A fresh snapshot
alone does not fix missing focus; stale can also mean expired evidence, changed
geometry, or changed control, so do not assume that focus is always the cause.
On an unknown external outcome, inspect the resulting app state and report the
uncertainty. Never automatically repeat a click, submission, or text entry whose
outcome is unknown. If the desktop is locked, Windows requests elevation, the app
Expand Down
29 changes: 25 additions & 4 deletions crates/colossus-api-runtime/src/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2306,7 +2306,7 @@ fn released_computer_error(output: &str) -> Option<String> {
(
ComputerDriverError::Stale,
"target_or_observation_changed",
"The computer target or observation changed. Check computer.status and take a fresh snapshot before choosing another action.",
"The computer target or observation changed. Check computer.status and take a fresh snapshot. If the selected window is in the background, use computer.focus with its observed window_id, then take a new snapshot before input.",
),
(
ComputerDriverError::InvalidInput,
Expand All @@ -2328,6 +2328,9 @@ fn released_computer_error(output: &str) -> Option<String> {
colossus_policy::GatewayError::Execution(native.to_string()).to_string();
let released_message = error.get("message").and_then(serde_json::Value::as_str);
if released_message == Some(native_message.as_str())
// Existing history retains this exact pre-guidance native message.
|| (native == ComputerDriverError::Stale
&& released_message == Some("effect failed: computer target or observation is stale"))
|| (error.get("reason").and_then(serde_json::Value::as_str) == Some(reason)
&& released_message == Some(guidance))
{
Expand Down Expand Up @@ -3726,10 +3729,28 @@ mod tests {
);
assert_eq!(released_computer_error(&preview), Some(preview));
}
let stale = ComputerDriverError::Stale;
let current_message = format!("effect failed: {stale}");
let output = |message: &str| {
serde_json::json!({"error": {
"type": "execution_error", "message": message, "recoverable": true,
}})
.to_string()
};
assert_eq!(
released_computer_error(&output(
"effect failed: computer target or observation is stale"
)),
released_computer_error(&output(&current_message)),
"exact historical native Stale errors retain the current safe recovery guidance"
);
for message in [
"computer target or observation is stale",
"effect failed: computer target or observation is stale PRIVATE_SUFFIX",
"PRIVATE_PREFIX effect failed: computer target or observation is stale",
stale.to_string(),
format!("{current_message} PRIVATE_SUFFIX"),
format!("PRIVATE_PREFIX {current_message}"),
"computer target or observation is stale".into(),
"effect failed: computer target or observation is stale PRIVATE_SUFFIX".into(),
"PRIVATE_PREFIX effect failed: computer target or observation is stale".into(),
] {
let output = serde_json::json!({"error": {
"type": "execution_error", "message": message, "recoverable": true,
Expand Down
2 changes: 1 addition & 1 deletion crates/colossus-cli/tests/ci_contract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1831,7 +1831,7 @@ fn owned_windows_chromium_is_real_required_native_acceptance_with_bounded_eviden
assert_eq!(field(native, "continue-on-error").as_bool(), Some(true));
assert_eq!(
field(native, "run").as_str(),
Some("./scripts/ci/browser-windows-native.ps1")
Some("./scripts/ci/browser-windows-native.ps1 -HostProfile Release")
);
let aggregate = named_step(desktop, "Require every Windows Desktop acceptance check");
assert_eq!(
Expand Down
4 changes: 3 additions & 1 deletion crates/colossus-ports/src/computer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,9 @@ pub enum ComputerDriverError {
#[error("computer control stopped by user")]
UserStopped,
/// Native identity, control generation, geometry, focus, or observation changed.
#[error("computer target or observation is stale")]
#[error(
"computer target or observation is stale; check computer.status and take a fresh snapshot; if the selected observation.window_id has foreground=false in observation.windows, use computer.focus for that observed window, then take a new snapshot before input"
)]
Stale,
/// Closed typed payload failed bounded validation.
#[error("invalid computer input")]
Expand Down
136 changes: 110 additions & 26 deletions crates/colossus-runtime/src/browser_package/windows_preview.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ use colossus_browser_bridge::BrowserHostFactory as _;
use colossus_contracts::{BrowserActionKind, BrowserCapabilities, BrowserLimits, BrowserMode};
use colossus_ports::BrowserDriverError;
use colossus_sandbox::{BrowserEgressLimits, WindowsBrowserConfig, WindowsBrowserSupervisor};
use colossus_windows_native::{BoundPath, PrivateDirectoryCreation};
use colossus_windows_native::{
BoundPath, PrivateDirectoryCreation, current_user_profile_directory,
};
use std::{
collections::VecDeque,
os::windows::ffi::OsStrExt as _,
path::Path,
sync::{
Arc, Mutex, OnceLock,
Expand Down Expand Up @@ -49,25 +52,37 @@ struct Cleanup {
supervisor: Option<Arc<WindowsBrowserSupervisor>>,
state: Mutex<Option<PrivateDirectoryCreation>>,
_image: BoundPath,
_instance: BoundPath,
slot: Mutex<Option<Slot<'static>>>,
}

// Install hashes the full inventory on a blocking worker. A cancelled awaiting
// future must transfer the exact allocation before its best-effort Drop runs.
struct Setup(Option<(PrivateDirectoryCreation, BoundPath, Slot<'static>)>);
struct SetupAllocation {
state: PrivateDirectoryCreation,
image: BoundPath,
instance: BoundPath,
slot: Slot<'static>,
}
struct Setup(Option<SetupAllocation>);
impl Drop for Setup {
fn drop(&mut self) {
if let Some((state, image, slot)) = self.0.take() {
retain(Arc::new(Cleanup {
supervisor: None,
state: Mutex::new(Some(state)),
_image: image,
slot: Mutex::new(Some(slot)),
}));
if let Some(allocation) = self.0.take() {
retain(Arc::new(Cleanup::new(allocation, None)));
}
}
}
impl Cleanup {
fn new(allocation: SetupAllocation, supervisor: Option<Arc<WindowsBrowserSupervisor>>) -> Self {
Self {
supervisor,
state: Mutex::new(Some(allocation.state)),
_image: allocation.image,
_instance: allocation.instance,
slot: Mutex::new(Some(allocation.slot)),
}
}

async fn shutdown(&self) -> Result<(), BrowserDriverError> {
if let Some(supervisor) = &self.supervisor {
supervisor.shutdown().await?;
Expand Down Expand Up @@ -112,20 +127,42 @@ pub(super) async fn install(
.map_err(|_| BrowserPackageError::Invalid)?
.try_into()
.map_err(|_| BrowserPackageError::Invalid)?;
// Retain the authenticated managed instance even though Chromium's cache
// cannot inherit its potentially long workspace ancestry on Windows.
let instance = bind_instance(instance_root)?;
let Some(state_name) = short_state_name() else {
// No state or native process has been allocated. Never fall back to the
// unsupported long managed path or an ambient temporary directory.
return Ok(None);
};
// Native FILE_CREATE returns custody before verification; existing names and
// reparse parents cannot become this allocation. Never adopt a prior profile.
let slot = Slot::reserve(&INSTALLATION_ACTIVE).ok_or(BrowserPackageError::CleanupUnknown)?;
let state = match PrivateDirectoryCreation::create(
&instance_root.join(format!("browser-preview-{}", uuid::Uuid::new_v4().simple())),
) {
let state = match PrivateDirectoryCreation::create_in_current_user_profile(&state_name) {
Ok(state) => state,
Err(_) => {
slot.retire();
return Err(BrowserPackageError::Invalid);
}
};
let state_root = state.path().to_owned();
let mut setup = Setup(Some((state, image, slot)));
let mut setup = Setup(Some(SetupAllocation {
state,
image,
instance,
slot,
}));
// Recheck the actual retained allocation too: token profile resolution can
// change between the preallocation budget and the atomic native creator.
if !short_native_profile(&state_root) {
let allocation = setup.0.take().ok_or(BrowserPackageError::Invalid)?;
let cleanup = Arc::new(Cleanup::new(allocation, None));
if cleanup.shutdown().await.is_err() {
retain(cleanup);
return Err(BrowserPackageError::CleanupUnknown);
}
return Ok(None);
}
let limits = BrowserLimits::default();
let installation =
WindowsBrowserSupervisor::install_for_functional_preview(WindowsBrowserConfig {
Expand All @@ -143,18 +180,13 @@ pub(super) async fn install(
},
})
.await;
let (state, image, slot) = setup.0.take().ok_or(BrowserPackageError::Invalid)?;
let allocation = setup.0.take().ok_or(BrowserPackageError::Invalid)?;
let supervisor = Arc::new(match installation {
Ok(supervisor) => supervisor,
// Only the native preview constructor classifies ordinary-user desktop
// prerequisites this way; invalid publisher bytes remain fatal.
Err(error) => {
let cleanup = Arc::new(Cleanup {
supervisor: None,
state: Mutex::new(Some(state)),
_image: image,
slot: Mutex::new(Some(slot)),
});
let cleanup = Arc::new(Cleanup::new(allocation, None));
if cleanup.shutdown().await.is_err() {
retain(cleanup);
return Err(BrowserPackageError::CleanupUnknown);
Expand All @@ -166,12 +198,7 @@ pub(super) async fn install(
};
}
});
let cleanup = Arc::new(Cleanup {
supervisor: Some(Arc::clone(&supervisor)),
state: Mutex::new(Some(state)),
_image: image,
slot: Mutex::new(Some(slot)),
});
let cleanup = Arc::new(Cleanup::new(allocation, Some(Arc::clone(&supervisor))));
if supervisor.capabilities().restrictive_egress {
retain(cleanup);
return Err(BrowserPackageError::Invalid);
Expand All @@ -190,6 +217,39 @@ pub(super) async fn install(
})))
}

fn short_native_profile(state: &Path) -> bool {
// The supervisor appends a fresh 32-hex session and the profile leaf. Keep
// twenty UTF-16 units below MAX_PATH before Chromium adds its own files.
let profile = state
.join("session-00000000000000000000000000000000")
.join("profile");
let native = profile.as_os_str().encode_wide().collect::<Vec<_>>();
native
.strip_prefix(&[92, 92, 63, 92])
.unwrap_or(&native)
.len()
< 240
}

fn bind_instance(root: &Path) -> Result<BoundPath, BrowserPackageError> {
let instance = BoundPath::open_directory(root).map_err(|_| BrowserPackageError::Invalid)?;
instance
.validate_private_owner_dacl()
.and_then(|()| instance.validate_ancestor_namespace_authority())
.and_then(|()| instance.revalidate())
.map_err(|_| BrowserPackageError::Invalid)?;
Ok(instance)
}

fn short_state_name() -> Option<std::ffi::OsString> {
let parent = current_user_profile_directory().ok()?;
let name = std::ffi::OsString::from(format!(
"ColossusBrowserPreview-{}",
uuid::Uuid::new_v4().simple()
));
short_native_profile(&parent.join(&name)).then_some(name)
}

fn capabilities(limits: BrowserLimits) -> BrowserCapabilities {
use BrowserActionKind::*;
BrowserCapabilities {
Expand Down Expand Up @@ -309,3 +369,27 @@ fn retry(keeper: &'static Keeper) {
}
}
}

#[cfg(test)]
mod path_tests {
use super::short_native_profile;
use std::path::PathBuf;

#[test]
fn native_profile_budget_counts_utf16_and_excludes_extended_path_prefix() {
let short = PathBuf::from(format!(r"C:\{}", "x".repeat(187)));
assert!(short_native_profile(&short));
assert!(short_native_profile(&PathBuf::from(format!(
r"\\?\{}",
short.display()
))));
assert!(!short_native_profile(&PathBuf::from(format!(
r"C:\{}",
"x".repeat(188)
))));
assert!(!short_native_profile(&PathBuf::from(format!(
r"C:\{}",
"😀".repeat(94)
))));
}
}
Loading
Loading