diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index de5b202f..65f9e297 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -154,7 +154,7 @@ Ported in part (the test name carries `Partial_` or a `PARTIAL` suffix): Not ported, because the feature is not in the port: -- No `ListReporter`, run header, or live window; no JUnit or Markdown report renderer; no serial groups, workers, shards, runner, `e2e run`, `e2e init`, or `e2e list`; no JavaScript config loader; no `test.extend`; no managed process; no telemetry; no wordmark; no `projectId`; no `runWithRetries`: the `report-list`, `report-junit`, `report-markdown`, `cli-run`, `cli-boot`, `config-load`, `parallel-workers`, `managed-process`, `extended-fixtures`, `telemetry-*`, `wordmark`, `target-identity`, and `retry` tests, and the three `report-status` tests about the last attempt of a serial group +- No `ListReporter`, run header, or live window; no JUnit or Markdown report renderer; no serial groups, workers, shards, runner, `e2e run`, `e2e init`, or `e2e list`; no JavaScript config loader; no `test.extend`; no telemetry; no wordmark; no `projectId`; no `runWithRetries`: the `report-list`, `report-junit`, `report-markdown`, `cli-run`, `cli-boot`, `config-load`, `parallel-workers`, `extended-fixtures`, `telemetry-*`, `wordmark`, `target-identity`, and `retry` tests, and the three `report-status` tests about the last attempt of a serial group - No value expectations (`expect-values` `truthy/falsy/null/undefined fail on the wrong value, and pass negated`), no `toHaveClass`, `frameLocator`, `*css=` captures, or click-modifier tests; the `waitForResponse`, `route`, and `unroute` predicate tests and the `an option ... does not take is refused` tests cannot happen, because C# overloads are typed - No run-wide usage accumulator (the port has `ModelUsage`, but no overflow-capping total or usage events), `withStallGuard`, `SessionStore`, `realmSlot`, or `verifyEndState` with `waitMs`: the `model-usage`, `tool-loop-stall`, `session-envelope`, `realm-slot`, and `trace-replay` end-wait tests - No `pixelFallback` observe option, `tap_at` or `select_at`, upload tool, or `schema` and `vision` options on `act`: the `agent-fallback-handoff`, `agent-act-pixel-tools`, `agent-act-verbs` hidden-file, `act-validation`, and `agent-screen-update` tests @@ -227,7 +227,7 @@ Upstream's `4982d8d` (#1333) documents that the browser engine's semantic tree l `skills/create-verification-skill/` ports upstream's skill of the same name (commit `d85702a2`, `feat(skills): create-verification-skill generates a verify- skill on e2e (#981)`). Like upstream, it is not bundled in `E2E.Cli` and `e2e guide` does not read it; install it with `npx skills add hardkoded/e2e-dotnet --skill create-verification-skill`. Differences: -- The steps use what the port has. The launch is the app started by hand (the port has no `app.command`), the doctor is a smoke test run with `dotnet test --filter`, and the evidence is the `dotnet test` result and the test's own checks, since the port writes no report or trace page, and a video only when `video` is on. +- The steps use what the port has. The launch is the target's `app.command` or the app started by hand, the doctor is a smoke test run with `dotnet test --filter`, and the evidence is the `dotnet test` result and the test's own checks, since the port writes no report or trace page, and a video only when `video` is on. - Not ported: the generated bug bash fan-out. It depends on `e2e.bugbash.config.ts`, which the port lacks, and runs many `e2e explore` commands at once, so a feature's charters are goals for a live MCP walk, and a finding is a bug once a repro test fails with `ASSERTION_FAILED`. - Not ported: `npx e2e init` as the setup step, `e2e list` as a config check, `--output` for proofs, and the `.gitignore` lines for `.e2e/bugbash/` and `.e2e/proof/`. The repo already ignores `.e2e/`. - The example under `references/example/` is `verify-todomvc` for `samples/TodoMvc`, in place of `verify-playground` for upstream's `apps/testbed`, which the port does not have. It has three features (adding, completing, filtering), not the playground's three (todos, forms, login). @@ -249,7 +249,7 @@ Not ported: `e2e init`, which scaffolds the config, installs the skill into `.ag `E2E.Mobile.NUnit.MobileAppTest` and `E2E.Mobile.XUnit.V3.MobileAppTest` derive from `E2ETest` and make a device test one `using` and one config file. Upstream declares a device in `e2e.config.ts` and its runner owns the engine and the hosted device. The port's adapters do, so these classes do it. Every symbol here is .NET-only. Differences: -- `e2e.config.json` accepts `"platform": "ios"` and `"android"`. A device target takes `app.bundleId` and `app.appPath`, as upstream's `app`. It does not take the other upstream app keys (`identity`, `environment`, `launchArguments`, `permissions`). The port still runs one target. +- `e2e.config.json` accepts `"platform": "ios"` and `"android"`. A device target takes `app.bundleId` and `app.appPath`, as upstream's `app`. It takes `app.command` and `app.readyUrl` too (see [Config](#config)). It does not take the other upstream app keys (`identity`, `environment`, `launchArguments`, `permissions`). The port still runs one target. - `E2E_MOBILE_PLATFORM` and `E2E_MOBILE_APP_PATH` win over the config, so one project can run on iOS and Android. `E2E_MOBILE_DEVICE`, `E2E_MOBILE_PROVIDER` (`local`, `limrun`, `devflow`), `E2E_DEVFLOW_PORT`, and `E2E_DEVFLOW_APP` are .NET-only. - A hosted run leases one device on the first fixture and releases it when the process ends, because the adapters have no hook after the last test. A killed process is covered by the provider's hard timeout (15 minutes for Limrun). - `MobileAppHost` is one source file, compiled into both packages. `HostedDeviceProviders` (the provider each `E2E_MOBILE_PROVIDER` name makes) is a second, compiled into both packages and into `E2E.Cli`. `CreateMobileOptions`, `AppId`, `AppPath`, `Platform`, and `AttachesToRunningApp` are the members a test project overrides or reads. @@ -487,9 +487,18 @@ Not ported: ## Config -`E2ETest` loads the nearest `e2e.config.json` above the test assembly directory, then above the working directory, once per run. Supported keys: `targets` (one entry with `name`, `platform`, and `app.url`), `timeout`, `launchTimeout`, `actionTimeout`, `assertionTimeout`, `cleanupTimeout`, `retries`, `agents`, `cache` (`mode`, `dir`, `strict`), and `secrets`. Defaults, the CI cache demotion to `read-only`, `E2E_SECRET_` overrides, the 6 code point secret minimum, and `INVALID_CONFIG` for unknown keys follow upstream. +`E2ETest` loads the nearest `e2e.config.json` above the test assembly directory, then above the working directory, once per run. Supported keys: `targets` (one entry with `name`, `platform`, and `app` with `url`, `command`, and `readyUrl`), `timeout`, `launchTimeout`, `actionTimeout`, `assertionTimeout`, `cleanupTimeout`, `retries`, `agents`, `cache` (`mode`, `dir`, `strict`), and `secrets`. Defaults, the CI cache demotion to `read-only`, `E2E_SECRET_` overrides, the 6 code point secret minimum, and `INVALID_CONFIG` for unknown keys follow upstream. -`app.url` is normalized as upstream does, with `System.Uri` instead of WHATWG: a missing scheme becomes `https://`, or `http://` for a loopback host. Userinfo, a query, a fragment, a scheme other than http(s), and plain HTTP on a non-loopback host are `INVALID_APP_URL`, with upstream's messages. A base URL set in code (`E2EFixture.BaseUrl`, `E2ESessionOptions.BaseUrl`) follows the same rule. Port `0` on `127.0.0.1` or `[::1]` is accepted by the parser, but the port has no `app.command` to start the app on a free port, so it is `INVALID_CONFIG`; any other host with port `0` is `INVALID_APP_URL`. Keys of the old .NET shape (`app`, `engine`, `agent`, `timeouts`) fail with a hint to the upstream key. +`app.url` is normalized as upstream does, with `System.Uri` instead of WHATWG: a missing scheme becomes `https://`, or `http://` for a loopback host. Userinfo, a query, a fragment, a scheme other than http(s), and plain HTTP on a non-loopback host are `INVALID_APP_URL`, with upstream's messages. A base URL set in code (`E2EFixture.BaseUrl`, `E2ESessionOptions.BaseUrl`) follows the same rule. Port `0` on `127.0.0.1` or `[::1]` asks for a free port and needs an `app.command` to start the app on it (`INVALID_CONFIG` without one, or with `reuseExisting`); any other host with port `0` is `INVALID_APP_URL`. A base URL set in code with port `0` is `INVALID_CONFIG`, since only `app.command` starts an app on a free port. Keys of the old .NET shape (`app`, `engine`, `agent`, `timeouts`) fail with a hint to the upstream key. + +`app.command` and `app.readyUrl` port upstream's `config/command.ts` and `config/app.ts` (`CommandConfig`, `Command.Normalize`, `TargetApps.ResolveTargetApp`), and `ManagedProcess` ports `run/managed-process.ts` (readiness polling, the preflight probe and `reuseExisting`, the startup-failure tail of `command.log`, `inheritEnv`). `executable` is started with an argument list, never a shell. `{port}` expands in `args`, in `env` values, and in `readyUrl`, and a `url` on port `0` takes a free loopback port (`AppPorts.AllocateTargetPorts`, upstream's `allocateTargetPorts`). The first test, `e2e explore`, or `e2e mcp` session that needs the command starts it (`E2EFixture.StartSessionAsync`, `Explore`, and the MCP session call `RunApps.StartAsync`) and it runs until the process ends, where `ManagedProcess.KillManagedProcessGroups` takes it down on the way out; on Windows the process runs in its own job object (`ProcessJob`, .NET-only), which ends it and what it started when it is stopped, and when this process is killed outright, as a test host is once its run ends. On Unix a descendant is read with `pgrep -P` when the app is stopped, so one that outlives a leader that exited earlier is not found. The app command differs from upstream's in these ways: + +- There is no runner, so a command starts once per test process (a .NET-only holder, `RunApps`), not once per run. Two processes that declare one command, such as `dotnet test` and `e2e mcp`, each start their own, so give a shared app a fixed port and `reuseExisting`. Upstream's `SharedAppProcesses` pool, `declaredCommands`, and the engine `prepare` hooks that `startDeclaredProcesses` shares a provision with are not ported (one target, no pool). +- A started process is stopped with `SIGTERM` and then `Process.Kill(entireProcessTree: true)` after `shutdownTimeout`. On Windows, which has no signal to ask a process to stop, the first step is the kill. There are no process groups, so a descendant that outlives its leader is not signalled; the two `managed-process` integration tests about that (`stops a surviving child after the command leader exits on SIGTERM`, `lets a child finish its shutdown after the leader exits`) are not ported. `stdout` and `stderr` are read together into `log` (or dropped when there is none), line by line as the process writes them. +- A cancelled token ends `ManagedProcess.StartAsync` by throwing `OperationCanceledException` after taking the process down, where upstream's `start` returns and the caller reads the signal. `startupTimeout` and `shutdownTimeout` are `TimeSpan`s (`CommandConfig.StartupTimeout`, `ShutdownTimeout`), and a value over 2147483647 milliseconds, which a .NET timer cannot wait, is `INVALID_CONFIG`. A readiness probe still waiting when the process exits ends there. +- JSON has no `secrets.get()` handle, so the message that names one for a string value of `args`, `env`, or `url` is not reachable, and an `env` entry of `null` stands for upstream's `undefined`. `command.log` containment resolves symlinks as `relativeToProjectRoot` does (`Paths.RelativeToProjectRoot`). An app key outside the port's set is named in the port's own words (`unknown targets[0].app key "readyURL"`), where upstream says `target "web" app has unknown key`; the command's own keys use upstream's message. +- The resolved app has `Base`, `PortRequest`, `Command`, and `ReadyUrl`; `site`, `environment`, and `identity` are not ported, and neither is the config digest (`digestCommand`, `digestTargetApp`), so the `config.test.ts` tests that digest an app, and the `identity` and `environment` tests, are not ported. +- Tests: `ManagedProcess/` ports the `it` of `tests/unit/managed-process.test.ts` and `ManagedProcessIntegration/` those of `tests/integration/managed-process.test.ts`, with the program they start (`E2E.Tests.App`) where upstream runs `node -e`; the notice tests wait out a real budget where upstream moves a fake clock. `AppPorts/` ports `app-ports.test.ts` (partial: one target, and no `identity` or digest). `Config/` ports the command and `readyUrl` tests of `config.test.ts`. `RunApps/` is .NET-only. Not ported: `target-processes.test.ts` and `shared-app-processes.test.ts`, which drive the runner. Differences: diff --git a/E2E.slnx b/E2E.slnx index 5dc734d0..35f64133 100644 --- a/E2E.slnx +++ b/E2E.slnx @@ -36,6 +36,7 @@ + diff --git a/docs-site/public/api/E2E.CommandConfig.html b/docs-site/public/api/E2E.CommandConfig.html new file mode 100644 index 00000000..54999ea8 --- /dev/null +++ b/docs-site/public/api/E2E.CommandConfig.html @@ -0,0 +1,13 @@ + + + + + + +Redirecting… + + + +

Continue

+ + diff --git a/docs-site/src/content/docs/api/E2E/AppConfig.md b/docs-site/src/content/docs/api/E2E/AppConfig.md index 60e94b13..69dd5057 100644 --- a/docs-site/src/content/docs/api/E2E/AppConfig.md +++ b/docs-site/src/content/docs/api/E2E/AppConfig.md @@ -21,7 +21,7 @@ public sealed class AppConfig **Inheritance:** [object](https://learn.microsoft.com/dotnet/api/system.object) -[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L700) +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L708) ## Properties @@ -40,7 +40,7 @@ The build a device target installs, relative to the folder of `e2e.config.json`. **Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) -[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L712) +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L720) @@ -57,7 +57,43 @@ The bundle id (iOS) or package (Android) of an installed app a device target lau **Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) -[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L709) +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L717) + + + + +### Command + + +```csharp +public CommandConfig? Command { get; init; } +``` + + +The command that starts the app (upstream's `app.command`), started before the first test and stopped when the process ends. +`{port}` in [Args](/api/E2E/CommandConfig/#E2E_CommandConfig_Args) and [Env](/api/E2E/CommandConfig/#E2E_CommandConfig_Env) is the port the app is served on. A +[Url](#E2E_AppConfig_Url) with port 0 asks for a free one, and needs this. + +**Value:** [CommandConfig](/api/E2E/CommandConfig/) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L727) + + + + +### ReadyUrl + + +```csharp +public string? ReadyUrl { get; init; } +``` + + +The URL that answers once [Command](#E2E_AppConfig_Command) has started the app (upstream's `app.readyUrl`). Unset, [Url](#E2E_AppConfig_Url). + +**Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L730) @@ -75,4 +111,4 @@ Relative URLs resolve against it. **Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) -[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L706) +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/E2EConfig.cs#L714) diff --git a/docs-site/src/content/docs/api/E2E/CommandConfig.md b/docs-site/src/content/docs/api/E2E/CommandConfig.md new file mode 100644 index 00000000..e3e24aa9 --- /dev/null +++ b/docs-site/src/content/docs/api/E2E/CommandConfig.md @@ -0,0 +1,183 @@ +--- +title: "CommandConfig" +description: "The command a target's app.command starts the app with (upstream's CommandConfig). It is a program and its arguments, never a shell line: the executable is s..." +editUrl: false +prev: false +next: false +sidebar: + label: "CommandConfig" + hidden: true +--- + + + +Namespace: [E2E](/api/E2E/) + + +```csharp +public sealed class CommandConfig +``` + + +The command a target's `app.command` starts the app with (upstream's `CommandConfig`). It is a program and its arguments, +never a shell line: the executable is started directly and each argument is passed as it is. + +**Inheritance:** [object](https://learn.microsoft.com/dotnet/api/system.object) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L17) + +## Properties + + + + +### Args + + +```csharp +public IReadOnlyList? Args { get; init; } +``` + + +Its arguments, passed one by one. `{port}` is replaced by the port the app is served on. + +**Value:** [IReadOnlyList<string>](https://learn.microsoft.com/dotnet/api/system.collections.generic.ireadonlylist-1) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L23) + + + + +### Cwd + + +```csharp +public string? Cwd { get; init; } +``` + + +The working directory, relative to the config's directory. Unset, that directory. + +**Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L26) + + + + +### Env + + +```csharp +public IReadOnlyDictionary? Env { get; init; } +``` + + +Variables the process gets, on top of [InheritEnv](#E2E_CommandConfig_InheritEnv). `{port}` in a value is replaced. + +**Value:** [IReadOnlyDictionary<string, string>](https://learn.microsoft.com/dotnet/api/system.collections.generic.ireadonlydictionary-2) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L29) + + + + +### Executable + + +```csharp +public required string Executable { get; init; } +``` + + +The program to start. Required. + +**Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L20) + + + + +### InheritEnv + + +```csharp +public IReadOnlyList? InheritEnv { get; init; } +``` + + +Variable names, or prefixes ending in `*`, copied from the environment of the run. The process gets only these, +[Env](#E2E_CommandConfig_Env), and `PATH`, `HOME`, `TMPDIR`, `TMP`, `TEMP`, `SystemRoot`, and `COMSPEC`. + +**Value:** [IReadOnlyList<string>](https://learn.microsoft.com/dotnet/api/system.collections.generic.ireadonlylist-1) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L35) + + + + +### Log + + +```csharp +public string? Log { get; init; } +``` + + +A file inside the config's directory that takes the output of the process, appended to across runs. + +**Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L44) + + + + +### ReuseExisting + + +```csharp +public bool? ReuseExisting { get; init; } +``` + + +Attach to an app that already answers at the readiness URL instead of failing. Ignored in CI. + +**Value:** [bool?](https://learn.microsoft.com/dotnet/api/system.boolean) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L47) + + + + +### ShutdownTimeout + + +```csharp +public TimeSpan? ShutdownTimeout { get; init; } +``` + + +How long the app has to stop before it is killed. 10 seconds when unset. + +**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L41) + + + + +### StartupTimeout + + +```csharp +public TimeSpan? StartupTimeout { get; init; } +``` + + +How long the app has to become ready. 60 seconds when unset. + +**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Config/Command.cs#L38) diff --git a/docs-site/src/content/docs/api/E2E/index.md b/docs-site/src/content/docs/api/E2E/index.md index 6984a815..384f07ea 100644 --- a/docs-site/src/content/docs/api/E2E/index.md +++ b/docs-site/src/content/docs/api/E2E/index.md @@ -44,6 +44,7 @@ The core. Agent, Screen, Expect, the engines, model clients, config, and the rep | [CacheInfo](/api/E2E/CacheInfo/) | | | [CacheLookup](/api/E2E/CacheLookup/) | | | [ClickOptions](/api/E2E/ClickOptions/) | What a tap-family action takes. Upstream's ClickOptions, of which the port has the modifiers. | +| [CommandConfig](/api/E2E/CommandConfig/) | The command a target's app.command starts the app with (upstream's CommandConfig). It is a program and its arguments, never a shell line: the executable is started directly and each argument is passed as it is. | | [ConfigurationException](/api/E2E/ConfigurationException/) | e2e.config.json is invalid. The code is INVALIDCONFIG. | | [Credentials](/api/E2E/Credentials/) | Reads E2EUSER{NAME}USERNAME and E2EUSER{NAME}PASSWORD. The name is uppercased and dashes become underscores. | | [DefaultAgent](/api/E2E/DefaultAgent/) | Upstream's createBuiltInAgent (agent/default-agent.ts): the tool loop with the built-in rules, tools, and opening prompt. | diff --git a/skills/create-verification-skill/SKILL.md b/skills/create-verification-skill/SKILL.md index 14008494..3c6345d7 100644 --- a/skills/create-verification-skill/SKILL.md +++ b/skills/create-verification-skill/SKILL.md @@ -43,7 +43,7 @@ observe. | Question | Where the answer is | What it decides | | --- | --- | --- | | Surface | What a user touches. A web UI, an iOS or Android app, an API | One skill per primary surface; name the others for a later run. A web UI is the config's web target. A mobile app is driven by `MobileEngine` in code (topic `setup`); `e2e mcp` also drives the documented mobile and Windows targets (topics `mobile` and `windows`). An API with no UI has no engine to drive; tests call it with `HttpClient`, and the skill says so. A Windows desktop app is driven by `WinAppEngine` through a `windows` target (topic `windows`), which `e2e mcp` also drives on Windows; a desktop app on another OS has no e2e engine, say so and stop | -| Run | The documented dev command, its ports, env vars, seed data, auth | The command the generated skill tells the reader to start by hand. The port has no `app.command`, so the app must answer on the target's `app.url` before `dotnet test` or `open_session` (topic `setup`). A dev server that compiles a route on first visit reads as broken to an agent, so prefer a production build command when the repo has one | +| Run | The documented dev command, its ports, env vars, seed data, auth | The command the generated skill tells the reader to start by hand, or the target's `app.command`, which starts the app and polls `app.readyUrl` (default `app.url`) before the first test, `e2e explore`, or `open_session` needs it (topic `setup`). A dev server that compiles a route on first visit reads as broken to an agent, so prefer a production build command when the repo has one | | e2e state | `e2e.config.json`, a test project that references `E2E.NUnit` or `E2E.XUnit.V3`, the `E2E.Cli` tool, `.agents/skills/e2e-dotnet/`, the MCP server in `.mcp.json` or `.cursor/mcp.json`, and any Playwright or Cypress suite | Nothing there means topic `setup` first, then `claude mcp add e2e -- e2e mcp` or a `.mcp.json` entry (topic `mcp`). An existing Playwright or Cypress suite is the best source of locators, seeds, and sign-in steps; read it. A config with no model runs the deterministic steps and the MCP tools, and nothing else | | Observe | Screenshots, the `dotnet test` log, and what the app leaves behind in files, rows, sent messages, logs | The side-effect stores the Evidence section names. Find them now | | Isolate | Can two instances run side by side? | Tests share one `app.url`, so two runs against one app share its data. Use `Values.Unique(...)` for data a test creates. A shared instance that cannot be duplicated (one database, one simulator) becomes a rule in the generated skill. Refusing to double-drive it beats corrupting the user's session | diff --git a/skills/e2e-dotnet/SKILL.md b/skills/e2e-dotnet/SKILL.md index dbd4f36f..734047bd 100644 --- a/skills/e2e-dotnet/SKILL.md +++ b/skills/e2e-dotnet/SKILL.md @@ -41,15 +41,33 @@ Two sibling skills install with `npx skills add hardkoded/e2e-dotnet --skill OpenSessionAsync(string id, OpenSessionOptions option _sessions.ClaimEngine(id, target.Name); } + // The target's app command starts before the session opens, and the URL is on the port it was started on. + target = await RunApps.StartAsync(config, target, null, abort.Token).ConfigureAwait(false); attempt = await Standalone.OpenAttemptAsync( new E2ESessionOptions { diff --git a/src/E2E/Config/App.cs b/src/E2E/Config/App.cs new file mode 100644 index 00000000..01a89f39 --- /dev/null +++ b/src/E2E/Config/App.cs @@ -0,0 +1,111 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using System.Collections.ObjectModel; +using System.Globalization; +using E2E.Internal; + +namespace E2E; + +/// +/// A declared URL with port 0 asks the run for a free port on its loopback host (upstream's PortRequest). +/// is the one the run assigned, null until it did: a config resolved outside a run keeps the :0 URL. +/// +/// The hostname to bind, as the URL spelled it. +/// The port the run assigned, or null. +internal sealed record PortRequest(string Host, int? Port); + +/// +/// The app one target drives, as resolved the target's app declaration (upstream's +/// ResolvedApp): the part of it the port has, the base URL on the run's port and the command that starts the app there. +/// +/// The normalized base URL on the run's port, or null without a URL. +/// The free-port request the declared URL made, or null when it names a port or there is no URL. +/// app.command with {port} expanded to the port the app is served on. +/// Readiness probe for ; set whenever the command is. +internal sealed record ResolvedApp(NormalizedBaseUrl? Base, PortRequest? PortRequest, CommandConfig? Command, string? ReadyUrl); + +/// Per-target app resolution (upstream's config/app.ts): the declared app checked, then resolved on the port the run assigned when app.url asked for a free one. +internal static class TargetApps +{ + /// The token a command or a readiness URL writes where the app's port goes. + private const string PortToken = "{port}"; + + /// + /// Resolves one target's app from its declaration. is the free port the run assigned to a URL declared with + /// port 0: it replaces the 0 in the base URL and the default readiness probe. {port} in the command and its readiness URL + /// expands to the port the app is served on, assigned or fixed; while a requested port is unassigned that is 0, so the config resolves + /// the same way before and after assignment. A free port nothing starts on, and reuseExisting on a free port, are refused: what + /// already answers cannot serve a port the run assigns. + /// + public static ResolvedApp ResolveTargetApp(string targetName, AppConfig app, string projectRoot, int? port = null) + { + var where = "target \"" + targetName + "\""; + var declaredBase = app.Url is null ? null : Urls.NormalizeBaseUrl(app.Url); + var portRequest = declaredBase is null || !Urls.RequestsFreePort(declaredBase) ? null : new PortRequest(new Uri(declaredBase.Href).Host, port); + var baseUrl = declaredBase is not null && portRequest?.Port is { } assigned ? Urls.WithPort(declaredBase, assigned) : declaredBase; + int? appPort = baseUrl is null ? null : Urls.PortOf(baseUrl); + + if (app.Command is null && portRequest is not null) + { + throw new ConfigurationException("INVALID_CONFIG", where + " app.url asks for a free port (port 0), but nothing starts on it: add app.command to start the app there on " + PortToken); + } + + var command = app.Command is null ? null : ExpandCommandPort(app.Command, where + " app.command", appPort); + if (command is not null) + { + Command.CheckLog(command, where + " app.command", projectRoot); + } + + if (command?.ReuseExisting == true && portRequest is not null) + { + throw new ConfigurationException( + "INVALID_CONFIG", + where + " app.command.reuseExisting cannot find an app already running on a free port: port 0 is a new port every run; give the app a fixed port, or drop reuseExisting"); + } + + var readyUrl = app.ReadyUrl is null ? baseUrl?.Href : HttpUrl(ExpandPort(app.ReadyUrl, appPort, where + " app.readyUrl"), where + " app.readyUrl"); + if (command is not null && readyUrl is null) + { + throw new ConfigurationException("APP_URL_REQUIRED", where + " declares app.command without a URL to poll: declare app.url or app.readyUrl beside it"); + } + + return new ResolvedApp(baseUrl, portRequest, command, command is null ? null : readyUrl); + } + + /// A present value that must be an http(s) URL (upstream's httpUrl). + public static string HttpUrl(string value, string label) => + Uri.TryCreate(value, UriKind.Absolute, out var parsed) && parsed.Scheme is "http" or "https" + ? value + : throw new ConfigurationException("INVALID_CONFIG", label + " must be an http(s) URL"); + + /// Substitutes {port} in one configured string. A target without a URL has no port to offer, so the token there is a config error naming the field. + private static string ExpandPort(string value, int? port, string label) + { + if (!value.Contains(PortToken, StringComparison.Ordinal)) + { + return value; + } + + return port is { } number + ? value.Replace(PortToken, number.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal) + : throw new ConfigurationException("INVALID_CONFIG", label + " uses " + PortToken + ", but the target declares no url to take the port from"); + } + + /// The command with {port} expanded in every args entry and env value. + private static CommandConfig ExpandCommandPort(CommandConfig command, string label, int? port) => new() + { + Executable = command.Executable, + Args = command.Args?.Select(arg => ExpandPort(arg, port, label + ".args")).ToList().AsReadOnly(), + Cwd = command.Cwd, + Env = command.Env is null + ? null + : new ReadOnlyDictionary(command.Env.ToDictionary(pair => pair.Key, pair => ExpandPort(pair.Value, port, label + ".env." + pair.Key), StringComparer.Ordinal)), + InheritEnv = command.InheritEnv, + StartupTimeout = command.StartupTimeout, + ShutdownTimeout = command.ShutdownTimeout, + Log = command.Log, + ReuseExisting = command.ReuseExisting, + }; +} diff --git a/src/E2E/Config/Command.cs b/src/E2E/Config/Command.cs new file mode 100644 index 00000000..eb4f3cac --- /dev/null +++ b/src/E2E/Config/Command.cs @@ -0,0 +1,243 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using System.Collections.ObjectModel; +using System.Globalization; +using System.Text.Json; +using System.Text.RegularExpressions; +using E2E.Internal; + +namespace E2E; + +/// +/// The command a target's app.command starts the app with (upstream's CommandConfig). It is a program and its arguments, +/// never a shell line: the executable is started directly and each argument is passed as it is. +/// +public sealed class CommandConfig +{ + /// The program to start. Required. + public required string Executable { get; init; } + + /// Its arguments, passed one by one. {port} is replaced by the port the app is served on. + public IReadOnlyList? Args { get; init; } + + /// The working directory, relative to the config's directory. Unset, that directory. + public string? Cwd { get; init; } + + /// Variables the process gets, on top of . {port} in a value is replaced. + public IReadOnlyDictionary? Env { get; init; } + + /// + /// Variable names, or prefixes ending in *, copied from the environment of the run. The process gets only these, + /// , and PATH, HOME, TMPDIR, TMP, TEMP, SystemRoot, and COMSPEC. + /// + public IReadOnlyList? InheritEnv { get; init; } + + /// How long the app has to become ready. 60 seconds when unset. + public TimeSpan? StartupTimeout { get; init; } + + /// How long the app has to stop before it is killed. 10 seconds when unset. + public TimeSpan? ShutdownTimeout { get; init; } + + /// A file inside the config's directory that takes the output of the process, appended to across runs. + public string? Log { get; init; } + + /// Attach to an app that already answers at the readiness URL instead of failing. Ignored in CI. + public bool? ReuseExisting { get; init; } +} + +/// +/// The shape of an app.command, checked once where it is declared (upstream's config/command.ts), so the rest of the +/// resolution only ever sees checked values. +/// +internal static partial class Command +{ + private static readonly string[] CommandKeys = + ["executable", "args", "cwd", "env", "inheritEnv", "startupTimeout", "shutdownTimeout", "log", "reuseExisting"]; + + private const long MaxTimerMs = int.MaxValue; + + /// + /// Checks the shape of one command and returns it (upstream's normalizeCommand): only the keys a command takes, a non-empty + /// executable, string args and env values, inheritEnv names and trailing-* prefixes, when set positive integer + /// timeouts, a non-empty log, and a boolean reuseExisting. A misspelled key would otherwise be dropped without a word. + /// + public static CommandConfig Normalize(JsonElement command, string label) + { + if (command.ValueKind != JsonValueKind.Object) + { + throw Invalid(label + " must be an object"); + } + + RejectUnknownKeys(label, command); + if (Get(command, "executable") is not { ValueKind: JsonValueKind.String } executable || string.IsNullOrWhiteSpace(executable.GetString())) + { + throw Invalid(label + ".executable is required"); + } + + IReadOnlyList? args = null; + if (Get(command, "args") is { } rawArgs) + { + if (rawArgs.ValueKind != JsonValueKind.Array) + { + throw Invalid(label + ".args must be an array of strings"); + } + + args = rawArgs.EnumerateArray().Select((item, index) => CommandString(item, label + ".args[" + Number(index) + "]")).ToList().AsReadOnly(); + } + + IReadOnlyDictionary? env = null; + if (Get(command, "env") is { } rawEnv) + { + if (rawEnv.ValueKind != JsonValueKind.Object) + { + throw Invalid(label + ".env must be an object of variable name to string"); + } + + // A null value is dropped, as a process start drops it: a variable that is unset starts the process without it. + env = new ReadOnlyDictionary(rawEnv.EnumerateObject() + .Where(entry => entry.Value.ValueKind != JsonValueKind.Null) + .ToDictionary(entry => entry.Name, entry => CommandString(entry.Value, label + ".env." + entry.Name), StringComparer.Ordinal)); + } + + IReadOnlyList? inheritEnv = null; + if (Get(command, "inheritEnv") is { } rawInherit) + { + if (rawInherit.ValueKind != JsonValueKind.Array) + { + throw Invalid(label + ".inheritEnv must be an array of variable names"); + } + + inheritEnv = rawInherit.EnumerateArray().Select((item, index) => InheritPattern(item, label + ".inheritEnv[" + Number(index) + "]")).ToList().AsReadOnly(); + } + + string? cwd = null; + if (Get(command, "cwd") is { } rawCwd) + { + cwd = rawCwd.ValueKind == JsonValueKind.String ? rawCwd.GetString() : throw Invalid(label + ".cwd must be a string"); + } + + var startupTimeout = PositiveMs(Get(command, "startupTimeout"), label + ".startupTimeout"); + var shutdownTimeout = PositiveMs(Get(command, "shutdownTimeout"), label + ".shutdownTimeout"); + string? log = null; + if (Get(command, "log") is { } rawLog) + { + log = rawLog.ValueKind == JsonValueKind.String && !string.IsNullOrWhiteSpace(rawLog.GetString()) + ? rawLog.GetString() + : throw Invalid(label + ".log must be a non-empty path"); + } + + bool? reuseExisting = null; + if (Get(command, "reuseExisting") is { } rawReuse) + { + reuseExisting = rawReuse.ValueKind is JsonValueKind.True or JsonValueKind.False + ? rawReuse.GetBoolean() + : throw Invalid(label + ".reuseExisting must be a boolean"); + } + + return new CommandConfig + { + Executable = executable.GetString()!, + Args = args, + Cwd = cwd, + Env = env, + InheritEnv = inheritEnv, + StartupTimeout = startupTimeout, + ShutdownTimeout = shutdownTimeout, + Log = log, + ReuseExisting = reuseExisting, + }; + } + + /// A log must be a file inside the project root: a log outside it would let config write anywhere (upstream's checkLog). + public static void CheckLog(CommandConfig command, string label, string projectRoot) + { + if (command.Log is { } log && !Paths.InsideProjectRoot(projectRoot, Path.GetFullPath(log, projectRoot))) + { + throw Invalid(label + ".log must be a file inside the project root, got " + JsonSerializer.Serialize(log)); + } + } + + /// A value as the author wrote it, for messages that quote the offending input (upstream's describeValue). + internal static string DescribeValue(JsonElement value) => value.ValueKind switch + { + JsonValueKind.String => JsonSerializer.Serialize(value.GetString()), + JsonValueKind.Number => value.GetRawText(), + JsonValueKind.True => "true", + JsonValueKind.False => "false", + JsonValueKind.Null => "null", + JsonValueKind.Array => "an array", + _ => "an object", + }; + + private static JsonElement? Get(JsonElement value, string name) => value.TryGetProperty(name, out var property) ? property : null; + + private static string Number(int value) => value.ToString(CultureInfo.InvariantCulture); + + private static ConfigurationException Invalid(string message) => new("INVALID_CONFIG", message); + + /// Refuses the first key outside the ones a command takes, naming the nearest when it is a plausible typo and every key otherwise. + private static void RejectUnknownKeys(string label, JsonElement command) + { + foreach (var property in command.EnumerateObject()) + { + if (CommandKeys.Contains(property.Name, StringComparer.Ordinal)) + { + continue; + } + + var hint = Suggestions.DidYouMean(property.Name, CommandKeys); + throw Invalid(label + " has unknown key \"" + property.Name + "\"" + (hint.Length == 0 ? "; expected one of " + string.Join(", ", CommandKeys) : hint)); + } + } + + private static string CommandString(JsonElement value, string label) => + value.ValueKind == JsonValueKind.String ? value.GetString()! : throw Invalid(label + " must be a string, got " + DescribeValue(value)); + + /// + /// One inheritEnv entry: a variable name, or a prefix ending in *. A * anywhere else, an = or whitespace, + /// and a bare * are INVALID_CONFIG; a bare * would pass every variable, model keys included. + /// + private static string InheritPattern(JsonElement value, string label) + { + if (value.ValueKind != JsonValueKind.String) + { + throw Invalid(label + " must be a string, got " + DescribeValue(value)); + } + + var text = value.GetString()!; + if (text == "*") + { + throw Invalid(label + " cannot be \"*\": it would pass every variable to the app, model keys included; list the names or prefixes it needs"); + } + + if (!InheritName().IsMatch(text)) + { + throw Invalid(label + " must be a variable name or a prefix ending in * (such as \"STRIPE_*\"), got " + JsonSerializer.Serialize(text)); + } + + return text; + } + + private static TimeSpan? PositiveMs(JsonElement? value, string label) + { + if (value is not { } raw) + { + return null; + } + + if (raw.ValueKind != JsonValueKind.Number || !raw.TryGetInt64(out var ms) || ms <= 0) + { + throw Invalid(label + " must be a positive safe integer of milliseconds, got " + DescribeValue(raw)); + } + + // A .NET timer cannot wait longer than int.MaxValue milliseconds. + return ms > MaxTimerMs + ? throw Invalid(label + " must be at most " + MaxTimerMs.ToString(CultureInfo.InvariantCulture) + " milliseconds, got " + ms.ToString(CultureInfo.InvariantCulture)) + : TimeSpan.FromMilliseconds(ms); + } + + [GeneratedRegex(@"\A[^*=\s]+\*?\z", RegexOptions.CultureInvariant)] + private static partial Regex InheritName(); +} diff --git a/src/E2E/Config/E2EConfig.cs b/src/E2E/Config/E2EConfig.cs index e963c60a..5dea9c6c 100644 --- a/src/E2E/Config/E2EConfig.cs +++ b/src/E2E/Config/E2EConfig.cs @@ -45,12 +45,12 @@ public sealed class E2EConfig private static readonly string[] UnsupportedTargetKeys = ["engine"]; - private static readonly string[] AppKeys = ["url"]; + private static readonly string[] AppKeys = ["url", "command", "readyUrl"]; - private static readonly string[] MobileAppKeys = ["bundleId", "appPath"]; + private static readonly string[] MobileAppKeys = ["bundleId", "appPath", "command", "readyUrl"]; private static readonly string[] UnsupportedAppKeys = - ["identity", "environment", "launchArguments", "permissions", "command", "readyUrl"]; + ["identity", "environment", "launchArguments", "permissions"]; private static readonly string[] Platforms = ["web", "ios", "android", "windows"]; @@ -221,7 +221,7 @@ private static E2EConfig Parse(string json, string projectRoot, string? configPa ConfigPath = configPath, ProjectRoot = projectRoot, Ci = ci, - Targets = ReadTargets(Property(root, "targets"), trace, video), + Targets = ReadTargets(Property(root, "targets"), projectRoot, trace, video), Timeout = Milliseconds(root, "timeout", E2EDefaults.TestTimeout), LaunchTimeout = Milliseconds(root, "launchTimeout", E2EDefaults.LaunchTimeout), ActionTimeout = Milliseconds(root, "actionTimeout", E2EDefaults.ActionTimeout), @@ -235,7 +235,7 @@ private static E2EConfig Parse(string json, string projectRoot, string? configPa } } - private static List ReadTargets(JsonElement? raw, RunRecording trace, RunRecording video) + private static List ReadTargets(JsonElement? raw, string projectRoot, RunRecording trace, RunRecording video) { if (raw is null) { @@ -286,10 +286,15 @@ private static List ReadTargets(JsonElement? raw, RunRecording tra CheckKeys(rawApp, "targets[0].app", pinned ? MobileAppKeys : AppKeys, UnsupportedAppKeys, null); app = new AppConfig { - Url = OptionalString(rawApp, "url", "targets[0].app.url") is { } url ? Urls.NormalizeAppUrl(url, "targets[0].app.url").Href : null, + Url = OptionalString(rawApp, "url", "targets[0].app.url") is { } url ? Urls.NormalizeBaseUrl(url).Href : null, BundleId = OptionalString(rawApp, "bundleId", "targets[0].app.bundleId"), AppPath = OptionalString(rawApp, "appPath", "targets[0].app.appPath"), + Command = Property(rawApp, "command") is { } command ? Command.Normalize(command, "target \"" + name + "\" app.command") : null, + ReadyUrl = OptionalString(rawApp, "readyUrl", "targets[0].app.readyUrl"), }; + + // Resolved once here so a bad command, a free port nothing starts on, or a {port} without a URL fails at load, as upstream's resolveConfig does. + TargetApps.ResolveTargetApp(name, app, projectRoot); } return [new TargetConfig { Name = name, Platform = platform, App = app, Trace = trace.For(ownTrace), Video = video.For(ownVideo) }]; @@ -670,6 +675,9 @@ public sealed class TargetConfig public required AppConfig App { get; init; } + /// This target with another app: the one resolved on the port its command was started on. + internal TargetConfig WithApp(AppConfig app) => new() { Name = Name, Platform = Platform, App = app, Trace = Trace, Video = Video }; + /// Which attempts write trace.md: E2E_TRACE, else this target's trace, else the config's, else retain-on-failure. public ResolvedRecording Trace { get; init; } = new(RecordingMode.RetainOnFailure, RecordingSource.Default); @@ -710,6 +718,19 @@ public sealed class AppConfig /// The build a device target installs, relative to the folder of e2e.config.json. public string? AppPath { get; init; } + + /// + /// The command that starts the app (upstream's app.command), started before the first test and stopped when the process ends. + /// {port} in and is the port the app is served on. A + /// with port 0 asks for a free one, and needs this. + /// + public CommandConfig? Command { get; init; } + + /// The URL that answers once has started the app (upstream's app.readyUrl). Unset, . + public string? ReadyUrl { get; init; } + + /// This app with another URL: the one on the port its command was started on. + internal AppConfig WithUrl(string? url) => new() { Url = url, BundleId = BundleId, AppPath = AppPath, Command = Command, ReadyUrl = ReadyUrl }; } /// diff --git a/src/E2E/Explore/Explore.cs b/src/E2E/Explore/Explore.cs index 48ad3c57..cdeab240 100644 --- a/src/E2E/Explore/Explore.cs +++ b/src/E2E/Explore/Explore.cs @@ -223,6 +223,8 @@ public static async Task ExploreAsync(ExploreOptions options, Ca notice(message); } + // The target's app command starts before the session opens, and the URL is on the port it was started on. + target = await RunApps.StartAsync(config, target, notice, cancellationToken).ConfigureAwait(false); var sessionOptions = new E2ESessionOptions { Engine = engine, diff --git a/src/E2E/Internal/Paths.cs b/src/E2E/Internal/Paths.cs index 43858e1d..0003ec30 100644 --- a/src/E2E/Internal/Paths.cs +++ b/src/E2E/Internal/Paths.cs @@ -14,6 +14,24 @@ internal static class Paths /// public static string OnDiskPath(string target) => OnDiskPath(target, 0); + /// + /// relative to the project root once both are resolved through the filesystem, or null when it is not strictly + /// below the root (upstream's relativeToProjectRoot): an in-project symlink pointing outside is refused, a symlinked project root + /// still counts as the root, and a name that merely starts with .. is an ordinary entry. + /// + public static string? RelativeToProjectRoot(string projectRoot, string target) + { + var relative = Path.GetRelativePath(OnDiskPath(projectRoot), OnDiskPath(target)); + var inside = relative.Length > 0 && relative != "." && relative != ".." + && !relative.StartsWith(".." + Path.DirectorySeparatorChar, StringComparison.Ordinal) + && !relative.StartsWith(".." + Path.AltDirectorySeparatorChar, StringComparison.Ordinal) + && !Path.IsPathRooted(relative); + return inside ? relative : null; + } + + /// Whether is a path strictly below the project root; see . + public static bool InsideProjectRoot(string projectRoot, string target) => RelativeToProjectRoot(projectRoot, target) is not null; + private static string OnDiskPath(string target, int hops) { var full = Path.GetFullPath(target); diff --git a/src/E2E/Internal/Urls.cs b/src/E2E/Internal/Urls.cs index 49b2bc5b..909144af 100644 --- a/src/E2E/Internal/Urls.cs +++ b/src/E2E/Internal/Urls.cs @@ -63,15 +63,15 @@ public static NormalizedBaseUrl NormalizeBaseUrl(string raw) } /// - /// Normalizes an app URL the run will use as given. Port 0 asks the run to start the app on a free port, and the port - /// has no app.command to do that, so it is INVALID_CONFIG where upstream starts the command. + /// Normalizes an app URL a session will use as given, one set in code. Port 0 asks the run to start the app on a free port with + /// app.command, and a session holds the URL the app is already served on, so it is INVALID_CONFIG. /// public static NormalizedBaseUrl NormalizeAppUrl(string raw, string where) { var url = NormalizeBaseUrl(raw); if (RequestsFreePort(url)) { - throw new ConfigurationException("INVALID_CONFIG", where + " asks for a free port (port 0), but nothing starts on it: the .NET port has no app.command, so start the app on a fixed port"); + throw new ConfigurationException("INVALID_CONFIG", where + " asks for a free port (port 0), which only app.command can start an app on: give the URL the app is served on"); } return url; diff --git a/src/E2E/Run/AppPorts.cs b/src/E2E/Run/AppPorts.cs new file mode 100644 index 00000000..b15230e9 --- /dev/null +++ b/src/E2E/Run/AppPorts.cs @@ -0,0 +1,58 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using System.Net; +using System.Net.Sockets; + +namespace E2E; + +/// +/// Free ports for app URLs declared with port 0 (upstream's run/app-ports.ts). The run picks one once, after the config loads and before +/// anything spawns, and resolves the target's app on it. +/// +internal static class AppPorts +{ + /// + /// The target's app resolved on a port for its URL when that asked for one: a free port on its loopback host. A target that names its port, + /// or has no URL, resolves as it is. A loopback host this machine cannot bind is a bad app URL, reported before anything starts. This is + /// upstream's allocateTargetPorts, which hands the one target back as it now reads. + /// + public static ResolvedApp AllocateTargetPorts(string targetName, AppConfig app, string projectRoot) + { + var pending = TargetApps.ResolveTargetApp(targetName, app, projectRoot); + if (pending.PortRequest is not { } request) + { + return pending; + } + + return TargetApps.ResolveTargetApp(targetName, app, projectRoot, Reserve(targetName, request.Host)); + } + + /// Binds an ephemeral port on and releases it, so the command can take it. + private static int Reserve(string targetName, string host) + { + try + { + // A URL spells an IPv6 host in brackets; the socket API wants it bare. + var address = IPAddress.Parse(host.StartsWith('[') && host.EndsWith(']') ? host[1..^1] : host); + var listener = new TcpListener(address, 0); + try + { + listener.Start(); + return ((IPEndPoint)listener.LocalEndpoint).Port; + } + finally + { + listener.Stop(); + } + } + catch (Exception ex) when (ex is SocketException or FormatException or ArgumentException) + { + throw new ConfigurationException( + "INVALID_APP_URL", + "target \"" + targetName + "\" asks for a free port on " + host + ", which this machine cannot bind: " + ex.Message, + ex); + } + } +} diff --git a/src/E2E/Run/E2EFixture.cs b/src/E2E/Run/E2EFixture.cs index 931e428e..8823c9e9 100644 --- a/src/E2E/Run/E2EFixture.cs +++ b/src/E2E/Run/E2EFixture.cs @@ -179,6 +179,16 @@ protected async Task StartSessionAsync(string title, int attempt, ActionHow a spawned process counts as ready: a URL that answers, or the process itself exiting with code 0 (upstream's Readiness). +/// The URL to poll; null means the process exiting with code 0 is the readiness event. +internal sealed record Readiness(string? ReadyUrl = null) +{ + /// The process exiting with code 0 is the readiness event. + public static Readiness WaitForExit { get; } = new(); +} + +/// +/// What the runner tells a managed process about the run: whether CI mode is active (where reuseExisting is ignored), the environment the +/// run resolves against (what inheritEnv copies from; default the process environment), where to narrate run-level progress such as a +/// reused app, and where to report the process starting and becoming ready. +/// +internal sealed class ManagedProcessHooks +{ + public bool Ci { get; init; } + + public IReadOnlyDictionary? Env { get; init; } + + public Action? Notice { get; init; } + + public Action? Starting { get; init; } + + /// The label, how long the start took, and whether the process attached to one already serving the URL instead of spawning. + public Action? Ready { get; init; } +} + +/// What one started process, or everything a run or a session started, is released by (upstream's AppProcesses). +internal interface IAppProcesses +{ + /// Stops what was started, in reverse. Every failure is reported through and never skips the rest. + Task StopAsync(Action onFailure); + + /// False once what was started has ended, so a holder can start it again (a .NET-only member: the port's holder outlives its callers). + bool Running { get; } +} + +/// +/// One command the runner owns: spawned, waited on until its readiness contract holds, and terminated signal-then-force (upstream's +/// ManagedProcess). label names it in every error (target "web" command, service "postgres"). +/// +internal sealed partial class ManagedProcess : IAsyncDisposable +{ + private static readonly string[] InheritedEnv = ["PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SystemRoot", "COMSPEC"]; + + // Readiness polling starts fast and backs off; a booting server answers late, not on a schedule. + private static readonly TimeSpan ReadyPollMin = TimeSpan.FromMilliseconds(25); + private static readonly TimeSpan ReadyPollMax = TimeSpan.FromMilliseconds(250); + + // One readiness probe never outlives this, so a half-open server cannot stall the deadline check. + private static readonly TimeSpan ReadyProbeTimeout = TimeSpan.FromSeconds(2); + + // A wait past half its budget narrates once, so a stalled service is visible while it stalls and not only 180 s later. + // Under this floor a short budget stays quiet: half of it is not long enough to call a stall. + private static readonly TimeSpan ProgressNoticeMin = TimeSpan.FromSeconds(5); + + private static readonly TimeSpan DefaultStartupTimeout = TimeSpan.FromSeconds(60); + private static readonly TimeSpan DefaultShutdownTimeout = TimeSpan.FromSeconds(10); + + // How long a leader's exit waits for the last of its output to land in the log. + private static readonly TimeSpan OutputDrain = TimeSpan.FromSeconds(2); + + // How much of the process's own output a startup failure quotes. + private const int OutputTailLines = 20; + private const int OutputTailBytes = 4_096; + + // command.env and inheritEnv values this long are redacted from the quoted output; shorter ones (a port, a flag) would blank ordinary + // text without hiding anything. The shared redactor has no floor of its own. + private const int RedactedEnvMinLength = 4; + + private static readonly HttpClient Probe = new(new SocketsHttpHandler { AllowAutoRedirect = false, UseProxy = false }) + { + Timeout = Timeout.InfiniteTimeSpan, + }; + + // Every spawned process still running, for the exit that cannot wait on StopAsync. + private static readonly ConcurrentDictionary Live = new(); + + private readonly string _label; + private readonly CommandConfig _command; + private readonly string _projectRoot; + private readonly Readiness _readiness; + private readonly ManagedProcessHooks _hooks; + private Process? _process; + private OutputPump? _pump; + private ProcessJob? _job; + private bool _reusedExisting; + private bool _spawned; + + // Hides command.env and inheritEnv values in a quoted startup failure; built at spawn. + private Redactor _redact = Redactor.None; + + static ManagedProcess() + { + // The last resort before the process exits on the spot, so nothing it started outlives it. + AppDomain.CurrentDomain.ProcessExit += (_, _) => KillManagedProcessGroups(); + } + + public ManagedProcess(string label, CommandConfig command, string projectRoot, Readiness readiness, ManagedProcessHooks? hooks = null) + { + _label = label; + _command = command; + _projectRoot = projectRoot; + _readiness = readiness; + _hooks = hooks ?? new ManagedProcessHooks(); + } + + /// True once spawned a process: not when it attached to one already serving the URL, found one there it may not reuse, or failed to spawn. + public bool Spawned => _spawned; + + /// Whether the app is still there: a process this one spawned that has not exited, or one it attached to. + public bool Running => _reusedExisting || (_process is { } process && !HasExited(process)); + + private static bool HasExited(Process process) + { + try + { + return process.HasExited; + } + catch (InvalidOperationException) + { + return true; + } + } + + /// + /// Kills every process tree this module spawned and has not yet released, synchronously and without waiting: the last resort before the + /// process exits on the spot, so nothing it started outlives it. A reused process was never spawned here and is left alone. + /// + public static void KillManagedProcessGroups() + { + foreach (var process in Live.Keys) + { + KillTree(process); + } + } + + /// + /// Spawns the process and waits for readiness. A token that is already cancelled spawns nothing; one that is cancelled during the wait + /// ends it and takes the process down with it, and the call throws . + /// + /// A readiness URL that already answers before the spawn belongs to a process this run did not start, and HTTP readiness cannot tell which + /// process answered later. So the preflight probe decides up front: with reuseExisting (outside CI) nothing is spawned and + /// leaves the process alone; otherwise the launch fails with APP_ALREADY_RUNNING rather than letting the old + /// server pass as the new command's readiness. The probe spends from the same startupTimeout budget as the wait. + /// + /// + public async Task StartAsync(CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var startedAt = Stopwatch.StartNew(); + _hooks.Starting?.Invoke(_label); + await LaunchAsync(cancellationToken).ConfigureAwait(false); + _hooks.Ready?.Invoke(_label, startedAt.Elapsed, _reusedExisting); + } + + /// Gracefully terminates the process, then force-kills its tree. A reused process was never ours to stop. Safe to call before start and more than once. + public async Task StopAsync() + { + if (_reusedExisting) + { + return; + } + + var process = Interlocked.Exchange(ref _process, null); + var pump = Interlocked.Exchange(ref _pump, null); + if (process is null) + { + return; + } + + try + { + var shutdownTimeout = _command.ShutdownTimeout ?? DefaultShutdownTimeout; + + // The leader may hand its job to a child and exit at once (`npm run dev`), so the tree is read before anything is signalled. + var descendants = OperatingSystem.IsWindows() ? [] : await DescendantsAsync(process.Id).ConfigureAwait(false); + if (!process.HasExited) + { + // Windows has no signal to ask a process to stop: the first step there is already the kill. + if (!OperatingSystem.IsWindows()) + { + Terminate(process); + } + else + { + KillTree(process); + } + + if (!await ExitedWithin(process, shutdownTimeout).ConfigureAwait(false)) + { + KillTree(process); + await ExitedWithin(process, TimeSpan.FromSeconds(5)).ConfigureAwait(false); + } + } + + await EndDescendantsAsync(descendants, shutdownTimeout).ConfigureAwait(false); + + // On Windows what the leader started is in its job, which ends with it whether or not the leader is still there. + _job?.Terminate(); + } + finally + { + Live.TryRemove(process, out _); + Interlocked.Exchange(ref _job, null)?.Dispose(); + if (pump is not null) + { + await pump.DrainAsync(OutputDrain).ConfigureAwait(false); + pump.Dispose(); + } + + process.Dispose(); + } + } + + public async ValueTask DisposeAsync() => await StopAsync().ConfigureAwait(false); + + /// + /// The child's environment: the default names from the process environment, what inheritEnv matches in + /// (only variables that are set), then command.env, each layer replacing the one before. Windows reads names case-insensitively + /// (Path), so there names match and replace that way too. inherited is what inheritEnv copied. + /// + internal static (Dictionary Env, List<(string Name, string Value)> Inherited) ChildEnv( + CommandConfig command, + IReadOnlyDictionary source, + bool caseInsensitive) + { + var comparer = caseInsensitive ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal; + var patterns = command.InheritEnv ?? []; + var names = new HashSet(patterns.Where(pattern => !pattern.EndsWith('*')), comparer); + var prefixes = patterns.Where(pattern => pattern.EndsWith('*')).Select(pattern => pattern[..^1]).ToList(); + var inherited = source + .Where(entry => entry.Value is not null + && (names.Contains(entry.Key) || prefixes.Any(prefix => entry.Key.StartsWith(prefix, caseInsensitive ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal)))) + .Select(entry => (entry.Key, entry.Value!)) + .ToList(); + var byName = new Dictionary(comparer); + foreach (var key in InheritedEnv) + { + if (Environment.GetEnvironmentVariable(key) is { } value) + { + byName[key] = (key, value); + } + } + + foreach (var entry in inherited) + { + byName[entry.Key] = entry; + } + + foreach (var entry in command.Env ?? new Dictionary()) + { + byName[entry.Key] = (entry.Key, entry.Value); + } + + return (byName.Values.ToDictionary(entry => entry.Name, entry => entry.Value, StringComparer.Ordinal), inherited); + } + + private static IReadOnlyDictionary ProcessEnvironment() + { + var result = new Dictionary(StringComparer.Ordinal); + foreach (System.Collections.DictionaryEntry entry in Environment.GetEnvironmentVariables()) + { + result[(string)entry.Key] = (string?)entry.Value; + } + + return result; + } + + /// One readiness probe: true when answers 200 through 499 within . + private static async Task AnswersAsync(string url, TimeSpan timeout, CancellationToken cancellationToken) + { + using var bound = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + bound.CancelAfter(timeout); + try + { + using var response = await Probe.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, bound.Token).ConfigureAwait(false); + return (int)response.StatusCode is >= 200 and <= 499; + } + catch (Exception ex) when (ex is HttpRequestException or OperationCanceledException or InvalidOperationException or IOException or NotSupportedException or UriFormatException) + { + cancellationToken.ThrowIfCancellationRequested(); + return false; + } + } + + private static async Task ExitedWithin(Process process, TimeSpan timeout) + { + using var bound = new CancellationTokenSource(timeout); + try + { + await process.WaitForExitAsync(bound.Token).ConfigureAwait(false); + return true; + } + catch (OperationCanceledException) + { + return process.HasExited; + } + catch (InvalidOperationException) + { + return true; + } + } + + /// The processes below (Unix only; pgrep -P, bounded), so a descendant that outlives its leader can still be ended. + private static async Task> DescendantsAsync(int pid) + { + var found = new List(); + var pending = new Queue([pid]); + while (pending.Count > 0 && found.Count < 256) + { + var parent = pending.Dequeue(); + Process? pgrepProcess = null; + try + { + var info = new ProcessStartInfo("pgrep") { UseShellExecute = false, RedirectStandardOutput = true, RedirectStandardError = true, CreateNoWindow = true }; + info.ArgumentList.Add("-P"); + info.ArgumentList.Add(parent.ToString(CultureInfo.InvariantCulture)); + var pgrep = Process.Start(info); + pgrepProcess = pgrep; + if (pgrep is null) + { + break; + } + + using var bound = new CancellationTokenSource(TimeSpan.FromSeconds(2)); + var error = pgrep.StandardError.ReadToEndAsync(bound.Token); + var text = await pgrep.StandardOutput.ReadToEndAsync(bound.Token).ConfigureAwait(false); + await pgrep.WaitForExitAsync(bound.Token).ConfigureAwait(false); + await error.ConfigureAwait(false); + foreach (var line in text.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + { + if (int.TryParse(line, NumberStyles.None, CultureInfo.InvariantCulture, out var child)) + { + found.Add(child); + pending.Enqueue(child); + } + } + } + catch (Exception ex) when (ex is System.ComponentModel.Win32Exception or InvalidOperationException or OperationCanceledException or IOException) + { + break; + } + finally + { + if (pgrepProcess is not null) + { + if (!HasExited(pgrepProcess)) + { + KillTree(pgrepProcess); + } + + pgrepProcess.Dispose(); + } + } + } + + return found; + } + + /// Asks each descendant still running to stop, and kills what is left once the shutdown budget is spent. + private static async Task EndDescendantsAsync(List pids, TimeSpan shutdownTimeout) + { + var running = pids.Where(IsRunning).ToList(); + if (running.Count == 0) + { + return; + } + + foreach (var pid in running) + { + Signal(pid, 15); + } + + var clock = Stopwatch.StartNew(); + while (clock.Elapsed < shutdownTimeout && running.Any(IsRunning)) + { + await Task.Delay(25).ConfigureAwait(false); + } + + foreach (var pid in running.Where(IsRunning)) + { + Signal(pid, 9); + } + } + + private static bool IsRunning(int pid) + { + try + { + using var process = Process.GetProcessById(pid); + return !process.HasExited; + } + catch (Exception ex) when (ex is ArgumentException or InvalidOperationException or System.ComponentModel.Win32Exception) + { + return false; + } + } + + private static void Signal(int pid, int signal) + { + try + { + _ = NativeKill(pid, signal); + } + catch (Exception ex) when (ex is DllNotFoundException or EntryPointNotFoundException) + { + // no libc to signal through + } + } + + /// Kills the process and everything it started; one already gone is fine. + private static void KillTree(Process process) + { + try + { + process.Kill(entireProcessTree: true); + } + catch (Exception ex) when (ex is InvalidOperationException or System.ComponentModel.Win32Exception or NotSupportedException) + { + // already gone + } + } + + /// Asks the process to stop with SIGTERM, so it can shut down cleanly; one already gone is fine. + private static void Terminate(Process process) + { + try + { + _ = NativeKill(process.Id, 15); + } + catch (Exception ex) when (ex is InvalidOperationException or DllNotFoundException or EntryPointNotFoundException) + { + KillTree(process); + } + } + + [DllImport("libc", EntryPoint = "kill", SetLastError = true)] + private static extern int NativeKill(int pid, int signal); + + private static string Seconds(TimeSpan value) => Math.Round(value.TotalSeconds).ToString(CultureInfo.InvariantCulture); + + private static string Ms(TimeSpan value) => ((long)value.TotalMilliseconds).ToString(CultureInfo.InvariantCulture); + + /// The preflight probe, the spawn, and the readiness wait behind . + private async Task LaunchAsync(CancellationToken cancellationToken) + { + // Without a URL to probe, the process exiting 0 is the readiness event. + var readyUrl = _readiness.ReadyUrl; + var startupTimeout = _command.StartupTimeout ?? DefaultStartupTimeout; + var clock = Stopwatch.StartNew(); + if (readyUrl is not null) + { + var reuse = _command.ReuseExisting == true && !_hooks.Ci; + if (_command.ReuseExisting == true && !reuse) + { + _hooks.Notice?.Invoke(_label + ": reuseExisting is ignored in CI, starting the command"); + } + + if (await AnswersAsync(readyUrl, Min(ReadyProbeTimeout, startupTimeout), cancellationToken).ConfigureAwait(false)) + { + if (reuse) + { + _reusedExisting = true; + _hooks.Notice?.Invoke(_label + ": reusing the process already serving " + Shown(readyUrl)); + return; + } + + throw new EngineException( + "APP_ALREADY_RUNNING", + Shown(readyUrl) + " already answered before " + _label + " started; " + + (_hooks.Ci ? "stop that process (reuseExisting is ignored in CI)" : "stop that process or set reuseExisting: true")); + } + } + + var (env, inherited) = ChildEnv(_command, _hooks.Env ?? ProcessEnvironment(), OperatingSystem.IsWindows()); + _redact = Redactor.For(inherited.Concat((_command.Env ?? new Dictionary()).Select(entry => (entry.Key, entry.Value))) + .Where(entry => entry.Value.Length >= RedactedEnvMinLength)); + + var log = OpenLog(); + long? logOffset = log?.Length; + Process process; + try + { + process = Spawn(env); + } + catch + { + log?.Dispose(); + throw; + } + + _process = process; + _spawned = true; + Live[process] = 0; + _job = ProcessJob.Create(process); + try + { + _pump = new OutputPump(process, log); + await WaitForReadinessAsync(process, readyUrl, startupTimeout, clock, logOffset, cancellationToken).ConfigureAwait(false); + } + catch + { + // A failed or cancelled start takes the process down with it, whatever ended the wait. + var owned = _pump is not null; + await StopAsync().ConfigureAwait(false); + if (!owned) + { + log?.Dispose(); + } + + throw; + } + } + + private async Task WaitForReadinessAsync(Process process, string? readyUrl, TimeSpan startupTimeout, Stopwatch clock, long? logOffset, CancellationToken cancellationToken) + { + var half = startupTimeout / 2; + var noticed = half < ProgressNoticeMin; + var pollMs = ReadyPollMin; + var exited = process.WaitForExitAsync(CancellationToken.None); + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + if (exited.IsCompleted) + { + var code = process.ExitCode; + if (readyUrl is null && code == 0) + { + // What the process printed before it exited is in the log by the time the start returns. + // The process is gone, and its log is closed with it. + await StopAsync().ConfigureAwait(false); + return; + } + + await StopAsync().ConfigureAwait(false); + throw Unreachable(_label + " exited with code " + code.ToString(CultureInfo.InvariantCulture) + (readyUrl is null ? " instead of 0" : " before becoming ready"), logOffset); + } + + var remaining = startupTimeout - clock.Elapsed; + if (remaining <= TimeSpan.Zero) + { + await StopAsync().ConfigureAwait(false); + throw Unreachable( + readyUrl is null + ? _label + " did not exit within " + Ms(startupTimeout) + " ms" + : _label + " was not reachable at " + Shown(readyUrl) + " within " + Ms(startupTimeout) + " ms", + logOffset); + } + + if (!noticed && remaining <= half) + { + noticed = true; + var waitingFor = readyUrl is null ? "waiting for it to exit" : "waiting for " + Shown(readyUrl); + var logNote = _command.Log is null ? "" : "; log: " + LogDisplayPath(); + _hooks.Notice?.Invoke(_label + " still starting after " + Seconds(startupTimeout - remaining) + "s: " + waitingFor + logNote); + } + + // Until the notice is out, neither a probe nor a sleep runs past the half mark, so it lands on time instead of after a 2 s probe. + var untilHalf = noticed ? TimeSpan.MaxValue : Max(ReadyPollMin, remaining - half); + if (readyUrl is not null) + { + // A probe that is still waiting when the process exits ends there: connecting to a closed port can take seconds. + using var probeEnds = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + var probe = AnswersAsync(readyUrl, Min(Min(ReadyProbeTimeout, remaining), untilHalf), probeEnds.Token); + var first = await Task.WhenAny(probe, exited).ConfigureAwait(false); + if (first == exited) + { + await probeEnds.CancelAsync().ConfigureAwait(false); + await probe.ContinueWith(_ => { }, TaskScheduler.Default).ConfigureAwait(false); + continue; + } + + if (await probe.ConfigureAwait(false)) + { + return; + } + } + + // The exit wakes the wait early so an exit is seen at once, not on the next poll. + var pause = Min(Min(pollMs, untilHalf), Max(TimeSpan.Zero, startupTimeout - clock.Elapsed)); + await Task.WhenAny(Task.Delay(pause, cancellationToken), exited).ConfigureAwait(false); + pollMs = Min(pollMs * 2, ReadyPollMax); + } + } + + /// A URL as a message shows it: without userinfo, query, or fragment, which may hold a token. + private static string Shown(string url) => + Uri.TryCreate(url, UriKind.Absolute, out var parsed) + ? parsed.GetComponents(UriComponents.SchemeAndServer | UriComponents.Path, UriFormat.UriEscaped) + : "the readiness URL"; + + private static TimeSpan Min(TimeSpan a, TimeSpan b) => a < b ? a : b; + + private static TimeSpan Max(TimeSpan a, TimeSpan b) => a > b ? a : b; + + private Process Spawn(Dictionary env) + { + string workingDirectory; + try + { + workingDirectory = Path.GetFullPath(string.IsNullOrWhiteSpace(_command.Cwd) ? "." : _command.Cwd, _projectRoot); + } + catch (ArgumentException ex) + { + throw new EngineException("APP_UNREACHABLE", _label + " failed to start: its cwd is not a path: " + ex.Message, ex); + } + + var info = new ProcessStartInfo(_command.Executable) + { + WorkingDirectory = workingDirectory, + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardInput = true, + RedirectStandardOutput = true, + RedirectStandardError = true, + StandardOutputEncoding = Encoding.UTF8, + StandardErrorEncoding = Encoding.UTF8, + }; + foreach (var arg in _command.Args ?? []) + { + info.ArgumentList.Add(arg); + } + + info.Environment.Clear(); + foreach (var (name, value) in env) + { + info.Environment[name] = value; + } + + Process? process = null; + try + { + process = Process.Start(info) ?? throw new EngineException("APP_UNREACHABLE", _label + " failed to start: no process was created"); + // Nothing is written to the process: closing its input tells a program that reads it that there is none. + process.StandardInput.Close(); + return process; + } + catch (Exception ex) when (ex is System.ComponentModel.Win32Exception or InvalidOperationException or IOException or UnauthorizedAccessException) + { + if (process is not null) + { + // It started but could not be set up: it is not tracked yet, so it ends here. + KillTree(process); + process.Dispose(); + } + + throw new EngineException("APP_UNREACHABLE", _label + " failed to start: " + ex.Message, ex); + } + } + + /// + /// The startup failure with the log's tail under it, so the report alone says what the process was doing when it stalled or died. The + /// message lands in the report, so command.env and inheritEnv values are redacted from the quote with the same redactor and + /// markers as the traces, the way the log itself never is. Without a log there is nothing to quote, and the message says how to get one. + /// + private EngineException Unreachable(string message, long? logOffset) + { + if (logOffset is not { } offset) + { + return new EngineException("APP_UNREACHABLE", message + "\nset log on this command to keep its output"); + } + + var lines = OutputSince(offset).Select(line => _redact.Redact(line)).ToList(); + var detail = lines.Count == 0 + ? "no output in " + LogDisplayPath() + : "output in " + LogDisplayPath() + " since " + _label + " started:\n" + string.Join("\n", lines.Select(line => " " + line)); + return new EngineException("APP_UNREACHABLE", message + "\n" + detail); + } + + /// + /// The last lines appended to the log past , terminal controls stripped, bounded in bytes before lines so a chatty + /// process cannot turn the message into the whole log. + /// + private List OutputSince(long offset) + { + try + { + using var stream = new FileStream(LogPath(), FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete); + var end = stream.Length; + var start = Math.Max(offset, end - OutputTailBytes); + if (end <= start) + { + return []; + } + + // One byte before a cut tells whether it landed mid-line: a fragment says nothing and goes, a line that ended right at the cut stays. + var readFrom = start > offset ? start - 1 : start; + stream.Seek(readFrom, SeekOrigin.Begin); + var buffer = new byte[end - readFrom]; + stream.ReadExactly(buffer); + var text = Encoding.UTF8.GetString(buffer); + var cutMidLine = start > offset && !(text.StartsWith('\r') || text.StartsWith('\n')); + var lines = LineBreak().Split(ControlSequences().Replace(text, "")) + .Select(line => line.TrimEnd()) + .Where(line => line.Length > 0) + .ToList(); + if (cutMidLine && lines.Count > 0) + { + lines.RemoveAt(0); + } + + return lines.Count > OutputTailLines ? lines[^OutputTailLines..] : lines; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + return []; + } + } + + private string LogPath() => Path.GetFullPath(_command.Log ?? "", _projectRoot); + + /// The log as the config names it, relative to the project root, for messages. + private string LogDisplayPath() => Path.GetRelativePath(_projectRoot, LogPath()); + + /// + /// Opens command.log for appending, creating its directory, and returns the stream the process writes stdout and stderr to. No + /// log means the output is discarded. The same file across services and runs accumulates, which is what a developer reading a + /// failed boot wants. + /// + private FileStream? OpenLog() + { + if (_command.Log is null) + { + return null; + } + + var path = LogPath(); + try + { + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + return new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite | FileShare.Delete); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + throw new EngineException("APP_UNREACHABLE", _label + " could not open its log file " + path + ": " + ex.Message, ex); + } + } + + [GeneratedRegex(@"\r\n|\n|\r", RegexOptions.CultureInvariant)] + private static partial Regex LineBreak(); + + [GeneratedRegex(@"\u001B(?:\[[0-?]*[ -/]*[@-~]|\][^\u0007\u001B]*(?:\u0007|\u001B\\)|[@-Z\\-_])", RegexOptions.CultureInvariant)] + private static partial Regex ControlSequences(); + + /// + /// Reads the process's stdout and stderr together, so neither pipe fills and blocks it, and appends both to the log (or drops them + /// when there is none). The stream is closed once the process is gone. + /// + private sealed class OutputPump : IDisposable + { + private FileStream? _log; + private readonly Lock _gate = new(); + private readonly Task _out; + private readonly Task _err; + private bool _closed; + + public OutputPump(Process process, FileStream? log) + { + _log = log; + _out = PumpAsync(process.StandardOutput.BaseStream); + _err = PumpAsync(process.StandardError.BaseStream); + } + + /// Waits for both pipes to end, up to : a descendant that outlives the process may hold them open. + public async Task DrainAsync(TimeSpan timeout) => + await Task.WhenAny(Task.WhenAll(_out, _err), Task.Delay(timeout)).ConfigureAwait(false); + + public void Dispose() + { + lock (_gate) + { + _closed = true; + _log?.Dispose(); + } + } + + private async Task PumpAsync(Stream source) + { + var buffer = new byte[8192]; + try + { + int read; + while ((read = await source.ReadAsync(buffer).ConfigureAwait(false)) > 0) + { + lock (_gate) + { + if (_closed) + { + continue; + } + + if (_log is not null) + { + try + { + _log.Write(buffer, 0, read); + _log.Flush(); + } + catch (Exception ex) when (ex is IOException or ObjectDisposedException or UnauthorizedAccessException) + { + // The log is gone (a full disk, a deleted file): the output is dropped, and the pipes still drain so the app never blocks on them. + _log = null; + } + } + } + } + } + catch (Exception ex) when (ex is IOException or ObjectDisposedException or InvalidOperationException) + { + // the pipe closed under the read + } + } + } +} + +/// +/// The Windows job object one started process runs in (a .NET-only guard: upstream signals a process group). It ends everything the process +/// started, whether or not the process is still there, and it ends it too when this process is killed outright, as a test host is once its +/// run ends and no ProcessExit handler runs. Elsewhere returns null. +/// +internal sealed class ProcessJob : IDisposable +{ + private const int JobObjectExtendedLimitInformation = 9; + private const uint KillOnJobClose = 0x2000; + + private IntPtr _job; + + private ProcessJob(IntPtr job) => _job = job; + + public static ProcessJob? Create(Process process) + { + if (!OperatingSystem.IsWindows()) + { + return null; + } + + var job = CreateJobObjectW(IntPtr.Zero, null); + if (job == IntPtr.Zero) + { + return null; + } + + // JOBOBJECT_EXTENDED_LIMIT_INFORMATION: LimitFlags is the fourth field of the basic limits, after two 8-byte times. + var size = IntPtr.Size == 8 ? 144 : 112; + var buffer = Marshal.AllocHGlobal(size); + try + { + for (var offset = 0; offset < size; offset += 4) + { + Marshal.WriteInt32(buffer, offset, 0); + } + + Marshal.WriteInt32(buffer, 16, (int)KillOnJobClose); + if (!SetInformationJobObject(job, JobObjectExtendedLimitInformation, buffer, (uint)size) || !AssignProcessToJobObject(job, process.Handle)) + { + CloseHandle(job); + return null; + } + + return new ProcessJob(job); + } + catch (Exception ex) when (ex is InvalidOperationException or System.ComponentModel.Win32Exception) + { + // the process is already gone + CloseHandle(job); + return null; + } + finally + { + Marshal.FreeHGlobal(buffer); + } + } + + /// Ends every process in the job. + public void Terminate() + { + if (_job != IntPtr.Zero) + { + _ = TerminateJobObject(_job, 1); + } + } + + public void Dispose() + { + var job = Interlocked.Exchange(ref _job, IntPtr.Zero); + if (job != IntPtr.Zero) + { + CloseHandle(job); + } + } + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode)] + private static extern IntPtr CreateJobObjectW(IntPtr attributes, string? name); + + [DllImport("kernel32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool SetInformationJobObject(IntPtr job, int infoClass, IntPtr info, uint size); + + [DllImport("kernel32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool AssignProcessToJobObject(IntPtr job, IntPtr process); + + [DllImport("kernel32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool TerminateJobObject(IntPtr job, uint exitCode); + + [DllImport("kernel32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool CloseHandle(IntPtr handle); +} diff --git a/src/E2E/Run/Provision.cs b/src/E2E/Run/Provision.cs new file mode 100644 index 00000000..84e38c00 --- /dev/null +++ b/src/E2E/Run/Provision.cs @@ -0,0 +1,73 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using E2E.Internal; + +namespace E2E; + +/// +/// What a run does before the first engine session (upstream's run/provision.ts): start the app command the target declares. The port has +/// no engine prepare hooks to run beside it, and one target, so there is nothing to deduplicate across targets. +/// +internal static class Provision +{ + /// + /// Starts the app command the target declares, when it declares one. Nothing more is acquired once is + /// cancelled, which throws; what was started by a failed start is stopped before the failure is thrown. + /// + public static async Task StartDeclaredProcessesAsync( + string targetName, + ResolvedApp app, + string projectRoot, + Func hooks, + DebugTrace debug, + CancellationToken cancellationToken) + { + if (app.Command is null) + { + return NoProcesses.Instance; + } + + var label = "target \"" + targetName + "\" command"; + var process = new ManagedProcess(label, app.Command, projectRoot, new Readiness(app.ReadyUrl), hooks()); + try + { + await debug.Time("app.start(" + label + ")", () => process.StartAsync(cancellationToken)).ConfigureAwait(false); + } + catch + { + // A start that failed or was cancelled stops its own process; this only makes certain of it. + await process.StopAsync().ConfigureAwait(false); + throw; + } + + return new StartedProcess(process); + } + + private sealed class NoProcesses : IAppProcesses + { + public static NoProcesses Instance { get; } = new(); + + public Task StopAsync(Action onFailure) => Task.CompletedTask; + + public bool Running => true; + } + + private sealed class StartedProcess(ManagedProcess process) : IAppProcesses + { + public bool Running => process.Running; + + public async Task StopAsync(Action onFailure) + { + try + { + await process.StopAsync().ConfigureAwait(false); + } + catch (Exception ex) + { + onFailure(ex); + } + } + } +} diff --git a/src/E2E/Run/RunApps.cs b/src/E2E/Run/RunApps.cs new file mode 100644 index 00000000..4f184877 --- /dev/null +++ b/src/E2E/Run/RunApps.cs @@ -0,0 +1,67 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using System.Collections.Concurrent; +using E2E.Internal; + +namespace E2E; + +/// +/// The app commands one test process has started (a .NET-only holder: a .NET test framework has no run-level start and finish hooks, where +/// upstream's runner owns them). The first test, e2e explore, or MCP session that needs a target's app.command starts it, and every +/// later one finds it running; nothing stops it before the process ends, where takes it +/// down. A start that failed is not remembered, so the next caller tries again. +/// +internal static class RunApps +{ + private static readonly ConcurrentDictionary Gates = new(StringComparer.Ordinal); + private static readonly ConcurrentDictionary Started = new(StringComparer.Ordinal); + + /// + /// The target as the run sees it: its app command started, if it declares one, and its URL on the port the app is served on. A target that + /// declares no command is returned as it is. + /// + public static async Task StartAsync(E2EConfig config, TargetConfig target, Action? notice, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(config); + ArgumentNullException.ThrowIfNull(target); + if (target.App.Command is null) + { + return target; + } + + var key = (config.ConfigPath ?? config.ProjectRoot) + "\0" + target.Name; + var gate = Gates.GetOrAdd(key, _ => new SemaphoreSlim(1, 1)); + await gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + if (Started.TryGetValue(key, out var running)) + { + if (running.Processes.Running) + { + return running.Target; + } + + // The app ended since (it crashed, or was stopped): start it again, on a new port when it asked for a free one. + Started.TryRemove(key, out _); + await running.Processes.StopAsync(_ => { }).ConfigureAwait(false); + } + + var app = AppPorts.AllocateTargetPorts(target.Name, target.App, config.ProjectRoot); + var hooks = new ManagedProcessHooks + { + Ci = config.Ci, + Notice = notice ?? (message => Console.Error.WriteLine("e2e: " + message)), + }; + var processes = await Provision.StartDeclaredProcessesAsync(target.Name, app, config.ProjectRoot, () => hooks, new DebugTrace(false), cancellationToken).ConfigureAwait(false); + var resolved = target.WithApp(target.App.WithUrl(app.Base?.Href)); + Started[key] = (resolved, processes); + return resolved; + } + finally + { + gate.Release(); + } + } +} diff --git a/tests/E2E.Tests.App/E2E.Tests.App.csproj b/tests/E2E.Tests.App/E2E.Tests.App.csproj new file mode 100644 index 00000000..b2a54c08 --- /dev/null +++ b/tests/E2E.Tests.App/E2E.Tests.App.csproj @@ -0,0 +1,8 @@ + + + Exe + false + E2E.Tests.App + false + + diff --git a/tests/E2E.Tests.App/Program.cs b/tests/E2E.Tests.App/Program.cs new file mode 100644 index 00000000..2c6c65c7 --- /dev/null +++ b/tests/E2E.Tests.App/Program.cs @@ -0,0 +1,119 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using System.Net; +using System.Net.Sockets; +using System.Runtime.InteropServices; +using System.Text; + +// The program the app-command tests start, standing in for the `node -e` scripts of upstream's managed-process tests. +// serve [ignore-sigterm] answers every request with 200 on 127.0.0.1 until it is stopped +// serve-for serves for that long, then exits +// spawn starts `serve ` as a child, then exits at once +// idle stays up and answers nothing +// exit exits with the code +// chatter prints a line to stdout and one to stderr, then exits with the code +// print ... prints each line to stdout, then exits with the code or stays up +// print-template