Port test filters: the filter seam, change detection, and report fields (upstream #1206, #1345)

COMPATIBILITY.md

@@ -199,7 +199,7 @@ The packages in tester-army/e2e at the target commit, and what the port has:
 | `@e2e-dev/limrun` | Ported | `E2E.Limrun`: see [Limrun](#limrun) |
 | `@e2e-dev/kernel` | Ported | `E2E.Kernel`: see [Kernel](#kernel) |
 | `@e2e-dev/testmu` | Not ported | A mobile device provider for TestMu AI, through agent-device. `E2E.TestMu.Mobile` serves the same devices with Appium: see [TestMu mobile](#testmu-mobile) |
-| `@e2e-dev/decision` | Not ported | A decision-model executor: a model that picks one action from a grammar (taps, points, vision). The port has the executor seam (`IStepExecutor`, see [Agents](#agents)) but no such executor |
+| `@e2e-dev/decision` | Not ported | `decisionFilter` (see [Test filters](#test-filters), where the filter seam it plugs into is ported) and a decision-model executor: a model that picks one action from a grammar (taps, points, vision). The port has the executor seam (`IStepExecutor`, see [Agents](#agents)) but no such executor |
 | `@e2e-dev/smol` | Not ported | A browser provider that runs each browser and the app in a branchable smol machines microVM. The port has the provider seam (`IBrowserProvider`, see [Web engine](#web-engine)) but no such provider |
 | `@e2e-dev/lightpanda` | Not ported | A browser provider for the Lightpanda headless browser. Upstream added it on 2026-10-15 (`b41afa2`), after the earlier target. The engine drives Chromium only |
 | `@e2e-dev/github` | Not ported | A reporter that posts the run as a pull request comment. The port has no reporters |
@@ -571,6 +571,28 @@ Upstream's `--trace` writes `<output>/results/<result>/trace.md`, a markdown pag
 - .NET-only names: `E2ESession.Evidence` (what the session captured; upstream reads it from the report), `E2ESession.CaptureFailureEvidenceAsync` and `E2EFixture.CaptureFailureEvidenceAsync` (upstream's local `captureEvidence` closure; the fixture's is for the adapters), `LooksBeforeTeardownAttribute` (NUnit and xUnit; internal), NUnit `E2ETest.CaptureEvidenceOf` (internal; reaches the protected fixture method), xUnit `E2ETest.LookAtFailure` and its `Running` map (the attribute is handed the test, not the instance), `FailureEvidence.ScreenFile`, `ScreenshotFile`, `ScreenArtifactId`, and `ScreenshotArtifactId` (upstream registers the files with its artifact sink), `FailureEvidenceOptions.Observe` and `Capture` (upstream hands the open session), `AttemptReport` (and its `ResultId` and `ResultSegmentOf`, which give the result id and directory name; a .NET test has no one source file, so the test class stands for upstream's `file`: the fixture passes its full name as `E2ESessionOptions.TestFile`, it joins the title in the id hash as upstream's `testId` does, and its simple name leads the segment; a session started with no `TestFile`, such as `explore` or a direct `E2ESession.StartAsync`, hashes the title alone), `AttemptReport.TestId` and `EncodeTitle` (internal; upstream's `testId` and `encodeTitle` in `internal/ids.ts`, with a title path of one title; the title is percent-encoded so a `::` in it cannot pass for the separator; the hash input carries a NUL mark so a title-only session cannot share an id with a test id), `E2ESessionOptions.TestFile` (public, optional; the one new member of the public API, set by `E2EFixture`, so the NUnit and xUnit adapters change nothing), `TraceOutput`, `WriteTracePagesOptions`, `ReportResult.Directory` (an attempt writes under `ArtifactPaths.ResultSegment` of the result's id and title, so two titles that slug alike keep apart; the segment leads with the test class's name when the fixture started the session, and the page goes in that directory), `StepRecorder.AddModelCall` (upstream's `usage.record` in `StepAccounting`), `FileStepCache.CacheDirectory` (internal), and `CacheInfo.Entry` (upstream's `entry` of `StepCacheInfo`), `OptionalTextConverter` (upstream's `readRecordedModel` tolerates a malformed `recordedBy`; the port's is a string, which a converter reads leniently), `PendingAct.Decided` (upstream's `decided` staged entry), and `CacheInfo.WithRecordedBy` (upstream spreads `recordedBy` into the info).
 - Tests: `Traces/` holds `unit/traces.test.ts` (all 11 `it`: `WriteTracePagesTests`, `TracePageNamesTests`, `TheStepsOnATracePageTests`, `TheAppLogOnATracePageTests`, over hand-built records) and `integration/traces.test.ts` (`TracesTests`, all 6 `it`, in a session over a fake engine, with no runner). In `TracesTests`, "keeps each step's events, files what the app logged ..." is named for the part it checks, `Keeps_each_steps_events_and_files_what_the_app_logged_under_the_step_it_happened_in_on_the_page` (partial: it reads no run event, which the port has no run to emit); the two "names the step ... on the page and in the summary" tests are partial (no summary reporter); "keeps the traces the mode asks for" runs sessions with the attempt number in place of a runner's retries. `ReportFailureText/` holds `report-failure-text.test.ts`: `detailLines`, `attemptsLine`, `failedStepOf`, `failureSource`, `evidenceOf`, and `renderTracePage` (15 `it`; the footer of "tells the whole story" has no `report.json` pointer). Not ported from that file: "lastTurnLines and screenLines" (the run page), "names the skip a failed attempt took ..." (no skip reason), and "a serial member on its page" (no serial groups). `.NET-only`: `TracePageRedactionTests`. `FailureEvidence/` holds `failure-evidence.test.ts` (`FailureEvidenceTests`, 6 of its 7 `it` in `E2E.Tests`, in sessions over a fake engine; the `it.each` is one theory). "looks at the screen before teardown runs ..." is `E2E.NUnit.Tests/FailureEvidence/FailureEvidenceTests` (through NUnit, with a `[TearDown]` in place of `afterEach`), so all 7 `it` of the file are ported, with a .NET-only soft-failure variant beside it; the xUnit adapter's look has no test project. "keeps the evidence of two tests whose ids sanitize to the same directory name apart, each report digest matching its file" is `Keeps_the_evidence_of_two_tests_whose_titles_slug_alike_apart` (partial: the port's report artifacts carry no sha256 to compare). `Keeps_the_evidence_of_two_tests_with_one_title_in_two_classes_apart` and `Gives_a_title_with_a_separator_in_it_another_id_than_the_same_text_split_into_file_and_title` are .NET-only (upstream's runner rejects a duplicate test id; the port keys on the class instead of the file). `Errors/ErrorDetailsTests` holds the `error details` tests of `unit/errors.test.ts`: two of three, the redaction one partial (the port keeps no stack text), and not "keeps the details of an error classified from another module copy". The details of the failure are asserted in `ScriptedMatchers/`, `ScreenshotAssertions/`, and `AgentExecutorContext/` where upstream asserts them, and the failure capture beside an assertion's own screenshot in `AssertEvidenceUnderACustomExecutorTests`. `open-host-trace.test.ts` is not ported with the hosted session (#403).
 
+## Test filters
+
+Upstream #1206 (`da13624`, `feat(runner): run only the tests a pull request can affect`) adds `filters` to the config, `--changed-since <ref>` on `e2e run` and `e2e list`, and the "left out" part of the report. #1345 (`47127f7`) adds `context.configFile`. The port has no runner, no `collect` or `select`, no `e2e run` or `e2e list`, and no reporters other than the JSON report, so it takes the filter seam, the change detection, and the report fields, and leaves the wiring to a host. A test framework selects tests itself (`dotnet test --filter`), and the port keeps no list of every test before the first one runs, which a filter needs, so nothing in `E2ETest` calls a filter.
+
+Ported:
+
+- `ITestFilter` (upstream's `TestFilter`: `Name`, and `FilterAsync` for `filter`), `TestFilterCandidate`, `TestFilterChanges`, `TestFilterContext` (with `ConfigFile`, #1345), and `TestFilterResult` (`Exclude`, `Note`) are public (`Run/TestFilter.cs`, upstream's `types.ts`). A host calls the internal `TestFilters.ApplyTestFilters` (`Run/TestFilters.cs`, upstream's `run/test-filters.ts`) with its candidates: filters run in order on the tests the ones before kept, a filter that throws, returns an invalid result, or runs past 5 minutes (`FilterTimeout`) leaves nothing out with a warning, and reasons, notes, and log lines are redacted and cut to one line of 1024 code points. `context.Redact` also masks every run of 8 or more characters of a secret. `TestFilters.RepoConfigFile` is upstream's `repoConfigFile`, `TestFilters.FilterRecords` and `FilterSummary` are `filterRecords` and `filterSummary`, and `LeftOut.LeftOutTests` is `leftOutTests` (`Report/LeftOut.cs`, `report/left-out.ts`).
+- `Changes.FindChangesAsync` (`Run/Changes.cs`, upstream's `findChanges`): every file that differs from the merge base of the ref and HEAD, staged, unstaged, deleted, and untracked files included, a rename as both paths, sorted by code point, with a 30 second budget. A ref git cannot measure from is `INVALID_CHANGED_SINCE` (a `ConfigurationException`). `Git.RunGitAsync`, `GitResult`, and `GitFailure` are upstream's `runGit`, `GitResult`, and `GitFailure`; `Paths.OnDiskPath` is `onDiskPath`; `Time.BudgetText` is `budgetText`; `ReportFormat.FilterDecision` is `filterDecision`.
+- The report: `run.changes` (`ReportChanges`) and `run.filters` (`ReportFilter`, `FilterOutcome`) on `E2E.Reports.ReportRun`, `SkipInfo.Filter` (a `filtered` skip names the filter), `ErrorPhase.Filter`, and the `filter` phase in the wire form. `schema/report-v1.schema.json` and both fixtures are upstream's at #1345.
+
+Differences:
+
+- An interrupt surfaces as an `OperationCanceledException`, where upstream rejects with `INTERRUPTED`. The context's `Signal` follows the interrupt only while the call runs. The work a timed-out or interrupted call leaves behind is not recorded (upstream's `leaveWorkBehind` keeps the CLI from waiting for it; the port has no CLI process to keep).
+- A result is typed, so the refusals of a result that is not `{ exclude, note? }` (an array, a string, an unknown key, a map, a numeric note, a value that cannot be shown as text, and a reason that is not a string) cannot happen. A null result, a null `Exclude`, an empty reason, and an id no collected test has are refused, with the port's wording for the first two.
+- The redactor reads a newline and a space alike in a run it masks, so it takes the space before a key's text into the marker in a diff line, where upstream keeps it. Nothing leaks.
+
+Not ported:
+
+- `filters` in `e2e.config.json` (a filter is code, and the JSON config holds none), `--changed-since`, `--no-filter`, `INVALID_CHANGED_SINCE` as a CLI exit 2, the `NO_FILTERS` warning, `filterSelection`, `filterCandidates`, `select` with `excluded` (`applyExclusions`), the `list` and `run` output of a left-out pair, the left-out lines of the list, markdown, and JUnit reporters, telemetry, the `filter` setup step of a run, `pull_request_target` guidance, and the CI recipe. They need the runner, `collect`, `select`, and the reporters.
+- `@e2e-dev/decision`'s `decisionFilter` and its tests, including #1345's `leaves nothing out, making no call, when the config the run loaded changed, whatever its name, and goes by name for any other`: the package is not ported (see [Upstream packages](#upstream-packages)).
+- Tests: `Changes/FindChangesWithChangedSinceTests` has all 10 `it` of `changes.test.ts` (the `INTERRUPTED` one is `Partial_rejects_once_the_run_is_interrupted`; the `--no-relative` one runs apart from Windows, as upstream's does). `Time/BudgetTextTests` is `budgetText`. `SchemaFixtures/ReportV1SchemaTests` has the filter `it` of `schema-fixtures.test.ts`. `TestFilters/ApplyTestFiltersTests`, `AFilterPastTheDeadlineTests`, and `FilterRecordsAndFilterSummaryTests` port `test-filters.test.ts` over candidates built by hand, since the port has no `collect` or `select`; the ones that differ are `Partial_`. Not ported from it: the two `filterSelection` `it`, the three `filterCandidates` `it`, the `refuses a result` cases above, and `leaves nothing out when a filter throws a value that cannot be shown as text`. `TestFilters/WhatAFilterIsToldTests` is #1345's `names the config file the run loaded` over `RepoConfigFile`, partial. Not ported: the rest of `integration/test-filters.test.ts`, `select.test.ts`'s cases for `excluded`, `config.test.ts`'s `filters` cases, `cli*.test.ts`, `report-list`, `report-markdown`, `report-junit`, `telemetry-events`, and `packages/decision`'s tests.
+
 ## Report document
 
 Upstream writes `report.json`, a `report-1` document of the whole run (`report/build.ts`), checked by `schema/report-v1.schema.json`. The port writes it too (#496), with the differences below.

docs/index.md

@@ -16,6 +16,7 @@ dotnet add package E2E.NUnit   # or E2E.XUnit.V3 for xUnit v3
 - [API](api/index.md): every public type in `E2E`, `E2E.NUnit`, and `E2E.XUnit.V3`.
 - [Rerun a test while you fix it](../README.md#rerun-a-test-while-you-fix-it): run one test with the replay cache on, so goals that passed replay without model calls.
 - [Compatibility](../COMPATIBILITY.md): how this port differs from the TypeScript [e2e](https://github.com/tester-army/e2e).
+- [Test filters](../COMPATIBILITY.md#test-filters): `ITestFilter`, `TestFilterContext`, and `TestFilterResult` leave out the tests a change cannot affect. The seam, the change detection (`INVALID_CHANGED_SINCE`), and the `run.changes` and `run.filters` report fields are ported; no `dotnet test` host calls a filter yet.
 
 ## Errors
 

schema/fixtures/report-v1.invalid.json

@@ -67,6 +67,16 @@
             ]
           }
         ]
+      },
+      {
+        "selected": false,
+        "status": "skipped",
+        "attempts": [],
+        "skip": {
+          "cause": "explicit",
+          "reason": "only a filtered test names the filter that left it out",
+          "filter": "areas"
+        }
       }
     ],
     "errors": [
@@ -92,6 +102,20 @@
       "interrupted": -1,
       "flaky": 0,
       "skipped": 0
-    }
+    },
+    "changes": {
+      "base": "main",
+      "files": 3,
+      "paths": ["src/app.ts"]
+    },
+    "filters": [
+      {
+        "name": "areas",
+        "version": "1",
+        "candidates": 2,
+        "excluded": 1,
+        "outcome": "answered"
+      }
+    ]
   }
 }

schema/fixtures/report-v1.valid.json

@@ -390,6 +390,34 @@
           "cause": "explicit",
           "reason": "the demo tenant has a single organization"
         }
+      },
+      {
+        "id": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
+        "testId": "tests/example.e2e.ts::imports%20a%20report",
+        "kind": "test",
+        "tags": [],
+        "declarationIndex": 3,
+        "titlePath": [
+          "imports a report"
+        ],
+        "file": "tests/example.e2e.ts",
+        "source": {
+          "file": "tests/example.e2e.ts",
+          "line": 16,
+          "column": 1
+        },
+        "targetId": "web",
+        "platform": "web",
+        "agent": "default",
+        "repeat": 0,
+        "selected": false,
+        "status": "skipped",
+        "attempts": [],
+        "skip": {
+          "cause": "filtered",
+          "reason": "the diff reaches none of its tags",
+          "filter": "areas"
+        }
       }
     ],
     "errors": [
@@ -408,7 +436,7 @@
       }
     ],
     "summary": {
-      "discovered": 2,
+      "discovered": 3,
       "selected": 2,
       "executed": 2,
       "passed": 1,
@@ -437,6 +465,32 @@
       "maxModelCallsInStep": 0,
       "maxActionStepsInStep": 0
     },
+    "changes": {
+      "base": "0123456789abcdef0123456789abcdef01234567",
+      "files": 3
+    },
+    "filters": [
+      {
+        "name": "areas",
+        "candidates": 2,
+        "excluded": 1,
+        "note": "src/todos/ changed"
+      },
+      {
+        "name": "jev",
+        "candidates": 1,
+        "excluded": 0,
+        "note": "did not finish within 5m, so it leaves nothing out",
+        "outcome": "timed-out"
+      },
+      {
+        "name": "owners",
+        "candidates": 1,
+        "excluded": 0,
+        "note": "failed, so it leaves nothing out: CODEOWNERS not found",
+        "outcome": "failed"
+      }
+    ],
     "carried": {
       "results": [
         {

schema/report-v1.schema.json

@@ -78,7 +78,7 @@
         "message": { "type": "string", "maxLength": 8192 },
         "retryable": { "type": "boolean" },
         "phase": {
-          "enum": ["config", "collection", "launch", "beforeAll", "beforeEach", "body", "afterEach", "afterAll", "cleanup", "report"]
+          "enum": ["config", "collection", "filter", "launch", "beforeAll", "beforeEach", "body", "afterEach", "afterAll", "cleanup", "report"]
         },
         "scopeId": { "type": "string", "minLength": 1, "maxLength": 1024 },
         "scope": {
@@ -428,8 +428,12 @@
           "enum": ["explicit", "filtered", "platform-unavailable", "capability-unavailable", "setup-failed", "serial-predecessor-failed", "hook-failed", "infrastructure-unavailable", "failure-limit", "time-limit"]
         },
         "reason": { "type": "string", "maxLength": 8192 },
-        "relatedId": { "type": "string", "minLength": 1 }
-      }
+        "relatedId": { "type": "string", "minLength": 1 },
+        "filter": { "description": "The name of the test filter that left the test out; only with cause filtered.", "type": "string", "minLength": 1, "maxLength": 256 }
+      },
+      "allOf": [
+        { "if": { "required": ["filter"] }, "then": { "properties": { "cause": { "const": "filtered" } } } }
+      ]
     },
     "serialMember": {
       "type": "object",
@@ -728,10 +732,39 @@
         "limits": { "$ref": "#/$defs/limits" },
         "usage": { "$ref": "#/$defs/usage" },
         "carried": { "$ref": "#/$defs/carried" },
+        "changes": { "$ref": "#/$defs/changes" },
+        "filters": { "type": "array", "items": { "$ref": "#/$defs/filter" } },
         "explore": { "$ref": "#/$defs/explore" },
         "extensions": { "$ref": "#/$defs/extensions" }
       }
     },
+    "filter": {
+      "description": "One test filter the run applied, in order. excluded counts the tests the run left out on its word, once serial groups took back their members.",
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["name", "candidates", "excluded"],
+      "properties": {
+        "name": { "type": "string", "minLength": 1, "maxLength": 256 },
+        "candidates": { "type": "integer", "minimum": 0 },
+        "excluded": { "type": "integer", "minimum": 0 },
+        "note": { "type": "string", "maxLength": 1024 },
+        "outcome": { "description": "Set when the filter left nothing out because the call did not answer: failed when it threw or answered with something other than { exclude, note? }, timed-out when it ran past 5 minutes and the run stopped waiting. Absent when it answered.", "enum": ["failed", "timed-out"] }
+      }
+    },
+    "changes": {
+      "description": "The change the run's test filters judged, by counts only, or why no filter ran. Present when filters are configured and the run passed --changed-since. base and files come together once git measured the change; skipped stands alone when no filter ran before git was asked.",
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "base": { "description": "The merge base the change is measured from.", "type": "string", "pattern": "^[a-f0-9]{40,64}$" },
+        "files": { "description": "How many files changed since base; 0 when there is no change.", "type": "integer", "minimum": 0 },
+        "skipped": { "description": "Why no filter ran, such as --last-failed.", "type": "string", "minLength": 1, "maxLength": 1024 }
+      },
+      "oneOf": [
+        { "required": ["base", "files"] },
+        { "required": ["skipped"], "properties": { "base": false, "files": false } }
+      ]
+    },
     "carried": {
       "description": "What earlier runs left unresolved that a --last-failed rerun did not run again: their results as they reported them, the serial groups those results' attempts live on, and the suite hook failures whose scope still holds one of them. The next --last-failed selects these tests too. Not part of the run's status, exit code, or summary.",
       "type": "object",