Skip to content

Epic: AI-generated E2E test coverage for pull requests #4189

Description

@JSONbored

Context

Research comparing CodeRabbit's AI test-generation feature against gittensory's own review stack found a clear, buildable gap:

  • CodeRabbit's version is narrow: explicit-ask only, unit-tests-only (no E2E/live-browser), gated behind their Pro+ tier, and it never executes what it generates — it just pushes a commit/PR and reactively parses the host repo's own CI logs on failure.
  • gittensory already has essentially every architectural piece needed to build a genuinely differentiated E2E version: the fix-handoff feature, the converged-feature config system (rag/reputation/unifiedComment/safety/grounding), the manifest_missing_tests signal, the @gittensory <verb> PR-comment command surface, openRepoDocPullRequest's existing commit/PR-writing pattern (src/github/repo-doc-pr.ts), and Playwright as an already-present dependency.
  • The one hard constraint on gittensory itself executing generated tests: only self-host (real Node) can do it at all, never the Cloudflare Worker. This constraint is sidestepped for v1 — see below.

Goal, refined: maintainers want a click-to-generate button — @gittensory generate-tests, maintainer-only — that produces real Playwright E2E tests and pushes them as a commit directly onto the PR branch, before merge, so the PR's own already-required CI validates them as part of the normal gate. Fully config-driven: a global kill-switch, a per-repo override, and repo/path-scoped instructions for what "good coverage" means (CodeRabbit-style path_instructions).

The boundary, reconciled: #1972 (closed, completed 2026-07-06) shipped a narrower mechanism specifically to protect contributor/miner scoring integrity — a miner's own harness writing with the miner's own credentials, so a contributor's score always reflects their own work. This epic is a deliberate, narrow exception to that boundary, safe because: (1) only a genuine maintainer/owner can trigger it — never the PR's own author, never a collaborator-tier actor, never a confirmed_miner (see #4195's authorization tightening — this is the first command in the whole @gittensory catalog scoped to ["maintainer"] alone, not ["maintainer", "collaborator", ...]); (2) the resulting commit is clearly attributed as maintainer-invoked/machine-generated (#4197); (3) it must never inflate the contributor's own scored contribution (#4201, a hard requirement, not a nice-to-have).

Security, reconciled: v1 does not need gittensory to execute anything itself. CodeRabbit's own answer to "how do you safely run untrusted generated test code" is that they don't — they delegate execution to the repo's own CI, which is infrastructure the owner already trusts. gittensory's gate already requires CI green before merge, so generate+commit alone (#4197) already satisfies "validated before it merges." Actual gittensory-side pre-commit execution (#4198) is real, but genuinely higher-risk (needs an ephemeral, network-egress-restricted, single-use sandbox — not the existing long-lived shared browserless sidecar, which is the wrong isolation model for running untrusted code rather than just navigating a URL) — it's deprioritized to an optional, later phase, not a v1 blocker.

Sub-issues

Expected outcome

A repo maintainer — and only a maintainer, never the PR's own author acting as a contributor/collaborator, never an autonomous miner — can comment @gittensory generate-tests on any PR and get real, AI-generated Playwright E2E tests pushed as a clearly-attributed commit directly onto that PR's branch before merge, respecting repo/path-scoped coverage instructions set in .gittensory.yml, validated by the PR's own existing CI gate, and never counted toward the contributor's own scored contribution.

Effort

Epic; each sub-issue is independently shippable. The v1 critical path is #4190#4191#4192#4195#4197 (+ #4200 for config rules, #4201 as a hard scoring-safety gate before #4197 ships) — #4193/#4194 support it as a confirmation layer, #4196 adds auto-triggering on top, and #4198/#4199 are post-v1.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions