diff --git a/README.md b/README.md index 944bb20..c6f74f1 100644 --- a/README.md +++ b/README.md @@ -294,6 +294,8 @@ workflow even when no native skill creator or repository validator is installed. - `scripts/install.mjs` stages, validates, and commits an installation. - `scripts/remote-install.mjs` stages and transfers an SSH environment install. - `scripts/validate.mjs` checks inventory, frontmatter, links, and portability. +- `scripts/check-harness-policy.mjs` validates a business repository's mixed + pstack/mstack policy and detects duplicate verification maps. - `.codex-plugin/` packages the same skill tree for Codex. The installer stages every selected skill, applies its adapter, validates the @@ -320,6 +322,22 @@ uses the current harness's adapter instead of naming one vendor's commands. Run `node scripts/validate.mjs` to print the validated skill count. +For a business repository that mixes pstack and mstack, initialize its shared +project workflow from this checkout: + +```bash +node /scripts/check-harness-policy.mjs init --root --app web --check 'node --test' --pstack +``` + +Replace the app, check command, and pstack revision with the project's values. +Initialization exports `.harness/check.mjs` and its library for committed CI +checks, without requiring an unpublished npm version. It leaves the application +contract for `/create-verification-skill` to create or migrate and prove. See +[mixed-Harness adoption](./docs/guide/11-mixed-harness.md) for wrapper generation, +receipts, repository protection, and reviewed checker upgrades. After a release +includes this CLI, use its exact package version for initialization and run +recording. + To validate and print a user's model configuration, run: ```bash diff --git a/automations/benny/FOR_AGENTS.md b/automations/benny/FOR_AGENTS.md index 8f76061..d5d4823 100644 --- a/automations/benny/FOR_AGENTS.md +++ b/automations/benny/FOR_AGENTS.md @@ -50,7 +50,13 @@ i want two cursor automations that work together in one slack issue channel. - budgets: `` - optional bot token capability: `` -start from [`configuration.example.yaml`](./templates/configuration.example.yaml) and [`feature-map.example.md`](./skills/reproduce-and-fix-issues/references/feature-map.example.md). copy and fill them outside this pack, for example under `.cursor/benny/`. keep secret values in a secret manager or environment. +start from [`configuration.example.yaml`](./templates/configuration.example.yaml). +copy and fill the configuration outside this pack, for example under +`.cursor/benny/`. when the project has a shared contract, set the feature-map +path to `.harness/verify//features/README.md`. read its linked feature files +and app contract instead of creating a benny-only map. use +[`feature-map.example.md`](./skills/reproduce-and-fix-issues/references/feature-map.example.md) +for the required content. keep secret values in a secret manager or environment. ## for the agent diff --git a/automations/benny/README.md b/automations/benny/README.md index c1e767d..ca9701d 100644 --- a/automations/benny/README.md +++ b/automations/benny/README.md @@ -18,6 +18,6 @@ the files in this directory are dormant setup and automation sources. they do no } ``` -4. keep user-owned configuration outside the copied pack, for example in `.cursor/benny/`. adapt [`configuration.example.yaml`](./templates/configuration.example.yaml) and [`feature-map.example.md`](./skills/reproduce-and-fix-issues/references/feature-map.example.md). +4. keep user-owned configuration outside the copied pack, for example in `.cursor/benny/`. adapt [`configuration.example.yaml`](./templates/configuration.example.yaml). point `control.feature_map_path` at `.harness/verify//features/README.md` when the project has a shared verification contract. benny reads the linked feature files and the same contract as other Harnesses. use the [map example](./skills/reproduce-and-fix-issues/references/feature-map.example.md) for required content, not a second copy of project facts. 5. commit `.cursor/settings.json`, `.cursor/automations/benny/`, and any secret-free configuration before enabling either automation. 6. review each new automation draft or update existing automations in their editors. then send a harmless test report and verify every source-channel post stays in the original thread. diff --git a/automations/benny/skills/reproduce-and-fix-issues/SKILL.md b/automations/benny/skills/reproduce-and-fix-issues/SKILL.md index 950339e..68a9e6a 100644 --- a/automations/benny/skills/reproduce-and-fix-issues/SKILL.md +++ b/automations/benny/skills/reproduce-and-fix-issues/SKILL.md @@ -10,6 +10,12 @@ Wait for a trusted triage marker in the source thread. Reproduce the exact sympt Load the external Benny configuration supplied by the automation. If the config, required actions, control adapter, or completed feature map is missing, fail closed. +Read `AGENTS.md` and `.harness/workflow.md` when present. Use the shared project +workflow for writable worktree isolation, declared revisions, and evidence. +Resolve `.harness/policy.json` before selecting an app map. The automation's +configuration must point at that app's canonical feature index. Report a stale +or conflicting map instead of silently following it. + ## Hard safety rules - Freeze the source channel and root thread coordinates before doing any work. @@ -125,9 +131,16 @@ If no operations channel is configured, keep detailed status in the automation r ## 5. Load and check the control adapter -Read `references/control-adapter.md` and the completed map at `control.feature_map_path`, then invoke the skill named by `control.skill_name`. +Read `references/control-adapter.md` and the completed map at +`control.feature_map_path`, then invoke the skill named by `control.skill_name`. +For a canonical `features/README.md`, read the matching linked feature file and +its app's `contract.md`. Resolve a wrapper's `metadata.verification-contract` +from the repository root. Adapt the contract to the available driver without +creating a second map or changing the required proof. -Find the feature-map section that matches the reported user path. Read it before driving the app. If no section covers the feature, mark the run blocked instead of inventing a path or selector. +Find the mapped feature that matches the reported user path. Read it before +driving the app. If no entry covers the feature, mark the run blocked instead of +inventing a path or selector. Require all seven capabilities: @@ -286,7 +299,10 @@ Run focused tests, then smoke the blast radius around the changed behavior. Cove Only after before-and-after proof: - Review the final diff for unrelated changes and secrets. -- Run the repository's required checks. +- Run the repository's required checks. When `.harness/policy.json` exists, + follow `.harness/workflow.md` to record the final commit and canonical feature + evidence. Include the receipt and retained artifacts in the review handoff. + Passing commands alone do not prove the reported UI behavior. - Create small ordered commits when the repository workflow allows it. - Open a draft pull request. Never merge or deploy from this workflow. - Link the configured tracker issue using the tracker's supported pull request syntax. diff --git a/automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md b/automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md index 3399491..425decb 100644 --- a/automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md +++ b/automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md @@ -4,7 +4,20 @@ Benny does not know how to start or drive every app. The user must configure one Set its skill name in `control.skill_name`. -Set the completed user-facing feature map path in `control.feature_map_path`. Copy and fill [`feature-map.example.md`](./feature-map.example.md) outside `.cursor/automations/benny/` instead of editing the copied example. +Set `control.feature_map_path` to the app's canonical feature index, such as +`.harness/verify/web/features/README.md`, when the repository uses a shared +project contract. Load its linked feature files and adjacent `contract.md`. +The control skill may be a neutral `verify-` wrapper that resolves +`metadata.verification-contract` from the repository root. + +Keep driver-specific capability adaptation in the installed control skill. +Launch commands, fixtures, user paths, expected results, and cleanup belong to +the canonical contract. Do not copy them into Benny configuration. + +For a legacy project without a shared policy, an existing completed external +map can remain until migration. The [map example](./feature-map.example.md) +describes the required content. Keep any user-owned map outside the copied +pack, and replace it with the canonical index during mixed-Harness adoption. If the skill, feature map, or a required capability is absent, ambiguous, or incomplete, repro and fix work must fail closed. @@ -52,7 +65,8 @@ Do not set internal state, call hidden app methods, write directly to storage, o ### Drive mapped features and states -Read the relevant feature-map section before driving the app. +Read the relevant feature file or legacy map section before driving the app. +An index is a list of links, not the complete feature definition. The adapter must expose ways to: @@ -134,7 +148,8 @@ The adapter must: - Surface startup failures as failures. - Bound retries. - Keep secrets out of logs and artifacts. -- Keep captures outside the repository. +- Keep captures outside tracked source. A shared project's ignored artifact + location can hold files referenced by its run receipt. - Support a fresh or reset state between the two repro attempts. - Avoid production changes unless the user explicitly configured a safe test action. diff --git a/automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md b/automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md index 620dded..99a55c1 100644 --- a/automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md +++ b/automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md @@ -1,8 +1,17 @@ # Feature-map example +In a project with `.harness/policy.json`, use these fields to complete the +canonical feature files under `.harness/verify//features/`. Set Benny's +`control.feature_map_path` to their `README.md` index. Do not create a separate +Benny map. This single-file example also describes legacy maps used before +shared-project adoption. + Map every user-facing feature Benny may reproduce. Read the relevant section before driving the app. Keep this map at the user point of view. Discover internals and current code paths at runtime instead of freezing them here. -Copy this file outside `.cursor/automations/benny/`, for example to `.cursor/benny/feature-map.md`, and set `control.feature_map_path` to the copy. Pack refreshes must not overwrite it. +For a legacy project without a shared contract, copy this file outside +`.cursor/automations/benny/`, then set `control.feature_map_path` to the completed +copy. Pack refreshes must not overwrite it. During mixed-Harness migration, +move its reconciled facts to the canonical feature files and retire the copy. ## Per-feature template diff --git a/automations/benny/skills/reproduce-and-fix-issues/references/verify-existing-fix.md b/automations/benny/skills/reproduce-and-fix-issues/references/verify-existing-fix.md index e3cda3c..dc6a4f8 100644 --- a/automations/benny/skills/reproduce-and-fix-issues/references/verify-existing-fix.md +++ b/automations/benny/skills/reproduce-and-fix-issues/references/verify-existing-fix.md @@ -31,6 +31,12 @@ Use regular `github.com` pull request links. ## Measure the baseline +Load the shared project workflow and canonical contract when present. Resolve +the configured feature index to the matching feature file. Give baseline and +patched builds separate instance state as the contract requires, and identify +each tested commit in the evidence. If an older revision lacks the mapped path, +report that limitation rather than applying the current map without checking. + For an open pull request, use its base branch as the baseline. For a merged fix, use the revision immediately before the fix when that revision builds and represents the old behavior. diff --git a/automations/benny/skills/setup-benny/SKILL.md b/automations/benny/skills/setup-benny/SKILL.md index 7827cc0..5760c5d 100644 --- a/automations/benny/skills/setup-benny/SKILL.md +++ b/automations/benny/skills/setup-benny/SKILL.md @@ -78,16 +78,35 @@ Open these copied examples: Create user-owned copies outside `.cursor/automations/benny/`. These are configuration files, not pack files. Example locations: - Project config, such as `.cursor/benny/configuration.yaml` -- Project feature map, such as `.cursor/benny/feature-map.md` +- Canonical feature index, such as `.harness/verify/web/features/README.md` - Project routing map, such as `.cursor/benny/routing.md` - User config, such as `~/.config/benny/configuration.yaml` -- User feature map, such as `~/.config/benny/feature-map.md` +- Legacy user feature map, when the project has no shared contract -Fill one feature-map section for every user-facing feature the automation may reproduce. Keep it at the user point of view. Do not freeze implementation details or current code paths in the map. +When the project has `.harness/policy.json`, read `.harness/workflow.md` and +select its canonical app. Set `control.feature_map_path` to that app's +`features/README.md`. Read the linked feature files and the adjacent +`contract.md`. Reuse them instead of making a Benny-only copy. The configured +control skill supplies the available driver and follows that contract. + +When canonical verification does not exist, use `/create-verification-skill` to +create or migrate it as part of authorized verification setup. A legacy project +without a shared policy may keep an existing completed external map until it +migrates. Never keep both definitions active after adoption. + +Map every user-facing feature the automation may reproduce. Keep user paths and +expected observations in the canonical feature files. Discover implementation +paths from source when investigating a report. Do not edit the copied examples. Pack refreshes may update source-managed files after conflict review, but they must never touch the user-owned copies. -Prefer committed, secret-free files in the target repository when a fresh automation checkout must read them. Otherwise paraphrase the required values into the live prompt. Reference a repository file only after the built-in `/automate` skill confirms that the file is committed in the repository where the automation runs. +Require the shared project workflow, canonical contract, and feature files to +be committed before enabling an automation that uses them. Reference their +repository paths instead of paraphrasing project facts into the prompt. For +configuration unrelated to verification, prefer committed, secret-free files +when a fresh automation checkout must read them. Otherwise paraphrase the +required values. Reference a repository file only after the built-in +`/automate` skill confirms that it is committed in the automation repository. Use stable repository-relative paths for committed pack and configuration files. Never reference the plugin source directory or a plugin cache path from a live automation. @@ -150,7 +169,10 @@ If no routing map is configured, triage may classify a report but must not guess ## 6. Verify the control adapter -Read `../reproduce-and-fix-issues/references/control-adapter.md` and the user's completed feature map. +Read `../reproduce-and-fix-issues/references/control-adapter.md` and the user's +completed feature map. For a canonical index, load every linked feature file +and its adjacent app contract. Verify that the configured map agrees with the +project policy. Run the policy check when present. Confirm that the named skill can: @@ -204,7 +226,10 @@ After the triage editor handoff is complete, give `automate` this complete repro - Trigger on the same new top-level reports in the configured source Slack channel. - Use the configured repository and default branch. - Read the source thread and reply only inside it. -- Include pull request creation and the configured tracker, control-adapter, and feature-map requirements. Paraphrase mapped user paths and states unless `automate` confirms an eligible committed file in the same repository. +- Include pull request creation and the configured tracker, control-adapter, + and feature-map requirements. For a shared project contract, require its + committed canonical paths. For a legacy project, paraphrase mapped user paths + only when `automate` cannot reference an eligible committed map. - Wait for a trusted triage marker before acting. - Reproduce the exact symptom twice through the mapped real UI and capture evidence. - Verify an existing fix without authoring over it. diff --git a/automations/benny/templates/configuration.example.yaml b/automations/benny/templates/configuration.example.yaml index 8616f5f..95f6299 100644 --- a/automations/benny/templates/configuration.example.yaml +++ b/automations/benny/templates/configuration.example.yaml @@ -46,7 +46,7 @@ routing: control: skill_name: "control-target-app" - feature_map_path: ".cursor/benny/feature-map.md" + feature_map_path: ".harness/verify/web/features/README.md" environment: "safe-test-environment-placeholder" artifact_directory: "/tmp/benny-artifacts" artifact_retention_hours: 24 diff --git a/automations/benny/templates/reproduce-automation-prompt.md b/automations/benny/templates/reproduce-automation-prompt.md index 5c66c92..5bf61af 100644 --- a/automations/benny/templates/reproduce-automation-prompt.md +++ b/automations/benny/templates/reproduce-automation-prompt.md @@ -22,6 +22,11 @@ Trigger: The creation intent should describe this as a new top-level report in the configured source Slack channel. It should include the configured repository, default branch, issue tracker, control adapter, feature map, and draft pull request capability. +When the repository has `.harness/policy.json`, include the instruction to read +`.harness/workflow.md` and use its canonical app contract. The configured +feature-map path must be that app's `features/README.md`; load its linked files +before driving. Keep the prompt as an entry pointer rather than a copied map. + Treat the source channel and root thread timestamp as immutable. If either is missing or does not match configuration, stop without posting. Wait for a configured triage marker from the configured triage identity in this exact thread. Proceed only for `[benny:bug]` or `[benny:performance]`. diff --git a/docs/guide/01-setup.md b/docs/guide/01-setup.md index d4a6379..fa40e47 100644 --- a/docs/guide/01-setup.md +++ b/docs/guide/01-setup.md @@ -29,9 +29,11 @@ Use `inherit-parent` for the current chat model through the Harness's documented ## Accept the verification offer, or don't -At the end of setup, `/setup-mstack` looks for a way to prove app behavior in your project, either a `verify-*` skill or an existing harness. If it finds neither, it offers once to generate one with [`/create-verification-skill`](../../skills/create-verification-skill/SKILL.md). +At the end of setup, `/setup-mstack` looks for a shared verification contract, a `verify-*` wrapper, or an existing repository driver. Several wrappers pointing at one contract count as one workflow. Setup reports conflicting legacy definitions instead of creating another copy. -Say yes and it writes `verify-/` under the active Harness's project skill root. See [the project skill paths](../../skills/create-verification-skill/references/harness-paths.md) for the root used by each Harness. The generated skill teaches agents to drive your app the way a user does, and setup proves it once before handing it over. Say no and setup moves on. You can run `/create-verification-skill` yourself any time. [Verify and ship](./06-verify-and-ship.md#create-a-project-verification-skill) covers when it earns its place. +If it finds neither a reusable verification workflow nor a repository driver, setup offers once to run [`/create-verification-skill`](../../skills/create-verification-skill/SKILL.md). When you accept, the generator writes `.harness/verify//contract.md`, its feature map, and thin discovery wrappers. It proves one feature in the real app before handing over the result. Model setup alone does not start the app or create a team policy. + +You can run `/create-verification-skill` yourself any time. [Verify and ship](./06-verify-and-ship.md#create-a-project-verification-skill) covers the proof. For a team using both native pstack and mstack, follow [the mixed-Harness adoption guide](./11-mixed-harness.md) to add the shared project workflow and CI checks. After setup, start a new session. The model configuration applies to new sessions. diff --git a/docs/guide/06-verify-and-ship.md b/docs/guide/06-verify-and-ship.md index ac8cffc..457ea79 100644 --- a/docs/guide/06-verify-and-ship.md +++ b/docs/guide/06-verify-and-ship.md @@ -34,11 +34,13 @@ The UI bullet above hides a real requirement. The agent needs a scripted way to [`/create-verification-skill`](../../skills/create-verification-skill/SKILL.md) interviews the repository, not you. It works out what a user touches, how the app launches locally, what can drive it (an existing harness first, otherwise browser and CDP, a PTY, or plain HTTP), what evidence proves behavior, and whether two instances can run side by side. It asks you only what the code can't answer. -It writes `verify-/` under the active Harness's project skill root. See [the project skill paths](../../skills/create-verification-skill/references/harness-paths.md) for the root used by each Harness. The generated skill contains exact Launch, Doctor, Drive, Evidence, and Cleanup sections, plus a feature map under `features/` that indexes what the app does and what result proves each feature works. The skill ships a [worked feature-map example](../../skills/create-verification-skill/references/feature-map-example/) with a README index and one file per feature using the four required H2s. Before handing it over, the generator proves the skill once end to end: launch, doctor check, drive one feature, capture evidence, and clean up. If that proof fails, don't use the output. +It writes one `.harness/verify//contract.md` with Launch, Isolation, Doctor, Drive, Evidence, Cleanup, and Helpers sections. The adjacent `features/README.md` indexes the app's user paths and expected results. Owned scripts live in `helpers/`. The [worked feature-map example](../../skills/create-verification-skill/references/feature-map-example/) shows the required detail. -From then on, "verify it in the app" is a step any agent can execute, in this repo, with no setup conversation. +The generator then renders thin `verify-` wrappers for the configured Harnesses. Each wrapper's `metadata.verification-contract` points at the same contract from the repository root. Wrappers select an available driver without copying launch commands, selectors, or proof requirements. Because Cursor also discovers `.agents/skills/`, a project using both Cursor and Codex has one shared wrapper there. See [the wrapper paths](../../skills/create-verification-skill/references/harness-paths.md). -Once the verify skill works, a [`/swarm`](../../skills/swarm/SKILL.md) can split a full pass by feature-map entry and aggregate the results. +Before handing over the result, the generator launches an isolated instance, runs Doctor, drives one feature, captures evidence, and cleans up. The evidence must survive cleanup. One successful feature does not prove every mapped path or every Harness. A missing required capability remains a reported gap. + +A [`/swarm`](../../skills/swarm/SKILL.md) can split a full pass by feature-map entry. Give concurrent drivers separate instances, ports, and data directories. When the app cannot isolate them, keep one coordinator driving serially. ## Keep the verification skill honest @@ -48,7 +50,11 @@ Apps change and feature maps rot. When yours drifts, run: /maintain-verification-skill ``` -[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means one PR of proven corrections, confined to the verification skill's own directory. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of papering over it in docs. +[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) resolves wrappers to the canonical contract. Read-only source readers cover features in parallel, then one coordinator drives every mapped feature. Corrections go into `.harness/verify//`, with wrapper regeneration only when needed. Cursor automation and a local Claude Code session maintain the same map. + +The outcome is `clean` when all features have source and live coverage without corrections, `changed` when one PR contains proven corrections, or `blocked` when coverage cannot finish. The pass never edits product code. It reports product regressions without changing the map to accept broken behavior. + +For repositories with `.harness/policy.json`, follow `.harness/workflow.md` to run required commands and record evidence at the final commit. A passing policy check or command receipt does not certify app behavior. Review the named artifacts against the mapped feature. [Mixed-Harness adoption](./11-mixed-harness.md) describes the CI and handoff steps. ## Open the PR diff --git a/docs/guide/11-mixed-harness.md b/docs/guide/11-mixed-harness.md new file mode 100644 index 0000000..5aece5b --- /dev/null +++ b/docs/guide/11-mixed-harness.md @@ -0,0 +1,256 @@ +# Mix pstack and mstack in one repository + +Use this procedure when teammates use native pstack in Cursor or Grok Bot and +mstack in Codex, Claude Code, OpenCode, or pi. Cursor and Grok Bot can also use +mstack when installed. Keep each person's workflow installation separate while +the repository owns one verification contract per app. + +The commands below use `mstack-policy`. This checkout adds the command; older +npm releases do not contain it. During development, replace `mstack-policy` with +`node /scripts/check-harness-policy.mjs`. After a release that +contains this command, invoke the team's exact package version with: + +```bash +npx --package @3metajun/mstack@ mstack-policy --help +``` + +Initialization exports a structural checker that CI runs from the business +repository. CI does not need to download this unreleased package. The full CLI +still comes from the reviewed development checkout or, after release, an exact +package version. Replace every example version, commit, command, app ID, and +repository path with the team's actual value. + +## Initialize the shared project workflow + +Work in a dedicated feature worktree with one writer. Inventory existing +verification skills before adding a new definition: + +```bash +rg --files --hidden -g SKILL.md -g '*feature*' -g '*contract*' -g '!node_modules' -g '!.git' +``` + +Choose the repository's actual base branch, required commands, supported +Harnesses, and exact workflow revisions. Record a pstack commit rather than a +branch or tag that can move. Initialize the project, for example: + +```bash +mstack-policy init --root . --app web --base main --check 'node --test' --pstack --harness cursor --harness codex +``` + +The command creates `.harness/policy.json`, `.harness/workflow.md`, and the +structural checker files `.harness/check.mjs` and +`.harness/harness-policy-lib.mjs`. It adds project-entry pointers and creates the +canonical app directory. Existing instructions remain in place. The command +refuses to replace a different generated file; review and reconcile the diff +before retrying. Omitting `--harness` selects all supported Harnesses. A +team using mstack only can omit `--pstack`, including in Cursor or Grok Bot. +Include `--pstack` whenever the team also uses native pstack. Selecting a +Harness does not select its workflow. + +The mstack revision comes from the CLI package version. During development, +retain the reviewed checkout commit as well, since an unreleased checkout can +share a package version with an older release. + +The generated `AGENTS.md` pointer directs agents to `.harness/workflow.md`. +Initialization also adds a `CLAUDE.md` pointer for Claude Code and a Cursor +`alwaysApply` rule for Cursor or Grok Bot when those Harnesses are selected. +These entry points matter for native `/poteto-mode`, +whose installed upstream instructions still exist. A JSON file cannot override +the upstream workflow by itself. Keep the project entry rules active in every +team Harness. User instructions retain their normal priority. + +Initialization does not write `contract.md` or discover application behavior. +The project check remains incomplete until the next step writes and proves the +contract and generates its wrappers. During this initial adoption or migration, +finish the contract in the isolated task worktree before running preflight. + +## Create or migrate one verification definition + +Run `/create-verification-skill` against the real application. Put the result in: + +```text +.harness/ + policy.json + workflow.md + check.mjs + harness-policy-lib.mjs + verify/ + web/ + contract.md + features/ + README.md + create-note.md + helpers/ +``` + +The contract has H2 sections named `Launch`, `Doctor`, `Drive`, `Evidence`, +`Cleanup`, and `Isolation`. Add `Helpers` to document owned scripts. These +sections own startup, instance state, user actions, expected results, evidence, +and cleanup. Keep available +Harness tool names in their installed driver skills. The canonical contract can +name a repository-owned Playwright script, API client, or CLI directly. + +When legacy `.cursor`, `.agents`, `.claude`, `.opencode`, or `.pi` skills contain +app facts, reconcile their differences against source and a running app. Move +the contract, maps, fixtures, and helpers together, then fix their relative +paths. Update maintenance jobs and Benny's `control.feature_map_path` to read +`.harness/verify/web/features/README.md` and its linked files. + +Prove the moved instructions before retiring the old definitions. Remove +obsolete maps in the same PR. The checker reports legacy conflicts; it does not +select the correct copy or silently delete one. Use the +[verification migration procedure](../../skills/create-verification-skill/references/harness-paths.md#migrate-existing-verification) +for the full inventory. + +Render wrappers after the canonical files exist: + +```bash +mstack-policy wrappers --root . +node .harness/check.mjs +``` + +Wrappers contain `metadata.verification-contract` with a repository-relative +path and instructions to use the available driver. They contain no separate +map or launch procedure. Cursor scans `.agents/skills/`, so a Cursor and Codex +project gets one shared wrapper there. Other configured Harnesses receive the +same wrapper in their discovery roots. + +For a project that only needs a reusable verification skill, generate wrappers +without adopting a policy: + +```bash +mstack-policy wrappers --root . --app web --harness codex --harness claude +``` + +Start from each wrapper and confirm it reaches the same contract. Drive at least +one mapped feature, inspect its evidence, and clean up the owned instance. A +static wrapper check does not prove that every installed Harness has a working +browser or desktop driver. + +## Start each development run in its own worktree + +Give each writable task a dedicated linked worktree and branch. Separate app +ports, data directories, profiles, and artifact locations as the contract +requires. If a resource cannot be isolated, follow its exclusive-use procedure. +A second agent must not edit a checkout another agent is using. + +Fetch the actual PR target before preflight, then run the shared workflow: + +```bash +git fetch origin main +mstack-policy preflight --root . --harness codex --workflow mstack --revision --base-ref origin/main +``` + +For native pstack, use the same project command with the actual Harness and +workflow declaration: + +```bash +mstack-policy preflight --root . --harness cursor --workflow pstack --revision --base-ref origin/main +``` + +Preflight checks the policy, clean dedicated worktree, branch, and declared +workflow revision. The `--revision` value is a declaration checked against the +project pin. It does not inspect another tool's installation. Confirm installed +versions through that Harness's supported installation records during rollout +and upgrades. + +Plain clones still support read-only inspection. The writable preflight requires +a dedicated linked worktree so concurrent tasks have separate checkout state. + +## Record evidence at the commit being reviewed + +Drive the changed user path through the canonical contract. Capture the action, +result, and a second read path for durable side effects. Retain artifacts outside +tracked source, under a location the project can attach to the run receipt. +Finish source changes and commit them before recording the final checks: + +```bash +mstack-policy record --root . --harness codex --workflow mstack --revision --base-ref origin/main --feature web/create-note --evidence .harness/runs/create-note.png --evidence .harness/runs/create-note.json +``` + +`record` runs the policy's required commands and writes command logs and a +receipt under `.harness/runs//`. The receipt binds the feature, declared +workflow, base, head, command results, and evidence digests. Keep this directory +ignored. Use an artifact archive when a reviewer or CI needs the receipt and +its evidence outside your machine. + +Validate the resulting receipt against the current checkout: + +```bash +mstack-policy check --root . --receipt .harness/runs//receipt.json +``` + +A command result of `PASS` means the recorded commands passed. Runtime review +remains required. A reviewer must inspect the named artifacts against the +feature's expected behavior. A unit test result cannot replace a required UI +flow, and a receipt cannot determine whether a screenshot proves that flow. + +After a code change, rebase, or target-branch update, rerun the required proof +and record a fresh receipt for the new base and head. For handoff, give the next +agent the branch, commit, canonical feature ID, receipt, artifacts, and unresolved +work. Do not require access to another Harness's transcript. + +## Add CI and repository protection + +Commit the exported checker and its library with the policy. Run it from the +business repository root in CI: + +```bash +node .harness/check.mjs +``` + +This command validates project structure without installing mstack or executing +repository verification commands. Run the policy's actual required commands in +separate CI steps. The exported checker accepts no arguments and does not +validate run receipts. + +For receipt validation, use the full CLI from the reviewed checkout or a release +that contains it. Restore the receipt, command logs, and evidence at their +recorded relative paths. Check out the exact PR head on the receipt's named +task branch and fetch its recorded target reference. A detached checkout or a +synthetic merge commit does not match that receipt. Then run: + +```bash +mstack-policy check --root . --receipt .harness/runs//receipt.json +``` + +Make these CI jobs required through the repository host's protected-branch +settings. Configure reviews and the team's merge permissions there. The policy +file and local CLI do not enable branch protection, prevent direct remote +pushes, or prove that a check is required. Verify those host settings before +calling the team's rollout complete. Protect the CI workflow and exported +checker through the team's review rules so a PR cannot silently remove its own +gate. A merge queue or a strict up-to-date requirement closes the gap between +verification and integration. + +Keep publishing under the team's existing release owner. If the project +publishes to npm and requires a matching GitHub Release, verify both from the +release workflow. Do not infer release authorization from a passing receipt. + +## Prove the team rollout and maintain it + +Test the adopted project through native pstack and an mstack Harness using +separate worktrees. Confirm that both enter through the shared project workflow, +resolve the same canonical map, run its real app path, and produce evidence a +reviewer can inspect. Then rerun `/maintain-verification-skill` from either +workflow and confirm corrections land in the canonical directory. + +An unavailable Harness or app prerequisite is an explicit rollout gap. A CLI +fixture or a passing static check does not establish real-team compatibility. + +Pin workflow revisions independently. During an upgrade, compare changes to +verification generation and maintenance instructions, regenerate wrappers when +needed, and repeat affected rollout checks. + +Review checker upgrades as project changes. Compare the proposed version's +`scripts/harness-check.mjs` and `scripts/harness-policy-lib.mjs` with the two +exported files. Copy both reviewed versions together in a dedicated change, +preserving any intentional local adaptation through a reviewed merge. Update the +project workflow or policy when their requirements change. Run the exported +checker and the app's affected verification before accepting the upgrade. +`init` preserves differing generated files; rerunning it is not an upgrade +command and does not silently replace the checker. + +Keep model choices in each user's Harness configuration. Delete completed plans +and retain durable lessons in the canonical contract, tests, or project +documentation. diff --git a/docs/guide/README.md b/docs/guide/README.md index 40352d5..4490f38 100644 --- a/docs/guide/README.md +++ b/docs/guide/README.md @@ -14,6 +14,7 @@ Here's what you'll learn: 8. [Steer with principle names](./08-principles.md). The 23 names that redirect an agent mid-task. 9. [Make it yours](./09-make-it-yours.md). Your own mode, plus how to test a skill change. 10. [Recipes and pitfalls](./10-recipes-and-pitfalls.md). Prompts to copy and mistakes to skip. +11. [Mix pstack and mstack in one repository](./11-mixed-harness.md). Share verification facts, isolate work, and check evidence across Harnesses. Read the pages in order the first time. After that, each page stands alone. diff --git a/docs/harness-adapters.md b/docs/harness-adapters.md index b6c1e3b..040deba 100644 --- a/docs/harness-adapters.md +++ b/docs/harness-adapters.md @@ -25,6 +25,36 @@ can read. Claude Code accepts `metadata` but does not act on its contents, so the `compatibility` field instead. OpenCode and pi retain `metadata` because their official references support it. +## Shared project verification wrappers + +Project verification has one canonical contract at +`.harness/verify//contract.md`, with its map and helpers beside it. +`mstack-policy wrappers` generates discovery wrappers separately from the +portable skill installer. Each wrapper retains +`metadata.verification-contract` as a repository-relative pointer, including +on Claude Code. The wrapper body also names the contract, so execution does +not depend on the Harness interpreting custom metadata. The policy checker +uses the metadata to validate the shared target. + +[Cursor's skill documentation](https://cursor.com/docs/context/skills) includes +both `.cursor/skills/` and `.agents/skills/` as project discovery locations. +For a project configured for Codex and Cursor or Grok Bot, generate one neutral +`verify-` wrapper in `.agents/skills/`. A project using only Cursor or Grok +Bot places it in `.cursor/skills/`. Claude Code, OpenCode, and pi use the project +roots in the table above. These wrappers contain the same capability-neutral +instructions, with no duplicate maps or host-specific driver commands. + +`compatibility` text is not a reliable discovery filter. Two wrappers with +different app instructions remain conflicting definitions even if one says +"Codex only". The project check rejects conflicting or stale wrapper structure. + +Native pstack installations keep their own workflow instructions. The project +initializer adds an `AGENTS.md` entry pointer and a Cursor `alwaysApply` rule +that load `.harness/workflow.md`. Claude Code also receives a `CLAUDE.md` +pointer when selected. The policy alone cannot change `/poteto-mode`. +See [mixed-Harness adoption](./guide/11-mixed-harness.md) for the commands, +migration, and evidence requirements. + ## Delegation and session records Codex stores agent configuration in `.codex/agents/*.toml` and diff --git a/package.json b/package.json index f2453ce..de7dbb9 100644 --- a/package.json +++ b/package.json @@ -16,6 +16,7 @@ "check-package": "node scripts/check-package.mjs", "check-models": "node scripts/model-config.mjs", "check-upstream": "node scripts/check-upstream.mjs", + "check-harness-policy": "node scripts/check-harness-policy.mjs", "skill-baseline": "node scripts/skill-baseline.mjs", "run-role": "node scripts/run-role.mjs", "remote-install": "node scripts/remote-install.mjs", @@ -24,10 +25,11 @@ "install-skills": "node scripts/install.mjs", "optimize-context": "node scripts/optimize-context.mjs", "reconcile-context": "node scripts/reconcile-context.mjs", - "test": "node scripts/validate.mjs && node scripts/skill-baseline.mjs --check && node --test scripts/install.test.mjs scripts/context.test.mjs scripts/model-config.test.mjs scripts/audit-context.test.mjs scripts/worktree-audit.test.mjs scripts/sync-upstream.test.mjs scripts/runtime.test.mjs scripts/environment.test.mjs scripts/skill-integrity.test.mjs scripts/skill-baseline.test.mjs scripts/version-integrity.test.mjs scripts/agent-format.test.mjs scripts/history.test.mjs" + "test": "node scripts/validate.mjs && node scripts/skill-baseline.mjs --check && node --test scripts/install.test.mjs scripts/context.test.mjs scripts/model-config.test.mjs scripts/audit-context.test.mjs scripts/worktree-audit.test.mjs scripts/sync-upstream.test.mjs scripts/runtime.test.mjs scripts/environment.test.mjs scripts/skill-integrity.test.mjs scripts/skill-baseline.test.mjs scripts/version-integrity.test.mjs scripts/agent-format.test.mjs scripts/history.test.mjs scripts/harness-policy.test.mjs scripts/harness-project.test.mjs scripts/check-upstream-adaptations.test.mjs" }, "bin": { - "mstack": "scripts/install.mjs" + "mstack": "scripts/install.mjs", + "mstack-policy": "scripts/check-harness-policy.mjs" }, "files": [ "skills", diff --git a/profiles/skill-manifest.json b/profiles/skill-manifest.json index 81d0f63..00bc968 100644 --- a/profiles/skill-manifest.json +++ b/profiles/skill-manifest.json @@ -83,7 +83,7 @@ "upstream": "pstack", "source": "skills/create-verification-skill/references/feature-map-example/README.md", "sourceDigest": "cb7bd782cf89968a4ba3d58a5151db837430db92d19a6f52a906973b77b516ba", - "targetDigest": "3810cf289bdd6657f3f23f974262536304a7365ee038e125b7a80343ce7000bf" + "targetDigest": "a23475211980e77339bf471bd02bb0c107f72f8149303ae1557de80e36df94df" }, "skills/create-verification-skill/references/feature-map-example/search.md": { "upstream": "pstack", @@ -94,13 +94,13 @@ "skills/create-verification-skill/references/harness-paths.md": { "upstream": null, "reason": "Local portability guidance or implementation; review alongside its owning skill.", - "targetDigest": "9b785b0c2dd47b2c9216710641107864a34db2940dd3633869c5d7ccdae182e5" + "targetDigest": "62705c0e0ae6832c20c1f1b53bb91b05c036dd6879f789de6bc4499815c99ab4" }, "skills/create-verification-skill/SKILL.md": { "upstream": "pstack", "source": "skills/create-verification-skill/SKILL.md", "sourceDigest": "644f2551403c1bca01a2855b34611b6e7be0ce0dc5b204514c376c0f6a6e6ac4", - "targetDigest": "7242b478eb0face01b1f1d27950efa69e415219b278993bc3e75f297707bba48" + "targetDigest": "eca219cac2808714d6aa3090fd3065d1da6a579d432f277aef334f1ca0dfe770" }, "skills/diagnosing-bugs/references/feedback-loops.md": { "upstream": null, @@ -182,7 +182,7 @@ "upstream": "pstack", "source": "skills/maintain-verification-skill/SKILL.md", "sourceDigest": "515c0eaa054b3f6be1b1fb06f2c2f173c80fddb58bbcac57576f89c479bc68e8", - "targetDigest": "bfa8b97a5ca35d618228c37eb02c74589c0fcb1af13d06201b10e0eb3274e32c" + "targetDigest": "b236ab1a439ae067d41d0fc191517992da1130d7935d3c8fa5084def26512296" }, "skills/make-bot-ui/SKILL.md": { "upstream": "pstack", @@ -343,7 +343,7 @@ "upstream": "pstack", "source": "skills/poteto-mode/SKILL.md", "sourceDigest": "6bbf35cb2111c358461c7ba90f210dad4573f3a3ada92f6459eac0c69d83b01a", - "targetDigest": "53cb6ae809072973802f78fac92fc75efbe111330271513fa756cd2f0d053ad4" + "targetDigest": "798df0dae0ca4959b81d6465b1c3c392ffc5cefd98f5b0834a53262eb1b2fa86" }, "skills/no-comments/SKILL.md": { "upstream": "pstack", @@ -539,7 +539,7 @@ "upstream": "pstack", "source": "skills/setup-pstack/SKILL.md", "sourceDigest": "5acdc3db4f15f3e9b81fb56eab7f17cdccde1a11a40053ed7743f7467729c8a9", - "targetDigest": "555c577bf494baafafa980ee3f647149c180a615a706ef409845cd03cd16e142" + "targetDigest": "5fbe454ff5cea061d8b29e5be4a0eb1d2f040f99927f23ccd6007ec5c02e5b6a" }, "skills/show-me-your-work/references/decision-log-template.tsv": { "upstream": "pstack", diff --git a/profiles/upstream-manifest.json b/profiles/upstream-manifest.json index 9c6fe65..aeffd94 100644 --- a/profiles/upstream-manifest.json +++ b/profiles/upstream-manifest.json @@ -85,7 +85,7 @@ }, "create-verification-skill": { "source": "644f2551403c1bca01a2855b34611b6e7be0ce0dc5b204514c376c0f6a6e6ac4", - "target": "7242b478eb0face01b1f1d27950efa69e415219b278993bc3e75f297707bba48" + "target": "eca219cac2808714d6aa3090fd3065d1da6a579d432f277aef334f1ca0dfe770" }, "figure-it-out": { "source": "982f2c1024a3a24f6417426ad8e8a3e027fe6210a8a442308fb85073f8f3f6ef", @@ -101,7 +101,7 @@ }, "maintain-verification-skill": { "source": "515c0eaa054b3f6be1b1fb06f2c2f173c80fddb58bbcac57576f89c479bc68e8", - "target": "bfa8b97a5ca35d618228c37eb02c74589c0fcb1af13d06201b10e0eb3274e32c" + "target": "b236ab1a439ae067d41d0fc191517992da1130d7935d3c8fa5084def26512296" }, "make-bot-ui": { "source": "3624c8ef16cf2e48deb2a4049e72c6962a9431c8d6c6af90e7bd7ea0375224ec", @@ -109,7 +109,7 @@ }, "meta-mode": { "source": "9319963075ef537e1d83edc4814f6bc6a7c90779ea261227934c984a0a5e88fb", - "target": "53cb6ae809072973802f78fac92fc75efbe111330271513fa756cd2f0d053ad4" + "target": "798df0dae0ca4959b81d6465b1c3c392ffc5cefd98f5b0834a53262eb1b2fa86" }, "no-comments": { "source": "5c5b0882297d704c3a9720c52b7a793c68b013eaf717989f0945624efdfe2b05", @@ -217,7 +217,7 @@ }, "setup-mstack": { "source": "a11137437e12831c73f228f449bdfefab487bf7e2c69a4b306189e8fb1e6ab3e", - "target": "555c577bf494baafafa980ee3f647149c180a615a706ef409845cd03cd16e142" + "target": "5fbe454ff5cea061d8b29e5be4a0eb1d2f040f99927f23ccd6007ec5c02e5b6a" }, "show-me-your-work": { "source": "831e85ba3f84f38bf338cd03e6af050fea357752e25a825e334c99f59d7f2077", @@ -251,5 +251,67 @@ "source": "222b93789754ba6ee901835137184acdca667f7e47c523bcc85aca86675fb249", "target": "873aedc86d1321093e9a195ac8bff1783241b309a1d33808ab74f181c4b8f22d" } + }, + "adaptedArtifacts": { + "automations/benny/FOR_AGENTS.md": { + "source": "9e86916edf7fdddf3325dac869d947d5cab767ee40f1b5eaf5d16e31f2a5322a", + "target": "5a8f7571f9aa66d16434e2d4bb6718c9e76141843f68c14e4e2f6a8fc39accce", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/README.md": { + "source": "339fa6a5e1b07dbd708e87bc423e31e25ab3ac7e4292b80d3330abe992ee3245", + "target": "4dacbeb44e84630bbb7d88d119bd3ed9a76f679fb691a280463f5473e192d69e", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md": { + "source": "58b0615d2fca895f9a20df83266e280042b0d585b5f2fd324e2094a682fd846b", + "target": "02c92e00d47c70c41bf9f469b556ea8d071483ad1eca70c3b27642dbc1f611f5", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md": { + "source": "1e5c7c8522daec3e43fcc2e73ceb97289ded65b552949ce65a11cc8eb3d2b6ee", + "target": "c8a6639a5f4c377da6ce4e3a75a1ab3a843b1a17bb640cf758939eee0c1af2ea", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/skills/reproduce-and-fix-issues/references/verify-existing-fix.md": { + "source": "c90eae00abcdec71a58fa8d60f2c495b082a526ad50b8e8e494837609129bf29", + "target": "d62d3a59cddf77f591ee62b123e367faf805f64ae7d18103741ac04207ebbe39", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/skills/reproduce-and-fix-issues/SKILL.md": { + "source": "2cb174442390be10cb7e7767e89bb1e94fda4d014426c40888397ad8286060ff", + "target": "6cd1740f9a1edaf6ab02eaf93f8690d6f4062e68b35be06c50585ac71f9798b6", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/skills/setup-benny/SKILL.md": { + "source": "c5bbca8d97cc6975d34e84c42ac4fd59e04c7f77892e0af534b3e58ca0fd43df", + "target": "269ab1c55733cbc176eb07c9268b891a5b03051be90fec1d053dc03ecc870b69", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/templates/configuration.example.yaml": { + "source": "13e24b4ce9b9a0913916e85fe3b080b88d45f8e55dccfe8ae38e0751a997b6f5", + "target": "6ba873ecb944db84f46965238f131e195f9ac9e26fd037d6c4a198d9404d3c3b", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "automations/benny/templates/reproduce-automation-prompt.md": { + "source": "71ec574f1844e379394ae4ef14c48d7df10afbaa319d395c788d7f856f0e6899", + "target": "f0238f77e8339eba897e2a85fa1ee328ed3845be14f65d01863a8f0938eb538d", + "reason": "Route native Benny verification and maintenance through the shared canonical contract and feature map." + }, + "docs/guide/01-setup.md": { + "source": "6786c9090de6092d87836214b690e2b5640ddc7f34b19c0b9e2af794fcc17e7e", + "target": "b67fd0a7950464e1094225a2f85add0714849c1daab6c5289ad7bbc653baa536", + "reason": "Document one canonical verification contract and the mixed pstack/mstack project workflow." + }, + "docs/guide/06-verify-and-ship.md": { + "source": "53a6bc8b3e850e34c4d8b1785a488eb7fbcbfe94ed588aeeaca252a5eb6e442c", + "target": "e99e7108a757eee96397ebb79c6bbbc6f30084a35eb32d98f8de2e2b9c1aad18", + "reason": "Document one canonical verification contract and the mixed pstack/mstack project workflow." + }, + "docs/guide/README.md": { + "source": "e76f2e20364a4021707394b0557751e3aa81be2c819b19aaa705de08086515a5", + "target": "fe7261350a49c8bf1961e6ad8536d776479e65afcf78b14e1b45d1c0f3969de9", + "reason": "Document one canonical verification contract and the mixed pstack/mstack project workflow." + } } } diff --git a/scripts/check-harness-policy.mjs b/scripts/check-harness-policy.mjs new file mode 100644 index 0000000..a804915 --- /dev/null +++ b/scripts/check-harness-policy.mjs @@ -0,0 +1,70 @@ +#!/usr/bin/env node +import { existsSync, realpathSync } from "node:fs"; +import { resolve } from "node:path"; +import { initProject, installWrappers, readPolicy, preflight, recordVerification, checkReceipt } from "./harness-project.mjs"; +import { checkProject } from "./harness-policy-lib.mjs"; + +const commands = { + check: ["root", "receipt"], + init: ["root", "app", "check", "pstack", "harness", "base"], + wrappers: ["root", "app", "harness"], + preflight: ["root", "harness", "workflow", "revision", "base-ref"], + record: ["root", "harness", "workflow", "revision", "base-ref", "feature", "evidence"], +}; + +function argumentsFor(argv) { + const args = [...argv]; + const command = args[0] && !args[0].startsWith("--") ? args.shift() : "check"; + if (!Object.hasOwn(commands, command)) throw new Error("Unknown command: " + command); + const repeated = command === "init" || command === "wrappers" ? ["app", "check", "harness"] : ["evidence"]; + const options = {}; + while (args.length) { + const flag = args.shift(); + const key = flag?.slice(2); + if (!flag?.startsWith("--") || !commands[command].includes(key)) throw new Error("Unknown option: " + flag); + const value = args.shift(); + if (!value?.trim() || value.startsWith("--")) throw new Error(flag + " requires a value"); + if (repeated.includes(key)) (options[key] ??= []).push(value); + else { + if (Object.hasOwn(options, key)) throw new Error("Duplicate option: " + flag); + options[key] = value; + } + } + return { command, options }; +} + +export function main(argv) { + if (argv.length === 1 && argv[0] === "--help") { + console.log("Usage: mstack-policy [check|init|wrappers|preflight|record] --root \n" + + "init: --app --check [--pstack ] [--base main] [--harness ]\n" + + "wrappers: uses policy, or --app --harness without one\n" + + "preflight: --harness --workflow --revision --base-ref \n" + + "record: preflight flags, --feature --evidence \n" + + "check: optional --receipt \n" + + "Repeat --app, --check, --harness during init; repeat --evidence during record.\n" + + "check validates files. preflight observes Git state. record runs repository commands."); + return 0; + } + const { command, options } = argumentsFor(argv); + const root = resolve(options.root ?? process.cwd()); + let result; + if (command === "init") result = initProject(root, options); + else if (command === "wrappers") result = installWrappers(root, options); + else { + const policy = readPolicy(root); + if (command === "preflight") result = preflight(root, policy, options); + else if (command === "record") result = recordVerification(root, policy, options); + else { + const errors = checkProject(root, policy); + if (errors.length) throw new Error(errors.join("\n")); + result = options.receipt ? checkReceipt(root, policy, options.receipt) : { status: "PASS", scope: "project contract structure" }; + } + } + console.log(JSON.stringify(result)); + return result.status === "FAIL" ? 1 : 0; +} + +if (process.argv[1] && existsSync(process.argv[1]) && realpathSync.native(process.argv[1]) === realpathSync.native(new URL(import.meta.url))) { + try { process.exitCode = main(process.argv.slice(2)); } + catch (error) { console.error("mstack-policy: " + error.message); process.exitCode = 1; } +} diff --git a/scripts/check-package.mjs b/scripts/check-package.mjs index 9c9a279..c763675 100644 --- a/scripts/check-package.mjs +++ b/scripts/check-package.mjs @@ -51,7 +51,12 @@ if (forbidden.length) { const baseline = JSON.parse(readFileSync(join(repoRoot, "profiles", "skill-manifest.json"), "utf8")); const packedFiles = new Set(report.files.map((file) => file.path)); -const missing = Object.keys(baseline.files).filter((path) => !packedFiles.has(path)); +const requiredFiles = [ + ...Object.keys(baseline.files), ...Object.values(packageManifest.bin), + "scripts/harness-project.mjs", "scripts/harness-policy-lib.mjs", + "scripts/harness-workflow.md", "scripts/harness-check.mjs", +]; +const missing = requiredFiles.filter((path) => !packedFiles.has(path)); if (missing.length) { throw new Error(`npm package is missing reviewed skill files:\n${missing.join("\n")}`); } diff --git a/scripts/check-upstream-adaptations.test.mjs b/scripts/check-upstream-adaptations.test.mjs new file mode 100644 index 0000000..72f3d92 --- /dev/null +++ b/scripts/check-upstream-adaptations.test.mjs @@ -0,0 +1,134 @@ +import assert from "node:assert/strict"; +import { execFileSync, spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; + +const scripts = dirname(fileURLToPath(import.meta.url)); +const digest = (content) => createHash("sha256").update(content.replaceAll("\r\n", "\n")).digest("hex"); +const artifact = "automations/benny/README.md"; +const upstream = "Use /poteto-mode with pstack.\n"; +const transformed = "Use /meta-mode with mstack.\n"; +const adapted = "Read the shared project contract before using /meta-mode.\n"; + +function write(root, path, content) { + mkdirSync(dirname(join(root, path)), { recursive: true }); + writeFileSync(join(root, path), content); +} + +function json(root, path, content) { + write(root, path, JSON.stringify(content, null, 2) + "\n"); +} + +function git(root, ...args) { + return execFileSync("git", ["-C", root, ...args], { encoding: "utf8", windowsHide: true, stdio: ["ignore", "pipe", "pipe"] }).trim(); +} + +function fixture(t) { + const temporary = mkdtempSync(join(tmpdir(), "mstack-adaptation-")); + t.after(() => rmSync(temporary, { recursive: true, force: true })); + const source = join(temporary, "source"); + const target = join(temporary, "target"); + const skill = "---\nname: sample\ndescription: Verify the sample.\n---\n\nRun the sample.\n"; + write(source, "skills/sample/SKILL.md", skill); + write(target, "skills/sample/SKILL.md", skill); + write(source, artifact, upstream); + write(target, artifact, adapted); + write(source, "automations/benny/unchanged.md", "Keep strict comparison.\n"); + write(target, "automations/benny/unchanged.md", "Keep strict comparison.\n"); + for (const path of ["agents/agent.md", "docs/guide/README.md", "skills/poteto-mode/scripts/tool.mjs"]) { + const sourcePath = path.replace("skills/poteto-mode/scripts", "tools"); + write(source, sourcePath, "Shared artifact.\n"); + write(target, sourcePath === "tools/tool.mjs" ? "tools/meta-mode/tool.mjs" : sourcePath, "Shared artifact.\n"); + } + git(source, "init", "--quiet"); + git(source, "config", "core.autocrlf", "false"); + const commit = () => { + git(source, "add", "."); + git(source, "-c", "user.name=Fixture", "-c", "user.email=fixture@example.invalid", "commit", "--quiet", "-m", "Fixture source"); + return git(source, "rev-parse", "HEAD"); + }; + const pinned = commit(); + const profile = { pstack: { repository: "https://example.invalid/pstack", commit: pinned, artifacts: { "meta-mode-tools": { source: "tools", target: "tools/meta-mode", compareContent: false } } } }; + const manifest = { + source: profile.pstack.repository, commit: pinned, + canonicalSkills: { sample: { source: digest(skill), target: digest(skill) } }, + artifacts: { automations: { [artifact]: digest(transformed), "automations/benny/unchanged.md": digest("Keep strict comparison.\n") } }, + adaptedArtifacts: { [artifact]: { source: digest(transformed), target: digest(adapted), reason: "Route verification through the shared project contract." } }, + }; + const save = () => { + json(target, "profiles/upstreams.json", profile); + json(target, "profiles/upstream-manifest.json", manifest); + }; + save(); + const run = (script, args) => spawnSync(process.execPath, [resolve(scripts, script), "--source", source, "--target", target, ...args], { encoding: "utf8", windowsHide: true }); + return { source, target, profile, manifest, commit, save, check: () => run("check-upstream.mjs", ["--strict"]), sync: () => run("sync-upstream.mjs", ["--apply"]) }; +} + +test("reviewed adaptations accept pinned source and target content, including CRLF, while unlisted files remain strict", (t) => { + const f = fixture(t); + write(f.target, artifact, adapted.replaceAll("\n", "\r\n")); + const passing = f.check(); + assert.equal(passing.status, 0, passing.stderr); + assert.match(passing.stdout, /reviewed artifact adaptations: 1 entries, 0 problem/); + assert.equal(f.manifest.artifacts.automations[artifact], digest(transformed)); + write(f.target, "automations/benny/unchanged.md", "Unreviewed change.\n"); + const rejected = f.check(); + assert.equal(rejected.status, 1); + assert.match(rejected.stderr, /changed: automations\/benny\/unchanged.md/); +}); + +test("a reviewed adaptation detects target edits and clean newly pinned source drift independently", (t) => { + const f = fixture(t); + write(f.target, artifact, adapted + "Unreviewed instructions.\n"); + const targetDrift = f.check(); + assert.equal(targetDrift.status, 1); + assert.match(targetDrift.stderr, /reviewed target content changed/); + write(f.target, artifact, adapted); + write(f.source, artifact, upstream + "New upstream requirement.\n"); + const commit = f.commit(); + f.profile.pstack.commit = commit; + f.manifest.commit = commit; + f.save(); + const sourceDrift = f.check(); + assert.equal(sourceDrift.status, 1); + assert.match(sourceDrift.stderr, /reviewed upstream source changed/); + assert.doesNotMatch(sourceDrift.stderr, /source provenance check failed/i); +}); + +test("adaptation metadata requires a reason, exact hashes, and an existing upstream artifact", (t) => { + const f = fixture(t); + const entry = f.manifest.adaptedArtifacts[artifact]; + for (const replacement of [{ ...entry, reason: " " }, { ...entry, source: "latest" }, { ...entry, target: null }]) { + f.manifest.adaptedArtifacts[artifact] = replacement; + f.save(); + const rejected = f.check(); + assert.equal(rejected.status, 1); + assert.match(rejected.stderr, /adaptation requires/); + } + f.manifest.adaptedArtifacts = { [artifact]: entry, "docs/guide/missing.md": entry }; + f.save(); + const stale = f.check(); + assert.equal(stale.status, 1); + assert.match(stale.stderr, /stale adaptation; no matching upstream artifact/); +}); + +test("sync preserves reviewed baselines and refuses to overwrite an adapted target", (t) => { + const f = fixture(t); + const before = readFileSync(join(f.target, "profiles/upstream-manifest.json"), "utf8"); + const refused = f.sync(); + assert.equal(refused.status, 1); + assert.match(refused.stderr, /Refusing to replace or remove changed upstream artifacts/); + assert.equal(readFileSync(join(f.target, artifact), "utf8"), adapted); + assert.equal(readFileSync(join(f.target, "profiles/upstream-manifest.json"), "utf8"), before); + write(f.target, artifact, transformed); + const applied = f.sync(); + assert.equal(applied.status, 0, applied.stderr); + const after = JSON.parse(readFileSync(join(f.target, "profiles/upstream-manifest.json"), "utf8")); + assert.deepEqual(after.adaptedArtifacts, f.manifest.adaptedArtifacts); + assert.equal(after.artifacts.automations[artifact], digest(transformed)); + assert.match(f.check().stderr, /reviewed target content changed/); +}); diff --git a/scripts/check-upstream.mjs b/scripts/check-upstream.mjs index edf1927..6fe8b83 100644 --- a/scripts/check-upstream.mjs +++ b/scripts/check-upstream.mjs @@ -22,7 +22,7 @@ if (args.includes("--help")) { "Usage: node scripts/check-upstream.mjs --source [--target ] [--strict]\n\n" + "Reports skill and transformed artifact drift, including files removed upstream according to the previous manifest. " + "--strict requires a clean source checkout at the pinned commit and exits non-zero for missing, changed, or removed artifacts. " + - "Manifest hashes are transformed upstream baselines; adapted compareContent=false files may differ.", + "Manifest hashes are transformed upstream baselines; reviewed adaptedArtifacts pin source and target content separately.", ); process.exit(0); } @@ -262,6 +262,35 @@ function equalContent(path, current, expected) { } let artifactProblems = 0; +const manifestPath = resolveInside(targetRoot, "profiles/upstream-manifest.json", "upstream manifest"); +let manifest; +if (!existsSync(manifestPath)) { + console.error("Missing upstream provenance manifest: profiles/upstream-manifest.json"); + if (strict) artifactProblems += 1; +} else { + try { manifest = JSON.parse(readFileSync(manifestPath, "utf8")); } + catch (error) { + console.error(`Invalid upstream provenance manifest: ${error.message}`); + if (strict) artifactProblems += 1; + } +} +const adaptations = new Map(); +const adaptationProblems = []; +if (manifest?.adaptedArtifacts !== undefined) { + if (!manifest.adaptedArtifacts || typeof manifest.adaptedArtifacts !== "object" || Array.isArray(manifest.adaptedArtifacts)) { + adaptationProblems.push("adaptedArtifacts must be an object"); + } else { + for (const [path, entry] of Object.entries(manifest.adaptedArtifacts)) { + const target = profilePath(path, "reviewed artifact adaptation", { allowRoot: false }); + if (target !== path || !entry || typeof entry !== "object" || Array.isArray(entry) + || Object.keys(entry).some((key) => !["source", "target", "reason"].includes(key)) + || !/^[a-f0-9]{64}$/.test(entry.source ?? "") || !/^[a-f0-9]{64}$/.test(entry.target ?? "") + || typeof entry.reason !== "string" || !entry.reason.trim()) { + adaptationProblems.push(`${path}: adaptation requires a normalized path, source and target SHA-256 hashes, and a review reason`); + } else adaptations.set(target, entry); + } + } +} const expectedTargets = new Map(); const missingSourceNames = new Set(); for (const [name, configured] of Object.entries(artifacts)) { @@ -306,8 +335,14 @@ for (const [name, configured] of Object.entries(artifacts)) { const changed = []; for (const [target, expectedContent] of expected) { const targetPath = resolveInside(targetRoot, target, `artifact ${name} computed target`); + const adaptation = adaptations.get(target); + if (adaptation) { + const contentDigest = (value) => textExtensions.has(extname(target).toLowerCase()) ? normalizedDigest(value) : digest(value); + if (adaptation.source !== contentDigest(expectedContent)) adaptationProblems.push(`${target}: reviewed upstream source changed`); + if (existsSync(targetPath) && adaptation.target !== contentDigest(readFileSync(targetPath))) adaptationProblems.push(`${target}: reviewed target content changed`); + } if (!existsSync(targetPath)) missing.push(target); - else if (spec.compareContent !== false && !equalContent(targetPath, readFileSync(targetPath), expectedContent)) { + else if (!adaptation && spec.compareContent !== false && !equalContent(targetPath, readFileSync(targetPath), expectedContent)) { changed.push(target); } } @@ -320,18 +355,14 @@ for (const [name, configured] of Object.entries(artifacts)) { } } -const manifestPath = resolveInside(targetRoot, "profiles/upstream-manifest.json", "upstream manifest"); -if (!existsSync(manifestPath)) { - console.error("Missing upstream provenance manifest: profiles/upstream-manifest.json"); - if (strict) artifactProblems += 1; -} else { - let manifest; - try { - manifest = JSON.parse(readFileSync(manifestPath, "utf8")); - } catch (error) { - console.error(`Invalid upstream provenance manifest: ${error.message}`); - if (strict) artifactProblems += 1; - } +for (const target of adaptations.keys()) { + if (!expectedTargets.has(target)) adaptationProblems.push(`${target}: stale adaptation; no matching upstream artifact`); +} +if (adaptations.size || adaptationProblems.length) console.log(`reviewed artifact adaptations: ${adaptations.size} entries, ${adaptationProblems.length} problem(s)`); +for (const problem of adaptationProblems) console.error(` ${problem}`); +if (strict) artifactProblems += adaptationProblems.length; + +{ if (manifest) { const canonicalSkills = manifest.canonicalSkills; if (!canonicalSkills || typeof canonicalSkills !== "object" || Array.isArray(canonicalSkills)) { diff --git a/scripts/harness-check.mjs b/scripts/harness-check.mjs new file mode 100644 index 0000000..3235712 --- /dev/null +++ b/scripts/harness-check.mjs @@ -0,0 +1,16 @@ +#!/usr/bin/env node +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { checkProject, safeProjectPath } from "./harness-policy-lib.mjs"; + +try { + if (process.argv.length !== 2) throw new Error("Run node .harness/check.mjs from the repository root"); + const root = resolve(process.cwd()); + const policy = JSON.parse(readFileSync(safeProjectPath(root, ".harness/policy.json"), "utf8")); + const errors = checkProject(root, policy); + if (errors.length) throw new Error(errors.join("\n")); + console.log("Shared verification structure passed; run the project's behavior checks separately."); +} catch (error) { + console.error("harness-check: " + error.message); + process.exitCode = 1; +} diff --git a/scripts/harness-policy-lib.mjs b/scripts/harness-policy-lib.mjs new file mode 100644 index 0000000..a69ca93 --- /dev/null +++ b/scripts/harness-policy-lib.mjs @@ -0,0 +1,341 @@ +import { lstatSync, readFileSync, readdirSync } from "node:fs"; +import { basename, dirname, isAbsolute, join, posix, resolve, win32 } from "node:path"; + +const HARNESSES = ["cursor", "grokbot", "codex", "claude", "opencode", "pi"]; +const SKILL_ROOTS = [".cursor/skills", ".agents/skills", ".claude/skills", ".opencode/skills", ".pi/skills"]; +const CONTRACT_SECTIONS = ["Launch", "Doctor", "Drive", "Evidence", "Cleanup", "Isolation"]; +const SEMVER = /^v?(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*)?$/; + +function object(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function objectFields(value, label, fields, errors) { + if (!object(value)) { + errors.push(`${label} must be an object`); + return false; + } + for (const key of Object.keys(value)) { + if (!fields.includes(key)) errors.push(`${label}.${key} is not supported`); + } + for (const key of fields) { + if (!Object.hasOwn(value, key)) errors.push(`${label}.${key} is required`); + } + return true; +} + +function stringList(value, label, errors, valid = () => true) { + if (!Array.isArray(value) || value.length === 0) { + errors.push(`${label} must be a non-empty array`); + return false; + } + const seen = new Set(); + for (const item of value) { + if (typeof item !== "string" || !item.trim() || item !== item.trim() || /[\x00-\x1f\x7f]/.test(item) || !valid(item)) { + errors.push(`${label} contains an invalid value: ${JSON.stringify(item)}`); + } else if (seen.has(item)) { + errors.push(`${label} contains a duplicate: ${item}`); + } + seen.add(item); + } + return true; +} + +function relativePath(value) { + return typeof value === "string" && value.length > 0 && !isAbsolute(value) && !win32.isAbsolute(value) + && !/[\\:\x00-\x1f\x7f<>"|?*]/.test(value) + && value.split("/").every((part) => part && part !== "." && part !== ".." && part.toLowerCase() !== ".git" && !/[. ]$/.test(part) + && !/^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/i.test(part)); +} + +function branchName(value) { + return typeof value === "string" && !!value && value !== "@" && !value.startsWith("-") + && !/[\s~^:?*\[\\\x00-\x1f\x7f]/.test(value) && !value.includes("..") && !value.includes("@{") + && !value.endsWith(".") && value.split("/").every((part) => part && !part.startsWith(".") && !part.endsWith(".lock")); +} + +function appName(value) { + return /^[a-z0-9](?:[a-z0-9-]{0,55}[a-z0-9])?$/.test(value) && relativePath(value); +} + +export function validateHarnessPolicy(policy) { + const errors = []; + if (!objectFields(policy, "policy", ["schemaVersion", "baseBranch", "allowedHarnesses", "workflows", "worktrees", "verification", "integration"], errors)) return errors; + if (policy.schemaVersion !== 1) errors.push("policy.schemaVersion must be 1"); + if (!branchName(policy.baseBranch)) errors.push("policy.baseBranch must be a valid Git branch name"); + stringList(policy.allowedHarnesses, "policy.allowedHarnesses", errors, (value) => HARNESSES.includes(value)); + if (!object(policy.workflows) || Object.keys(policy.workflows).length === 0) errors.push("policy.workflows must be a non-empty object"); + else { + for (const workflow of Object.keys(policy.workflows)) { + if (!["pstack", "mstack"].includes(workflow)) { + errors.push(`policy.workflows.${workflow} is not supported`); + continue; + } + const entry = policy.workflows[workflow]; + if (!objectFields(entry, `policy.workflows.${workflow}`, ["revision"], errors)) continue; + if (typeof entry.revision !== "string" || (!/^[a-f0-9]{40}$/i.test(entry.revision) && !SEMVER.test(entry.revision))) { + errors.push(`policy.workflows.${workflow}.revision must pin a full 40-character commit or exact SemVer`); + } + } + } + if (objectFields(policy.worktrees, "policy.worktrees", ["required", "sharedCheckout"], errors)) { + if (policy.worktrees.required !== true) errors.push("policy.worktrees.required must be true"); + if (policy.worktrees.sharedCheckout !== false) errors.push("policy.worktrees.sharedCheckout must be false"); + } + if (objectFields(policy.verification, "policy.verification", ["canonicalRoot", "apps", "requiredCommands"], errors)) { + const canonicalRoot = policy.verification.canonicalRoot; + if (!relativePath(canonicalRoot) || SKILL_ROOTS.some((root) => canonicalRoot.toLowerCase() === root || canonicalRoot.toLowerCase().startsWith(`${root}/`) || root.startsWith(`${canonicalRoot.toLowerCase()}/`))) { + errors.push("policy.verification.canonicalRoot must be a portable repository-relative path outside harness skill roots"); + } + stringList(policy.verification.apps, "policy.verification.apps", errors, appName); + stringList(policy.verification.requiredCommands, "policy.verification.requiredCommands", errors); + } + if (objectFields(policy.integration, "policy.integration", ["mode", "protectedBranches", "requireRebase"], errors)) { + if (policy.integration.mode !== "pull-request") errors.push("policy.integration.mode must be pull-request"); + if (policy.integration.requireRebase !== true) errors.push("policy.integration.requireRebase must be true"); + if (stringList(policy.integration.protectedBranches, "policy.integration.protectedBranches", errors, branchName) + && !policy.integration.protectedBranches.includes(policy.baseBranch)) errors.push("policy.integration.protectedBranches must include baseBranch"); + } + return errors; +} + +export function safeProjectPath(root, relative) { + if (typeof root !== "string" || !root || !relativePath(relative)) throw new Error(`Unsafe project path: ${String(relative)}`); + const absoluteRoot = resolve(root); + const rootInfo = lstatSync(absoluteRoot); + if (rootInfo.isSymbolicLink() || !rootInfo.isDirectory()) throw new Error(`Project root must be a directory, not a symlink: ${root}`); + let target = absoluteRoot; + for (const part of relative.split("/")) { + target = join(target, part); + let info; + try { + info = lstatSync(target); + } catch (error) { + if (error.code === "ENOENT") continue; + throw error; + } + if (info.isSymbolicLink()) throw new Error(`Symlink is not allowed in project path: ${relative}`); + } + return target; +} + +export function wrapperPaths(app, harnesses) { + if (!appName(app) || !Array.isArray(harnesses) || harnesses.some((name) => !HARNESSES.includes(name))) throw new Error("Invalid wrapper app or harnesses"); + const roots = []; + if (harnesses.includes("codex")) roots.push(".agents/skills"); + else if (harnesses.includes("cursor") || harnesses.includes("grokbot")) roots.push(".cursor/skills"); + for (const [harness, root] of [["claude", ".claude/skills"], ["opencode", ".opencode/skills"], ["pi", ".pi/skills"]]) { + if (harnesses.includes(harness)) roots.push(root); + } + return roots.map((root) => `${root}/verify-${app}/SKILL.md`); +} + +export function wrapperText(app, contractPath) { + if (!appName(app) || !relativePath(contractPath)) throw new Error("Invalid verification wrapper path or app"); + return `---\nname: verify-${app}\ndescription: Verify ${app} through the shared repository contract.\nmetadata:\n verification-contract: ${contractPath}\n---\n\n# Verify ${app}\n\nRead \`.harness/workflow.md\` from the repository root when present before the contract.\nResolve \`${contractPath}\` from the repository root, read it and its feature map,\nand follow the shared contract. Choose the available capability that can drive\nthe documented user path. Report an unavailable capability as a blocked step;\nnever replace required application evidence with a weaker check.\n\nKeep project facts in the canonical contract and feature map. This wrapper\ncontains no separate launch, driving, or evidence instructions.\n`; +} + +function readProjectFile(root, relative, errors) { + try { + const target = safeProjectPath(root, relative); + if (!lstatSync(target).isFile()) throw new Error("expected a regular file"); + return readFileSync(target, "utf8").replaceAll("\r\n", "\n"); + } catch (error) { + errors.push(`${relative}: ${error.message}`); + return undefined; + } +} + +function projectFiles(root, relative, errors, optional = false) { + const files = []; + let directory; + try { + directory = safeProjectPath(root, relative); + if (!lstatSync(directory).isDirectory()) throw new Error("expected a directory"); + } catch (error) { + if (!(optional && error.code === "ENOENT")) errors.push(`${relative}: ${error.message}`); + return files; + } + try { + for (const entry of readdirSync(directory, { withFileTypes: true }).sort((left, right) => left.name.localeCompare(right.name))) { + const path = `${relative}/${entry.name}`; + if (entry.isSymbolicLink()) errors.push(`${path}: symlinks are not allowed`); + else if (entry.isDirectory()) files.push(...projectFiles(root, path, errors)); + else if (entry.isFile()) files.push(path); + else errors.push(`${path}: expected a regular file or directory`); + } + } catch (error) { + errors.push(`${relative}: ${error.message}`); + } + return files; +} + +function markdownSections(text) { + const sections = []; + let fence; + for (const line of text.replace(//g, "").split("\n")) { + const marker = /^ {0,3}(`{3,}|~{3,})/.exec(line); + if (marker) { + if (!fence) fence = marker[1]; + else if (marker[1][0] === fence[0] && marker[1].length >= fence.length) fence = undefined; + sections.at(-1)?.body.push(line); + continue; + } + const heading = !fence && /^ {0,3}(#{1,6})\s+(.+?)\s*#*\s*$/.exec(line); + if (heading && heading[1].length <= 2) sections.push({ level: heading[1].length, title: heading[2], body: [] }); + else sections.at(-1)?.body.push(line); + } + return sections; +} + +function checkSections(text, relative, errors, feature = false) { + const sections = markdownSections(text); + const headings = sections.filter((section) => section.level === 2); + if (feature) { + const h1 = sections.find((section) => section.level === 1); + if (!h1 || sections[0] !== h1 || sections.filter((section) => section.level === 1).length !== 1 || !h1.body.join("\n").trim()) errors.push(`${relative}: feature needs one initial H1 and user-visible description`); + const expected = ["Sub-features", "How to get to it (user POV)", "Driving it with ", "Gotchas"]; + if (headings.length !== 4 || headings.some((section, index) => index === 2 ? !/^Driving it with \S.*$/.test(section.title) : section.title !== expected[index])) { + errors.push(`${relative}: feature must have exactly these four H2 sections in order: ${expected.join(", ")}`); + } + } else { + for (const title of CONTRACT_SECTIONS) { + if (headings.filter((section) => section.title === title).length !== 1) errors.push(`${relative}: requires one ## ${title} section`); + } + } + for (const section of headings) { + const body = section.body.join("\n"); + const instructions = body.replace(/^\s*(?:`{3,}|~{3,})[^\n]*$/gm, "").replace(/^\s*(?:#{1,6}\s.*|[-*+]\s*)$/gm, "").trim(); + if (!instructions || /\b(?:TODO|TBD)\b/i.test(body)) { + errors.push(`${relative}: ## ${section.title} must contain completed instructions, not empty content or TODO/TBD markers`); + } + } +} + +function markdownLinks(text) { + const clean = text.replace(//g, "").replace(/^ {0,3}(`{3,}|~{3,})[^\n]*\n[\s\S]*?^ {0,3}\1\s*$/gm, ""); + const definitions = new Map(); + for (const match of clean.matchAll(/^ {0,3}\[([^\]\n]+)\]:\s*(<[^>\n]+>|\S+)/gm)) definitions.set(match[1].toLowerCase(), match[2].replace(/^<|>$/g, "")); + const body = clean.replace(/^ {0,3}\[[^\]\n]+\]:[^\n]*$/gm, "").replace(/(`+)([^`]|(?!\1)`)*\1/g, ""); + const links = []; + for (const match of body.matchAll(/(?\n]+>|[^\s)]+)(?:\s+["'][^\n]*?["'])?\s*\)/g)) links.push(match[1].replace(/^<|>$/g, "")); + for (const match of body.matchAll(/(?/g, "").trim()) errors.push(".harness/workflow.md: shared workflow must be non-empty"); + const pointers = ["AGENTS.md"]; + if (policy.allowedHarnesses.includes("claude")) pointers.push("CLAUDE.md"); + if (policy.allowedHarnesses.some((harness) => harness === "cursor" || harness === "grokbot")) pointers.push(".cursor/rules/shared-verification.mdc"); + for (const path of pointers) { + const pointer = readProjectFile(root, path, errors); + if (pointer === undefined) continue; + if (!pointer.replace(//g, "").includes(".harness/workflow.md")) errors.push(`${path}: must reference .harness/workflow.md`); + if (path.endsWith(".mdc")) { + const frontmatter = /^\uFEFF?---\n([\s\S]*?)\n---(?:\n|$)/.exec(pointer); + const fields = [...(frontmatter?.[1] ?? "").matchAll(/^alwaysApply:\s*(.*)$/gm)]; + if (fields.length !== 1 || !/^true\s*(?:#.*)?$/.test(fields[0][1])) errors.push(`${path}: frontmatter must set alwaysApply: true`); + } + } + try { + for (const entry of readdirSync(safeProjectPath(root, canonicalRoot), { withFileTypes: true })) { + if (entry.isSymbolicLink()) errors.push(`${canonicalRoot}/${entry.name}: symlinks are not allowed`); + else if (entry.isDirectory() && !policy.verification.apps.includes(entry.name)) errors.push(`${canonicalRoot}/${entry.name}: unconfigured canonical app directory`); + } + } catch (error) { + errors.push(`${canonicalRoot}: ${error.message}`); + } + const expectedWrappers = new Map(); + for (const app of policy.verification.apps) { + const contractPath = `${canonicalRoot}/${app}/contract.md`; + const contract = readProjectFile(root, contractPath, errors); + if (contract !== undefined) checkSections(contract, contractPath, errors); + const files = projectFiles(root, `${canonicalRoot}/${app}`, errors); + const indexPath = `${canonicalRoot}/${app}/features/README.md`; + const index = readProjectFile(root, indexPath, errors); + const features = files.filter((path) => path.startsWith(`${canonicalRoot}/${app}/features/`) && path.endsWith(".md") && path !== indexPath); + if (features.length === 0) errors.push(`${canonicalRoot}/${app}/features: requires at least one documented feature`); + for (const path of features) { + const text = readProjectFile(root, path, errors); + if (text !== undefined) checkSections(text, path, errors, true); + } + if (index !== undefined) checkFeatureIndex(root, indexPath, index, features, errors); + for (const path of wrapperPaths(app, policy.allowedHarnesses)) expectedWrappers.set(path, wrapperText(app, contractPath)); + } + const found = new Set(); + const discoveries = new Map(); + for (const skillRoot of SKILL_ROOTS) { + const files = projectFiles(root, skillRoot, errors, true); + for (const path of files.filter((file) => posix.basename(file).toLowerCase() === "skill.md")) { + const text = readProjectFile(root, path, errors); + if (text === undefined) continue; + const name = skillName(text, path, errors); + const discoveryRoot = [".cursor/skills", ".agents/skills"].includes(skillRoot) ? ".cursor/skills + .agents/skills" : skillRoot; + const key = `${discoveryRoot}:${name}`; + if (discoveries.has(key)) errors.push(`${path}: duplicate discovered skill name ${name}; also present at ${discoveries.get(key)}`); + else discoveries.set(key, path); + if (!name.startsWith("verify-") && !posix.basename(posix.dirname(path)).startsWith("verify-")) continue; + found.add(path); + if (!expectedWrappers.has(path)) errors.push(`${path}: legacy or unconfigured verification skill; migrate facts into ${canonicalRoot} and keep only configured wrappers`); + else if (text !== expectedWrappers.get(path)) errors.push(`${path}: verification wrapper differs from the canonical generated wrapper`); + if (files.some((file) => file !== path && file.startsWith(`${posix.dirname(path)}/`))) errors.push(`${path}: verification wrappers may contain only SKILL.md; move supporting files into ${canonicalRoot}`); + const featureRoot = `${posix.dirname(path)}/features/`; + if (files.some((file) => file.startsWith(featureRoot))) errors.push(`${path}: duplicate feature map outside ${canonicalRoot}; migrate its facts before removing it`); + } + } + for (const path of expectedWrappers.keys()) { + if (!found.has(path)) errors.push(`${path}: required verification wrapper is missing`); + } + return errors; +} diff --git a/scripts/harness-policy.test.mjs b/scripts/harness-policy.test.mjs new file mode 100644 index 0000000..d470d7f --- /dev/null +++ b/scripts/harness-policy.test.mjs @@ -0,0 +1,340 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import test from "node:test"; +import { checkProject, safeProjectPath, validateHarnessPolicy, wrapperPaths, wrapperText } from "./harness-policy-lib.mjs"; + +const WEB_WRAPPER = `--- +name: verify-web +description: Verify web through the shared repository contract. +metadata: + verification-contract: .harness/verify/web/contract.md +--- + +# Verify web + +Read \`.harness/workflow.md\` from the repository root when present before the contract. +Resolve \`.harness/verify/web/contract.md\` from the repository root, read it and its feature map, +and follow the shared contract. Choose the available capability that can drive +the documented user path. Report an unavailable capability as a blocked step; +never replace required application evidence with a weaker check. + +Keep project facts in the canonical contract and feature map. This wrapper +contains no separate launch, driving, or evidence instructions. +`; + +const CONTRACT = `# Web verification + +## Launch + +Run \`node app.mjs --port 4182 --data ./scratch/run-12\` and record the child PID. + +## Doctor + +GET /health must identify the current commit and run-12 data directory. + +## Drive + +Use the feature map to create a note through the browser. + +## Evidence + +Capture the action and resulting page, then read the note through GET /notes. + +## Cleanup + +Stop the recorded child PID and remove this run's data after retaining evidence. + +## Isolation + +Allocate a distinct port and data directory per run. Never reuse another run's server. +`; + +const FEATURE = `# Create a note + +Save a note and reopen the saved result. + +## Sub-features + +Create and reopen a saved note. + +## How to get to it (user POV) + +Open /notes and choose New note. + +## Driving it with Playwright + +Fill Title with Release checklist, choose Save, and reopen the note. Confirm GET /notes contains Release checklist. + +## Gotchas + +The save notification appears before persistence completes. Wait for the note in the list. +`; + +function policy() { + return { + schemaVersion: 1, + baseBranch: "main", + allowedHarnesses: ["cursor", "grokbot", "codex", "claude", "opencode", "pi"], + workflows: { pstack: { revision: "a".repeat(40) }, mstack: { revision: "0.3.0" } }, + worktrees: { required: true, sharedCheckout: false }, + verification: { canonicalRoot: ".harness/verify", apps: ["web"], requiredCommands: ["node --test"] }, + integration: { mode: "pull-request", protectedBranches: ["main"], requireRebase: true }, + }; +} + +function write(root, path, content) { + mkdirSync(dirname(join(root, path)), { recursive: true }); + writeFileSync(join(root, path), content); +} + +function fixture(t) { + const directory = mkdtempSync(join(tmpdir(), "mstack-policy-test-")); + t.after(() => rmSync(directory, { recursive: true, force: true })); + const root = join(directory, "project"); + const config = policy(); + write(root, ".harness/workflow.md", "# Shared workflow\n\nRead .harness/policy.json before development and verification.\n"); + write(root, "AGENTS.md", "Read [.harness/workflow.md](.harness/workflow.md) before work.\n"); + write(root, "CLAUDE.md", "Read .harness/workflow.md before work.\n"); + write(root, ".cursor/rules/shared-verification.mdc", "---\nalwaysApply: true\n---\n\nRead .harness/workflow.md before work.\n"); + write(root, ".harness/verify/web/contract.md", CONTRACT); + write(root, ".harness/verify/web/features/README.md", "# Feature map\n\n- [Create a note](create-note.md)\n"); + write(root, ".harness/verify/web/features/create-note.md", FEATURE); + for (const skillRoot of [".agents", ".claude", ".opencode", ".pi"]) write(root, `${skillRoot}/skills/verify-web/SKILL.md`, WEB_WRAPPER); + return { root, directory, config, check: () => checkProject(root, config) }; +} + +test("a mixed-harness project shares canonical facts and uses one Cursor/Codex wrapper", (t) => { + const f = fixture(t); + assert.deepEqual(validateHarnessPolicy(f.config), []); + assert.deepEqual(f.check(), []); + assert.deepEqual(wrapperPaths("web", f.config.allowedHarnesses), [ + ".agents/skills/verify-web/SKILL.md", ".claude/skills/verify-web/SKILL.md", + ".opencode/skills/verify-web/SKILL.md", ".pi/skills/verify-web/SKILL.md", + ]); + assert.deepEqual(wrapperPaths("web", ["cursor", "grokbot"]), [".cursor/skills/verify-web/SKILL.md"]); + assert.deepEqual(wrapperPaths("web", ["grokbot", "codex"]), [".agents/skills/verify-web/SKILL.md"]); + assert.equal(wrapperText("web", ".harness/verify/web/contract.md"), WEB_WRAPPER); +}); + +test("workflow choices are independent of harness names and revisions are exact", () => { + const config = policy(); + config.workflows = { mstack: { revision: "v1.2.3-rc.1+test" } }; + assert.deepEqual(validateHarnessPolicy(config), []); + config.workflows = { pstack: { revision: "1.0.0" } }; + assert.deepEqual(validateHarnessPolicy(config), []); + for (const revision of ["latest", "main", "abc1234", "^1.2.3", "1.2", "01.2.3", "1.2.3-01", 123, null]) { + config.workflows.pstack.revision = revision; + assert.match(validateHarnessPolicy(config).join("\n"), /revision must pin/, String(revision)); + } +}); + +test("malformed JSON objects produce diagnostics rather than escaping exceptions", () => { + for (const value of [null, [], true, 12, "policy"]) assert.ok(validateHarnessPolicy(value).length); + for (const key of ["workflows", "worktrees", "verification", "integration"]) { + for (const value of [null, [], false, "value"]) { + const config = policy(); + config[key] = value; + assert.ok(validateHarnessPolicy(config).length, `${key}: ${value}`); + assert.ok(checkProject("missing-root", config).length); + } + } + const config = policy(); + config.workflows.pstack = null; + config.verification.apps = [null, "web", "web"]; + config.allowedHarnesses = ["cursor", 1, "unsupported"]; + config.integration.protectedBranches = null; + const errors = validateHarnessPolicy(config).join("\n"); + assert.match(errors, /workflows.pstack must be an object/); + assert.match(errors, /apps contains a duplicate: web/); + assert.match(errors, /protectedBranches must be a non-empty array/); +}); + +test("schema rejects missing and unknown fields and unsafe collaboration modes", () => { + const config = policy(); + delete config.baseBranch; + config.workflowAliases = {}; + config.worktrees.required = false; + config.worktrees.sharedCheckout = true; + config.integration.mode = "direct-push"; + config.integration.requireRebase = false; + config.verification.requiredCommands = []; + config.workflows.mstack.allowLatest = true; + const errors = validateHarnessPolicy(config).join("\n"); + for (const pattern of [/baseBranch is required/, /workflowAliases is not supported/, /required must be true/, /sharedCheckout must be false/, /mode must be pull-request/, /requireRebase must be true/, /requiredCommands must be a non-empty array/, /allowLatest is not supported/]) assert.match(errors, pattern); +}); + +test("portable project paths reject Unix escapes, Windows drive paths, UNC and device names on every OS", (t) => { + const f = fixture(t); + for (const path of ["../outside", "/tmp/verify", "C:/outside", "C:outside", "\\\\server\\share", "//server/share", ".harness/../outside", ".harness\\verify", "CON", ".harness/NUL.md", "verify.", "verify ", "a\u0000b", ".git/verify", ".Git/config", "vendor/.GIT/verify"]) { + f.config.verification.canonicalRoot = path; + assert.match(f.check().join("\n"), /canonicalRoot/, path); + assert.throws(() => safeProjectPath(f.root, path), /Unsafe project path/, path); + } + for (const path of [".agents/skills/verify", ".cursor/skills", ".claude", ".opencode/skills/verify", ".pi/skills", ".AGENTS/skills/verify", ".CURSOR/SKILLS", ".CLAUDE"]) { + f.config.verification.canonicalRoot = path; + assert.match(f.check().join("\n"), /outside harness skill roots/, path); + } + assert.equal(safeProjectPath(f.root, ".harness/new/contract.md"), join(f.root, ".harness/new/contract.md")); +}); + +test("required branch protection and valid Git branch names are checked", () => { + const config = policy(); + config.integration.protectedBranches = ["release"]; + assert.match(validateHarnessPolicy(config).join("\n"), /must include baseBranch/); + for (const branch of ["../main", "main.lock", "-main", "main..next", "main@{1}", "main~1", "@", "main/", "main next"]) { + config.baseBranch = branch; + assert.match(validateHarnessPolicy(config).join("\n"), /valid Git branch name/, branch); + } +}); + +test("missing facts, empty sections and explicit placeholders fail validation", (t) => { + const f = fixture(t); + write(f.root, ".harness/verify/web/contract.md", CONTRACT.replace("## Isolation", "## Other").replace("GET /health must identify the current commit and run-12 data directory.", "").replace("Use the feature map to create a note through the browser.", "TODO")); + rmSync(join(f.root, ".harness/verify/web/features/create-note.md")); + const errors = f.check().join("\n"); + assert.match(errors, /requires one ## Isolation section/); + assert.match(errors, /## Doctor must contain completed instructions/); + assert.match(errors, /## Drive must contain completed instructions/); + assert.match(errors, /requires at least one documented feature/); + assert.match(errors, /broken or unsafe link create-note.md/); + rmSync(join(f.root, ".harness/verify/web/contract.md")); + assert.match(f.check().join("\n"), /contract.md:.*ENOENT/); +}); + +test("feature files require the documented user-facing sections and index coverage", (t) => { + const f = fixture(t); + write(f.root, ".harness/verify/web/features/search.md", FEATURE.replace("## Gotchas", "## Internals")); + let errors = f.check().join("\n"); + assert.match(errors, /search.md: feature must have exactly these four H2 sections/); + assert.match(errors, /feature is not indexed: .harness\/verify\/web\/features\/search.md/); + write(f.root, ".harness/verify/web/features/search.md", FEATURE); + write(f.root, ".harness/verify/web/features/README.md", "# Map\n\n[Create](create-note.md)\n\n[Search][search]\n\n[search]: search.md\n"); + assert.deepEqual(f.check(), []); +}); + +test("heading examples inside code fences cannot fulfill required sections", (t) => { + const f = fixture(t); + write(f.root, ".harness/verify/web/contract.md", `# Contract\n\n\`\`\`md\n${CONTRACT}\n\`\`\`\n`); + assert.match(f.check().join("\n"), /requires one ## Launch section/); + write(f.root, ".harness/verify/web/contract.md", CONTRACT.replace("Use the feature map to create a note through the browser.", "```sh\n```")); + assert.match(f.check().join("\n"), /## Drive must contain completed instructions/); +}); + +test("unreferenced link definitions and code examples do not count as indexed features", (t) => { + const f = fixture(t); + write(f.root, ".harness/verify/web/features/README.md", "# Map\n\n`[Create](create-note.md)`\n\n[unused]: create-note.md\n"); + assert.match(f.check().join("\n"), /feature is not indexed/); +}); + +test("index links must resolve safely, including encoded traversal and Windows paths", (t) => { + const f = fixture(t); + for (const link of ["missing.md", "../../../../../../outside.md", "%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/outside.md", "C:/outside.md", "C%3a/outside.md", "\\\\server\\share", "file:///etc/passwd", "%00.md"]) { + write(f.root, ".harness/verify/web/features/README.md", `# Map\n\n[Create](create-note.md)\n\n[Unsafe](${link})\n`); + assert.match(f.check().join("\n"), /broken or unsafe link/, link); + } + write(f.root, ".harness/verify/web/features/README.md", "# Map\n\n[Create](create-note.md#gotchas)\n\n[Contract](../contract.md)\n\n[External](https://example.org/reference)\n"); + assert.deepEqual(f.check(), []); +}); + +test("wrapper CRLF is portable but changed instructions or metadata fail", (t) => { + const f = fixture(t); + const path = ".agents/skills/verify-web/SKILL.md"; + write(f.root, path, WEB_WRAPPER.replaceAll("\n", "\r\n")); + assert.deepEqual(f.check(), []); + write(f.root, path, `${WEB_WRAPPER}\n## Launch\n\nRun an alternate server.\n`); + assert.match(f.check().join("\n"), /verification wrapper differs/); + write(f.root, path, WEB_WRAPPER.replace("verification-contract: .harness/verify/web/contract.md", "verification-contract: old/contract.md")); + assert.match(f.check().join("\n"), /verification wrapper differs/); + rmSync(join(f.root, path)); + assert.match(f.check().join("\n"), /required verification wrapper is missing/); +}); + +test("Cursor rejects a duplicate .cursor and .agents discovery even with identical bytes", (t) => { + const f = fixture(t); + write(f.root, ".cursor/skills/verify-web/SKILL.md", WEB_WRAPPER); + const errors = f.check().join("\n"); + assert.match(errors, /duplicate discovered skill name verify-web/); + assert.match(errors, /legacy or unconfigured verification skill/); +}); + +test("renaming a legacy directory cannot hide its frontmatter verification name", (t) => { + const f = fixture(t); + write(f.root, ".cursor/skills/renamed/SKILL.md", "---\nname: 'verify-web'\ndescription: Legacy workflow.\n---\n\n## Launch\n\nRun the old app.\n"); + write(f.root, ".cursor/skills/renamed/features/old.md", FEATURE); + const errors = f.check().join("\n"); + assert.match(errors, /renamed\/SKILL.md: legacy or unconfigured verification skill/); + assert.match(errors, /duplicate discovered skill name verify-web/); + assert.match(errors, /duplicate feature map outside/); + write(f.root, ".cursor/skills/renamed/SKILL.md", '---\nname: "verify-\\u0077eb"\n---\n'); + assert.match(f.check().join("\n"), /duplicate discovered skill name verify-web/); +}); + +test("extra maps beside a valid wrapper and unregistered verification apps fail", (t) => { + const f = fixture(t); + write(f.root, ".agents/skills/verify-web/features/old.md", FEATURE); + write(f.root, ".pi/skills/verify-unused/SKILL.md", "---\nname: verify-unused\n---\n\n## Drive\n\nOld instructions.\n"); + const errors = f.check().join("\n"); + assert.match(errors, /duplicate feature map outside/); + assert.match(errors, /verify-unused\/SKILL.md: legacy or unconfigured verification skill/); +}); + +test("symlinks cannot redirect canonical reads, discovery scans or future writes", (t) => { + const f = fixture(t); + const outside = join(f.directory, "outside"); + mkdirSync(outside); + const canonical = join(f.root, ".harness/verify/web"); + renameSync(canonical, join(outside, "web")); + symlinkSync(join(outside, "web"), canonical, "junction"); + assert.match(f.check().join("\n"), /[Ss]ymlink/); + assert.throws(() => safeProjectPath(f.root, ".harness/verify/web/new.md"), /Symlink/); + rmSync(canonical); + renameSync(join(outside, "web"), canonical); + mkdirSync(join(outside, "skills")); + mkdirSync(join(f.root, ".cursor"), { recursive: true }); + symlinkSync(join(outside, "skills"), join(f.root, ".cursor/skills"), "junction"); + assert.match(f.check().join("\n"), /[Ss]ymlink/); + assert.equal(readFileSync(join(f.root, ".harness/verify/web/contract.md"), "utf8"), CONTRACT); +}); + +test("standalone checking a missing project reports filesystem diagnostics", () => { + assert.match(checkProject(join(tmpdir(), "mstack-policy-no-such-project", "missing"), policy()).join("\n"), /ENOENT/); +}); + +test("shared workflow and each configured harness entry point are required", (t) => { + const f = fixture(t); + write(f.root, "AGENTS.md", "\n"); + rmSync(join(f.root, "CLAUDE.md")); + write(f.root, ".cursor/rules/shared-verification.mdc", "---\nalwaysApply: false\n---\n\nRead .harness/workflow.md\n"); + write(f.root, ".harness/workflow.md", "\n"); + const errors = f.check().join("\n"); + assert.match(errors, /AGENTS.md: must reference/); + assert.match(errors, /CLAUDE.md:.*ENOENT/); + assert.match(errors, /frontmatter must set alwaysApply: true/); + assert.match(errors, /shared workflow must be non-empty/); +}); + +test("entry pointers accept project prose and require only configured harnesses", (t) => { + const f = fixture(t); + f.config.allowedHarnesses = ["codex"]; + rmSync(join(f.root, "CLAUDE.md")); + rmSync(join(f.root, ".cursor/rules/shared-verification.mdc")); + for (const root of [".claude", ".opencode", ".pi"]) rmSync(join(f.root, root), { recursive: true }); + write(f.root, "AGENTS.md", "# Project rules\n\nOur workflow is in `.harness/workflow.md`. Read it first.\n"); + assert.deepEqual(f.check(), []); +}); + +test("unconfigured canonical apps cannot retain a second feature map", (t) => { + const f = fixture(t); + write(f.root, ".harness/verify/old-web/features/create.md", FEATURE); + assert.match(f.check().join("\n"), /old-web: unconfigured canonical app directory/); +}); + +test("feature links cannot point at Git internals with any casing", (t) => { + const f = fixture(t); + for (const link of ["../../../../.git/config", "../../../../.Git/config"]) { + write(f.root, ".harness/verify/web/features/README.md", `# Map\n\n[Create](create-note.md)\n\n[Git](${link})\n`); + assert.match(f.check().join("\n"), /broken or unsafe link.*Unsafe project path/); + } +}); diff --git a/scripts/harness-project.mjs b/scripts/harness-project.mjs new file mode 100644 index 0000000..c77be02 --- /dev/null +++ b/scripts/harness-project.mjs @@ -0,0 +1,282 @@ +import { + existsSync, readFileSync, writeFileSync, mkdirSync, realpathSync, lstatSync, + openSync, closeSync, rmSync, +} from "node:fs"; +import { dirname, join, relative, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { execFileSync, spawnSync } from "node:child_process"; +import { createHash, randomUUID } from "node:crypto"; +import { hostname } from "node:os"; +import { validateHarnessPolicy, checkProject, wrapperText, wrapperPaths, safeProjectPath as projectPath } from "./harness-policy-lib.mjs"; + +export { projectPath }; + +const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const version = JSON.parse(readFileSync(join(packageRoot, "package.json"), "utf8")).version; +const allHarnesses = ["cursor", "grokbot", "codex", "claude", "opencode", "pi"]; +const jsonText = (value) => JSON.stringify(value, null, 2) + "\n"; +const digest = (value) => createHash("sha256").update(value).digest("hex"); +const normalized = (value) => value.replaceAll("\r\n", "\n"); + +export function readPolicy(root) { + const policy = JSON.parse(readFileSync(projectPath(root, ".harness/policy.json"), "utf8")); + const errors = validateHarnessPolicy(policy); + if (errors.length) throw new Error(errors.join("\n")); + return policy; +} + +function git(root, ...args) { + return execFileSync("git", ["-C", root, ...args], { + encoding: "utf8", windowsHide: true, stdio: ["ignore", "pipe", "pipe"], + }).trim(); +} + +function requireRepositoryRoot(root) { + const top = realpathSync.native(git(root, "rev-parse", "--show-toplevel")); + if (relative(realpathSync.native(root), top) !== "") throw new Error("--root must be the Git worktree root"); +} + +function writePlan(root, files) { + const pending = []; + for (const [path, content] of files) { + const absolute = projectPath(root, path); + if (existsSync(absolute)) { + if (normalized(readFileSync(absolute, "utf8")) !== normalized(content)) throw new Error("Existing file differs; reconcile it before retrying: " + path); + } else pending.push([absolute, content, path]); + } + for (const [absolute, content] of pending) { + mkdirSync(dirname(absolute), { recursive: true }); + writeFileSync(absolute, content, { flag: "wx" }); + } + return pending.map(([, , path]) => path); +} + +function pointerFile(root, path, marker, block) { + const absolute = projectPath(root, path); + const before = existsSync(absolute) ? readFileSync(absolute, "utf8") : ""; + if (before.includes(marker)) { + if (!normalized(before).includes(normalized(block))) throw new Error("Existing project pointer differs: " + path); + return { path, before, after: before }; + } + return { path, before, after: before + (before.endsWith("\n") || !before ? "" : "\n") + "\n" + block + "\n" }; +} + +export function initProject(root, options) { + requireRepositoryRoot(root); + const harnesses = options.harness ?? allHarnesses; + const apps = options.app ?? []; + if (!apps.length || !options.check?.length) throw new Error("init requires --app and at least one --check command"); + const policy = { + schemaVersion: 1, + baseBranch: options.base ?? "main", + allowedHarnesses: harnesses, + workflows: { ...(options.pstack ? { pstack: { revision: options.pstack } } : {}), mstack: { revision: version } }, + worktrees: { required: true, sharedCheckout: false }, + verification: { canonicalRoot: ".harness/verify", apps, requiredCommands: options.check }, + integration: { mode: "pull-request", protectedBranches: [options.base ?? "main"], requireRebase: true }, + }; + const errors = validateHarnessPolicy(policy); + if (errors.length) throw new Error(errors.join("\n")); + const workflow = readFileSync(join(packageRoot, "scripts", "harness-workflow.md"), "utf8"); + const files = [ + [".harness/policy.json", jsonText(policy)], + [".harness/workflow.md", workflow], + [".harness/check.mjs", readFileSync(join(packageRoot, "scripts/harness-check.mjs"), "utf8")], + [".harness/harness-policy-lib.mjs", readFileSync(join(packageRoot, "scripts/harness-policy-lib.mjs"), "utf8")], + ]; + const marker = ""; + const block = marker + "\nBefore development or verification, read [.harness/workflow.md](.harness/workflow.md) and follow the shared project contract."; + const pointers = [pointerFile(root, "AGENTS.md", marker, block)]; + if (harnesses.includes("claude")) pointers.push(pointerFile(root, "CLAUDE.md", marker, block)); + if (harnesses.includes("cursor") || harnesses.includes("grokbot")) { + files.push([".cursor/rules/shared-verification.mdc", + "---\ndescription: Shared project verification and ownership rules\nalwaysApply: true\n---\n\nRead .harness/workflow.md before development or verification. It defines this project's canonical verification target, evidence, and handoff rules for native pstack and mstack.\n"]); + } + pointers.push(pointerFile(root, ".gitignore", "# mstack local runs", + "# mstack local runs\n.harness/runs/\n.harness/scratch/")); + for (const app of apps) projectPath(root, ".harness/verify/" + app); + const created = writePlan(root, files); + for (const { path, before, after } of pointers) { + if (after === before) continue; + const absolute = projectPath(root, path); + const current = existsSync(absolute) ? readFileSync(absolute, "utf8") : ""; + if (current !== before) throw new Error("Instruction file changed during init: " + path); + writeFileSync(absolute, after, existsSync(absolute) ? {} : { flag: "wx" }); + created.push(path); + } + for (const app of apps) mkdirSync(projectPath(root, ".harness/verify/" + app), { recursive: true }); + return { status: "needs-contract", created, apps, next: "Write and prove each contract and feature map, then run wrappers and check." }; +} + +export function installWrappers(root, options) { + let policy; + if (existsSync(projectPath(root, ".harness/policy.json"))) { + if (options.app || options.harness) throw new Error("wrappers uses the existing policy; change it before overriding apps or harnesses"); + policy = readPolicy(root); + } else { + if (!options.app?.length || !options.harness?.length) throw new Error("Without a policy, wrappers requires --app and --harness"); + policy = { + schemaVersion: 1, baseBranch: "main", allowedHarnesses: options.harness, + workflows: { mstack: { revision: version } }, worktrees: { required: true, sharedCheckout: false }, + verification: { canonicalRoot: ".harness/verify", apps: options.app, requiredCommands: ["true"] }, + integration: { mode: "pull-request", protectedBranches: ["main"], requireRebase: true }, + }; + const errors = validateHarnessPolicy(policy); + if (errors.length) throw new Error(errors.join("\n")); + } + const files = policy.verification.apps.flatMap((app) => { + const contract = policy.verification.canonicalRoot + "/" + app + "/contract.md"; + if (!existsSync(projectPath(root, contract))) throw new Error("Write the canonical contract first: " + contract); + return wrapperPaths(app, policy.allowedHarnesses).map((path) => [path, wrapperText(app, contract)]); + }); + return { created: writePlan(root, files) }; +} + +function fetchedBase(root, policy, baseRef) { + const suffix = "/" + policy.baseBranch; + if (typeof baseRef !== "string" || !baseRef.endsWith(suffix) || baseRef.startsWith("-")) { + throw new Error("--base-ref must name the fetched target branch, such as origin/" + policy.baseBranch); + } + const fullRef = git(root, "rev-parse", "--symbolic-full-name", baseRef); + if (!fullRef.startsWith("refs/remotes/") || !fullRef.endsWith(suffix) || + fullRef.slice("refs/remotes/".length, -suffix.length).length === 0) { + throw new Error("--base-ref must resolve to a remote-tracking target branch"); + } + return git(root, "rev-parse", "--verify", fullRef + "^{commit}"); +} + +export function preflight(root, policy, options) { + requireRepositoryRoot(root); + const errors = checkProject(root, policy); + if (errors.length) throw new Error(errors.join("\n")); + if (!policy.allowedHarnesses.includes(options.harness)) throw new Error("Declare an allowed --harness"); + if (!Object.hasOwn(policy.workflows, options.workflow ?? "")) throw new Error("Declare a configured --workflow"); + if (options.revision !== policy.workflows[options.workflow].revision) throw new Error("Declared --revision differs from the pinned workflow revision"); + const branch = git(root, "branch", "--show-current"); + if (!branch || policy.integration.protectedBranches.includes(branch)) throw new Error("Use a dedicated task branch, not a protected or detached HEAD"); + const gitDir = realpathSync.native(git(root, "rev-parse", "--absolute-git-dir")); + const common = realpathSync.native(resolve(root, git(root, "rev-parse", "--git-common-dir"))); + if (relative(common, gitDir) === "") throw new Error("Use a dedicated linked worktree for development"); + if (git(root, "status", "--porcelain=v1", "--untracked-files=all")) throw new Error("Commit or preserve changes before preflight; the worktree is dirty"); + const baseRef = options["base-ref"]; + const base = fetchedBase(root, policy, baseRef); + const head = git(root, "rev-parse", "HEAD"); + try { git(root, "merge-base", "--is-ancestor", base, head); } + catch { throw new Error("Rebase or integrate the current target branch before verification"); } + return { branch, head, base, baseRef, harness: options.harness, workflow: options.workflow, revision: options.revision }; +} + +function mappedFeature(policy, featureId) { + if (typeof featureId !== "string") throw new Error("Declare a mapped app/feature"); + const [app, feature, extra] = featureId.split("/"); + if (extra !== undefined || !policy.verification.apps.includes(app) || !/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(feature)) { + throw new Error("Feature must identify a mapped app/feature"); + } + return policy.verification.canonicalRoot + "/" + app + "/features/" + feature + ".md"; +} + +function artifact(root, path) { + const absolute = projectPath(root, path); + if (!lstatSync(absolute).isFile()) throw new Error("Evidence must be a regular file: " + path); + const content = readFileSync(absolute); + if (!content.length) throw new Error("Evidence is empty: " + path); + return { path, sha256: digest(content) }; +} + +function shellCommand(command) { + if (process.platform !== "win32") return { command: "/bin/sh", args: ["-c", command] }; + const script = "$ErrorActionPreference = 'Stop'\n" + command + + "\nif (-not $?) { if ($LASTEXITCODE) { exit $LASTEXITCODE }; exit 1 }\n"; + return { + command: "powershell.exe", + args: ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", + "-EncodedCommand", Buffer.from(script, "utf16le").toString("base64")], + }; +} + +export function recordVerification(root, policy, options) { + const snapshot = preflight(root, policy, options); + if (!options.feature || !options.evidence?.length) throw new Error("record requires --feature app/feature and --evidence"); + const mappedPath = mappedFeature(policy, options.feature); + if (!existsSync(projectPath(root, mappedPath))) throw new Error("Unknown mapped feature: " + mappedPath); + const evidence = options.evidence.map((path) => artifact(root, path)); + const runPath = ".harness/runs/" + randomUUID(); + const runDir = projectPath(root, runPath); + mkdirSync(runDir, { recursive: true }); + const lockDir = join(git(root, "rev-parse", "--absolute-git-dir"), "mstack-verification.lock"); + try { mkdirSync(lockDir); } + catch { throw new Error("Another verification owns this worktree; inspect " + lockDir); } + const commands = []; + try { + writeFileSync(join(lockDir, "owner.json"), jsonText({ pid: process.pid, host: hostname(), runPath })); + for (const [index, command] of policy.verification.requiredCommands.entries()) { + const logPath = runPath + "/" + (index + 1) + ".log"; + const log = openSync(projectPath(root, logPath), "wx"); + let result; + try { + const invocation = shellCommand(command); + result = spawnSync(invocation.command, invocation.args, { + cwd: root, stdio: ["ignore", log, log], windowsHide: true, + }); + } finally { closeSync(log); } + commands.push({ command, exitCode: result.status, signal: result.signal, + error: result.error?.message ?? null, log: logPath, sha256: digest(readFileSync(projectPath(root, logPath))) }); + if (result.status !== 0 || result.error || result.signal) break; + } + const unchanged = git(root, "branch", "--show-current") === snapshot.branch && + git(root, "rev-parse", "HEAD") === snapshot.head && + git(root, "rev-parse", "--verify", snapshot.baseRef + "^{commit}") === snapshot.base && + git(root, "status", "--porcelain=v1", "--untracked-files=all") === ""; + const evidenceUnchanged = evidence.every((item) => { + try { return artifact(root, item.path).sha256 === item.sha256; } catch { return false; } + }); + const receipt = { + schemaVersion: 1, ...snapshot, recordedAt: new Date().toISOString(), + policyDigest: digest(normalized(readFileSync(projectPath(root, ".harness/policy.json"), "utf8"))), + feature: options.feature, evidence, commands, runtimeReview: "required", + status: unchanged && evidenceUnchanged && commands.length === policy.verification.requiredCommands.length && + commands.every((item) => item.exitCode === 0 && !item.error && !item.signal) ? "PASS" : "FAIL", + }; + const receiptPath = runPath + "/receipt.json"; + writeFileSync(projectPath(root, receiptPath), jsonText(receipt), { flag: "wx" }); + return { status: receipt.status, receipt: receiptPath, runtimeReview: receipt.runtimeReview }; + } finally { + rmSync(join(lockDir, "owner.json"), { force: true }); + rmSync(lockDir, { recursive: true }); + } +} + +export function checkReceipt(root, policy, path) { + const receipt = JSON.parse(readFileSync(projectPath(root, path), "utf8")); + if (!receipt || receipt.schemaVersion !== 1 || receipt.status !== "PASS" || receipt.runtimeReview !== "required") { + throw new Error("Receipt must record passing commands and require runtime review"); + } + if (typeof receipt.recordedAt !== "string" || !Number.isFinite(Date.parse(receipt.recordedAt))) throw new Error("Receipt recordedAt is invalid"); + if (!existsSync(projectPath(root, mappedFeature(policy, receipt.feature)))) throw new Error("Receipt feature is not mapped"); + if (receipt.branch !== git(root, "branch", "--show-current") || !receipt.branch || + policy.integration.protectedBranches.includes(receipt.branch)) throw new Error("Receipt branch differs from the current task branch"); + if (receipt.head !== git(root, "rev-parse", "HEAD")) throw new Error("Receipt is stale for this HEAD"); + if (receipt.policyDigest !== digest(normalized(readFileSync(projectPath(root, ".harness/policy.json"), "utf8")))) throw new Error("Receipt policy changed"); + if (receipt.base !== fetchedBase(root, policy, receipt.baseRef)) throw new Error("Receipt base changed"); + try { git(root, "merge-base", "--is-ancestor", receipt.base, receipt.head); } + catch { throw new Error("Receipt head does not include the target base"); } + if (git(root, "status", "--porcelain=v1", "--untracked-files=all")) throw new Error("Receipt cannot describe a dirty worktree"); + if (!policy.allowedHarnesses.includes(receipt.harness) || + !Object.hasOwn(policy.workflows, receipt.workflow ?? "") || + receipt.revision !== policy.workflows[receipt.workflow].revision) throw new Error("Receipt workflow declaration differs from policy"); + if (!Array.isArray(receipt.commands) || receipt.commands.length !== policy.verification.requiredCommands.length) { + throw new Error("Receipt does not cover all required commands"); + } + for (const [index, command] of receipt.commands.entries()) { + if (command?.command !== policy.verification.requiredCommands[index] || command.exitCode !== 0 || command.signal || command.error) { + throw new Error("Receipt command failed or differs from policy"); + } + if (digest(readFileSync(projectPath(root, command.log))) !== command.sha256) throw new Error("Command log changed: " + command.log); + } + if (!Array.isArray(receipt.evidence) || !receipt.evidence.length) throw new Error("Receipt evidence is missing"); + for (const item of receipt.evidence) { + if (!item || typeof item !== "object") throw new Error("Invalid evidence record"); + if (artifact(root, item.path).sha256 !== item.sha256) throw new Error("Evidence changed: " + item.path); + } + return { status: "PASS", runtimeReview: "required", receipt: path }; +} diff --git a/scripts/harness-project.test.mjs b/scripts/harness-project.test.mjs new file mode 100644 index 0000000..ae8cc4f --- /dev/null +++ b/scripts/harness-project.test.mjs @@ -0,0 +1,374 @@ +import assert from "node:assert/strict"; +import { execFileSync, spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, existsSync, rmSync, symlinkSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import test from "node:test"; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const cli = join(repo, "scripts/check-harness-policy.mjs"); +const version = JSON.parse(readFileSync(join(repo, "package.json"), "utf8")).version; +const pstack = "7366ac128bdf95f45e6734f412b49a4031800169"; + +function git(root, ...args) { + return execFileSync("git", ["-C", root, ...args], { encoding: "utf8", windowsHide: true }).trim(); +} + +function fixture(t) { + const temporary = mkdtempSync(join(tmpdir(), "mstack-project-")); + t.after(() => rmSync(temporary, { recursive: true, force: true })); + const root = join(temporary, "repo"); + mkdirSync(root); + git(root, "init", "-b", "main"); + git(root, "config", "user.name", "Fixture"); + git(root, "config", "user.email", "fixture@example.invalid"); + git(root, "config", "core.autocrlf", "false"); + writeFileSync(join(root, "README.md"), "Fixture app\n"); + writeFileSync(join(root, "AGENTS.md"), "Keep existing project instructions.\n"); + git(root, "add", "."); + git(root, "commit", "-m", "Initial application"); + git(root, "update-ref", "refs/remotes/origin/main", "HEAD"); + return { root, temporary }; +} + +function run(root, args, expected = 0) { + const result = spawnSync(process.execPath, [cli, ...args, "--root", root], { encoding: "utf8", windowsHide: true }); + assert.equal(result.status, expected, result.stdout + result.stderr); + return result; +} + +function init(root, checks = ["node verify.mjs"]) { + return run(root, ["init", "--app", "web", "--pstack", pstack, ...checks.flatMap((command) => ["--check", command])]); +} + +function contracts(root) { + const app = join(root, ".harness/verify/web"); + mkdirSync(join(app, "features"), { recursive: true }); + writeFileSync(join(app, "contract.md"), "# Web verification\n\n" + + "## Launch\nRun node verify.mjs in this worktree.\n\n" + + "## Doctor\nCheck this process exits successfully.\n\n" + + "## Drive\nCreate a note through the repository CLI.\n\n" + + "## Evidence\nRecord the action and read the saved note from disk.\n\n" + + "## Cleanup\nDelete only the fixture's temporary note directory.\n\n" + + "## Isolation\nEach process uses its own temporary directory.\n"); + writeFileSync(join(app, "features/README.md"), "# Features\n\n[Create a note](create-note.md)\n"); + writeFileSync(join(app, "features/create-note.md"), "# Create a note\n\nA user saves a note and reads it back.\n\n" + + "## Sub-features\nCreate and persist a note.\n\n" + + "## How to get to it (user POV)\nRun the note command from a terminal.\n\n" + + "## Driving it with Node\nRun node verify.mjs and inspect the saved text.\n\n" + + "## Gotchas\nUse a fresh temporary directory on each run.\n"); + writeFileSync(join(root, "verify.mjs"), + "import {mkdtempSync,writeFileSync,readFileSync,rmSync} from 'node:fs';\n" + + "import {tmpdir} from 'node:os'; import {join} from 'node:path'; import assert from 'node:assert/strict';\n" + + "const dir=mkdtempSync(join(tmpdir(),'note-')); try { const file=join(dir,'note.txt'); writeFileSync(file,'hello'); " + + "assert.equal(readFileSync(file,'utf8'),'hello'); console.log('note saved and read back'); } finally {rmSync(dir,{recursive:true});}\n"); +} + +function ready(t, checks) { + const fixtureState = fixture(t); + init(fixtureState.root, checks); + contracts(fixtureState.root); + run(fixtureState.root, ["wrappers"]); + run(fixtureState.root, ["check"]); + git(fixtureState.root, "add", "."); + git(fixtureState.root, "commit", "-m", "Adopt shared contract"); + git(fixtureState.root, "update-ref", "refs/remotes/origin/main", "HEAD"); + const worktree = join(fixtureState.temporary, "worktree"); + git(fixtureState.root, "worktree", "add", "-b", "task/verify", worktree); + return { ...fixtureState, worktree }; +} + +function declaration(workflow = "mstack", baseRef = "origin/main") { + return ["--harness", workflow === "pstack" ? "cursor" : "codex", "--workflow", workflow, + "--revision", workflow === "pstack" ? pstack : version, "--base-ref", baseRef]; +} + +function record(root, expected = 0) { + const evidencePath = ".harness/runs/action.txt"; + mkdirSync(join(root, ".harness/runs"), { recursive: true }); + writeFileSync(join(root, evidencePath), "Observed note creation and disk readback.\n"); + return JSON.parse(run(root, ["record", ...declaration(), "--feature", "web/create-note", "--evidence", evidencePath], expected).stdout); +} + +test("init preserves user instructions, repeats safely, and leaves unfinished contracts failing", (t) => { + const { root } = fixture(t); + const first = JSON.parse(init(root).stdout); + assert.equal(first.status, "needs-contract"); + assert.ok(readFileSync(join(root, "AGENTS.md"), "utf8").startsWith("Keep existing project instructions.\n")); + assert.ok(existsSync(join(root, ".cursor/rules/shared-verification.mdc"))); + const before = readFileSync(join(root, "AGENTS.md"), "utf8"); + assert.deepEqual(JSON.parse(init(root).stdout).created, []); + assert.equal(readFileSync(join(root, "AGENTS.md"), "utf8"), before); + const missing = run(root, ["check"], 1); + assert.match(missing.stderr, /contract\.md/); +}); + +test("Windows path aliases identify the same repository and linked worktree", { skip: process.platform !== "win32" }, (t) => { + const { root, temporary } = fixture(t); + const shortPath = (path) => execFileSync("powershell.exe", ["-NoProfile", "-NonInteractive", "-Command", + "(New-Object -ComObject Scripting.FileSystemObject).GetFolder($env:MSTACK_TEST_ALIAS_PATH).ShortPath"], { + encoding: "utf8", windowsHide: true, env: { ...process.env, MSTACK_TEST_ALIAS_PATH: path }, + }).trim(); + const alias = root.toUpperCase(); + assert.notEqual(alias, root); + const initialized = JSON.parse(init(shortPath(root)).stdout); + assert.equal(initialized.status, "needs-contract"); + assert.deepEqual(JSON.parse(init(alias).stdout).created, []); + contracts(root); + run(alias, ["wrappers"]); + git(root, "add", "."); + git(root, "commit", "-m", "Adopt contract through a Windows path alias"); + git(root, "update-ref", "refs/remotes/origin/main", "HEAD"); + const worktree = join(temporary, "worktree"); + git(root, "worktree", "add", "-b", "task/windows-alias", worktree); + const checked = JSON.parse(run(worktree.toUpperCase(), ["preflight", ...declaration()]).stdout); + assert.equal(checked.branch, "task/windows-alias"); + assert.equal(checked.head, git(worktree, "rev-parse", "HEAD")); + assert.equal(JSON.parse(run(shortPath(worktree), ["preflight", ...declaration()]).stdout).head, checked.head); + const nested = join(worktree, "nested"); + mkdirSync(nested); + assert.match(run(nested.toUpperCase(), ["init", "--app", "web", "--pstack", pstack, "--check", "node verify.mjs"], 1).stderr, /--root must be the Git worktree root/); +}); + +test("wrappers converge to shared facts and preserve conflicting legacy instructions", (t) => { + const { root } = fixture(t); + init(root); + contracts(root); + run(root, ["wrappers"]); + assert.equal(existsSync(join(root, ".cursor/skills/verify-web")), false); + assert.ok(existsSync(join(root, ".agents/skills/verify-web/SKILL.md"))); + assert.deepEqual(JSON.parse(run(root, ["wrappers"]).stdout).created, []); + const wrapper = join(root, ".agents/skills/verify-web/SKILL.md"); + writeFileSync(wrapper, "Existing user verification; do not replace.\n"); + run(root, ["wrappers"], 1); + assert.equal(readFileSync(wrapper, "utf8"), "Existing user verification; do not replace.\n"); + run(root, ["check"], 1); +}); + +test("Cursor can adopt mstack without declaring native pstack", (t) => { + const { root } = fixture(t); + run(root, ["init", "--harness", "cursor", "--app", "web", "--check", "node verify.mjs"]); + contracts(root); + run(root, ["wrappers"]); + run(root, ["check"]); + assert.deepEqual(JSON.parse(readFileSync(join(root, ".harness/policy.json"), "utf8")).workflows, + { mstack: { revision: version } }); + assert.ok(existsSync(join(root, ".cursor/skills/verify-web/SKILL.md"))); +}); + +test("CLI rejects missing, duplicate, and unknown arguments without writes", (t) => { + const { root } = fixture(t); + for (const args of [["init", "--app"], ["init", "--app", "web", "--bogus", "value"], ["check", "--root", root]]) { + run(root, args, 1); + assert.equal(existsSync(join(root, ".harness")), false); + } +}); + +test("the policy CLI runs through a linked entry while module imports stay silent", (t) => { + const { root, temporary } = fixture(t); + let linkedEntry; + if (process.platform === "win32") { + const linkedDirectory = join(temporary, "linked-mstack"); + symlinkSync(repo, linkedDirectory, "junction"); + linkedEntry = join(linkedDirectory, "scripts/check-harness-policy.mjs"); + } else { + linkedEntry = join(temporary, "mstack-policy"); + symlinkSync(cli, linkedEntry, "file"); + } + const invoke = (...args) => spawnSync(process.execPath, [linkedEntry, ...args], { + cwd: root, encoding: "utf8", windowsHide: true, + }); + const help = invoke("--help"); + assert.equal(help.status, 0, help.stderr); + assert.match(help.stdout, /Usage: mstack-policy/); + const rejected = invoke("init", "--app", "web", "--unknown", "value", "--root", root); + assert.equal(rejected.status, 1, rejected.stdout + rejected.stderr); + assert.match(rejected.stderr, /Unknown option: --unknown/); + assert.equal(existsSync(join(root, ".harness")), false); + const initialized = invoke("init", "--app", "web", "--check", "node verify.mjs", "--root", root); + assert.equal(initialized.status, 0, initialized.stderr); + assert.equal(JSON.parse(initialized.stdout).status, "needs-contract"); + assert.deepEqual(JSON.parse(readFileSync(join(root, ".harness/policy.json"), "utf8")).verification.apps, ["web"]); + assert.ok(existsSync(join(root, ".harness/check.mjs"))); + const importing = spawnSync(process.execPath, ["--input-type=module", "--eval", + `await import(${JSON.stringify(pathToFileURL(linkedEntry).href)})`], { + cwd: root, encoding: "utf8", windowsHide: true, + }); + assert.equal(importing.status, 0, importing.stderr); + assert.equal(importing.stdout, ""); + assert.equal(importing.stderr, ""); + const stdinImport = spawnSync(process.execPath, ["--input-type=module", "-"], { + input: `await import(${JSON.stringify(pathToFileURL(linkedEntry).href)})`, + cwd: root, encoding: "utf8", windowsHide: true, + }); + assert.equal(stdinImport.status, 0, stdinImport.stderr); + assert.equal(stdinImport.stdout, ""); + assert.equal(stdinImport.stderr, ""); +}); + +test("preflight accepts either workflow on one branch and rejects stale base, dirty source, and primary checkout", (t) => { + const { root, worktree } = ready(t); + run(worktree, ["preflight", ...declaration()]); + run(worktree, ["preflight", ...declaration("pstack")]); + run(root, ["preflight", ...declaration()], 1); + writeFileSync(join(worktree, "README.md"), "uncommitted work"); + assert.match(run(worktree, ["preflight", ...declaration()], 1).stderr, /dirty/); + git(worktree, "restore", "README.md"); + writeFileSync(join(root, "README.md"), "target changed"); + git(root, "add", "."); + git(root, "commit", "-m", "Advance target"); + git(root, "update-ref", "refs/remotes/origin/main", "HEAD"); + assert.match(run(worktree, ["preflight", ...declaration()], 1).stderr, /target branch/); +}); + +test("record runs actual commands, binds evidence and SHAs, and rejects stale or altered receipts", (t) => { + const { worktree } = ready(t); + const evidencePath = ".harness/runs/action.txt"; + mkdirSync(join(worktree, ".harness/runs")); + writeFileSync(join(worktree, evidencePath), "Observed note creation and disk readback.\n"); + const result = JSON.parse(run(worktree, ["record", ...declaration(), "--feature", "web/create-note", "--evidence", evidencePath]).stdout); + assert.equal(result.status, "PASS"); + assert.equal(result.runtimeReview, "required"); + const receipt = JSON.parse(readFileSync(join(worktree, result.receipt), "utf8")); + assert.equal(receipt.commands[0].command, "node verify.mjs"); + assert.equal(receipt.commands[0].exitCode, 0); + assert.match(readFileSync(join(worktree, receipt.commands[0].log), "utf8"), /note saved and read back/); + run(worktree, ["check", "--receipt", result.receipt]); + const policyFile = join(worktree, ".harness/policy.json"); + const policyText = readFileSync(policyFile, "utf8"); + git(worktree, "config", "core.autocrlf", "true"); + rmSync(policyFile); + // Update the index stat cache with the checkout so newline conversion leaves a clean worktree. + git(worktree, "checkout-index", "--force", "--index", "--", ".harness/policy.json"); + assert.equal(readFileSync(policyFile, "utf8"), policyText.replaceAll("\r\n", "\n").replaceAll("\n", "\r\n")); + assert.equal(git(worktree, "diff", "--cached", "--", ".harness/policy.json"), ""); + assert.equal(git(worktree, "status", "--porcelain=v1", "--untracked-files=all"), ""); + run(worktree, ["check", "--receipt", result.receipt]); + writeFileSync(join(worktree, evidencePath), "different claim"); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, /Evidence changed/); + git(worktree, "commit", "--allow-empty", "-m", "New revision"); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, /stale/); +}); + +test("failed commands produce a failing receipt and cannot satisfy handoff", (t) => { + const { worktree } = ready(t, ["node -e \"process.exit(7)\""]); + mkdirSync(join(worktree, ".harness/runs")); + writeFileSync(join(worktree, ".harness/runs/action.txt"), "attempt"); + const result = JSON.parse(run(worktree, ["record", ...declaration(), "--feature", "web/create-note", + "--evidence", ".harness/runs/action.txt"], 1).stdout); + assert.equal(result.status, "FAIL"); + const receipt = JSON.parse(readFileSync(join(worktree, result.receipt), "utf8")); + assert.equal(receipt.commands[0].exitCode, 7); + run(worktree, ["check", "--receipt", result.receipt], 1); +}); + +test("local branches cannot substitute for a fetched base in preflight or a receipt", (t) => { + const { worktree } = ready(t); + git(worktree, "branch", "pretend/main"); + const rejected = run(worktree, ["preflight", ...declaration("mstack", "pretend/main")], 1); + assert.match(rejected.stderr, /remote|fetched|target branch/i); + const result = record(worktree); + const receiptPath = join(worktree, result.receipt); + const receipt = JSON.parse(readFileSync(receiptPath, "utf8")); + receipt.baseRef = "pretend/main"; + writeFileSync(receiptPath, JSON.stringify(receipt)); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, /remote|fetched|base|target branch/i); + run(worktree, ["preflight", ...declaration("mstack", "refs/remotes/origin/main")]); +}); + +test("receipt validation rejects missing and inconsistent handoff metadata", (t) => { + const { worktree } = ready(t); + const result = record(worktree); + const receiptPath = join(worktree, result.receipt); + const receipt = JSON.parse(readFileSync(receiptPath, "utf8")); + for (const [field, pattern] of [["feature", /feature/i], ["branch", /branch/i], ["base", /base/i], ["baseRef", /base|target branch/i], ["recordedAt", /recordedAt/i]]) { + const missing = { ...receipt }; + delete missing[field]; + writeFileSync(receiptPath, JSON.stringify(missing)); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, pattern, field); + } + for (const [field, value, pattern] of [ + ["feature", "web/missing", /feature/i], ["branch", "main", /branch/i], + ["baseRef", "HEAD", /base|target branch/i], ["recordedAt", "not a date", /recordedAt/i], + ]) { + writeFileSync(receiptPath, JSON.stringify({ ...receipt, [field]: value })); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, pattern, field); + } + writeFileSync(receiptPath, JSON.stringify(receipt)); + const checked = JSON.parse(run(worktree, ["check", "--receipt", result.receipt]).stdout); + assert.equal(checked.runtimeReview, "required"); +}); + +test("init and wrapper retries preserve valid CRLF files without reporting conflicts", (t) => { + const { root } = fixture(t); + const created = JSON.parse(init(root).stdout).created; + const snapshots = new Map(); + for (const path of created) { + const content = readFileSync(join(root, path), "utf8").replaceAll("\r\n", "\n").replaceAll("\n", "\r\n"); + writeFileSync(join(root, path), content); + snapshots.set(path, content); + } + assert.deepEqual(JSON.parse(init(root).stdout).created, []); + for (const [path, content] of snapshots) assert.equal(readFileSync(join(root, path), "utf8"), content, path); + contracts(root); + const wrappers = JSON.parse(run(root, ["wrappers"]).stdout).created; + for (const path of wrappers) { + const content = readFileSync(join(root, path), "utf8").replaceAll("\n", "\r\n"); + writeFileSync(join(root, path), content); + snapshots.set(path, content); + } + assert.deepEqual(JSON.parse(run(root, ["wrappers"]).stdout).created, []); + for (const [path, content] of snapshots) assert.equal(readFileSync(join(root, path), "utf8"), content, path); + run(root, ["check"]); +}); + +test("a successful command that changes captured evidence produces a failing receipt", (t) => { + const { worktree } = ready(t); + writeFileSync(join(worktree, "verify.mjs"), + "import {writeFileSync} from 'node:fs';\n" + + "writeFileSync('.harness/runs/action.txt', 'New evidence from this command.\\n');\n" + + "console.log('Evidence was replaced.');\n"); + git(worktree, "add", "verify.mjs"); + git(worktree, "commit", "-m", "Regenerate evidence during verification"); + const result = record(worktree, 1); + assert.equal(result.status, "FAIL"); + const receipt = JSON.parse(readFileSync(join(worktree, result.receipt), "utf8")); + assert.equal(receipt.commands[0].exitCode, 0); + assert.equal(readFileSync(join(worktree, ".harness/runs/action.txt"), "utf8"), "New evidence from this command.\n"); + assert.match(run(worktree, ["check", "--receipt", result.receipt], 1).stderr, /passing commands/); +}); + +test("the exported checker runs independently and rejects broken project structure", (t) => { + const { root } = fixture(t); + init(root); + contracts(root); + run(root, ["wrappers"]); + const exported = () => spawnSync(process.execPath, [".harness/check.mjs"], { cwd: root, encoding: "utf8", windowsHide: true }); + const passing = exported(); + assert.equal(passing.status, 0, passing.stderr); + assert.match(passing.stdout, /structure passed/); + writeFileSync(join(root, "AGENTS.md"), "Project rules without the shared workflow pointer.\n"); + const rejected = exported(); + assert.equal(rejected.status, 1, rejected.stdout); + assert.match(rejected.stderr, /AGENTS.md: must reference .harness\/workflow.md/); +}); + +test("an existing worktree verification lock is preserved until its owner is inspected and released", (t) => { + const { worktree } = ready(t); + const lockDir = join(git(worktree, "rev-parse", "--absolute-git-dir"), "mstack-verification.lock"); + mkdirSync(lockDir); + const owner = JSON.stringify({ pid: 2147483647, host: "previous-verifier", runPath: ".harness/runs/prior-run" }); + writeFileSync(join(lockDir, "owner.json"), owner); + mkdirSync(join(worktree, ".harness/runs")); + writeFileSync(join(worktree, ".harness/runs/action.txt"), "Observed action.\n"); + const rejected = run(worktree, ["record", ...declaration(), "--feature", "web/create-note", "--evidence", ".harness/runs/action.txt"], 1); + assert.match(rejected.stderr, /Another verification owns this worktree; inspect/); + assert.equal(readFileSync(join(lockDir, "owner.json"), "utf8"), owner); + assert.equal(existsSync(lockDir), true); + rmSync(join(lockDir, "owner.json")); + rmSync(lockDir, { recursive: true }); + const result = record(worktree); + assert.equal(result.status, "PASS"); + assert.equal(existsSync(lockDir), false); +}); diff --git a/scripts/harness-workflow.md b/scripts/harness-workflow.md new file mode 100644 index 0000000..a5e5487 --- /dev/null +++ b/scripts/harness-workflow.md @@ -0,0 +1,79 @@ +# Shared project workflow + +Read `.harness/policy.json` before a task. It records accepted workflow revisions, +verification apps, commands, and the base branch. User instructions and the active +Harness's execution permissions still apply. + +## Create and maintain verification + +Native pstack and mstack both write shared facts under +`verification.canonicalRoot//`. Its `contract.md` owns Launch, Doctor, Drive, +Evidence, Cleanup, and Isolation. Its `features/` is the only feature map. +These project paths apply when an installed create or maintain skill suggests a +Harness-local directory or limits edits to that directory. Describe drivers by +required capability and observable result. + +Before creating a contract, inventory existing verification skills in all project +skill roots. Reconcile differences against the running app. Move helpers, fixtures, +and maps together, repair links and working-directory assumptions, then replace +legacy definitions with generated wrappers in the same change. Remove obsolete +maps after confirming all their features are preserved. Point existing Benny +configuration and saved automation prompts at the canonical map and wrapper too. + +Run `mstack-policy wrappers --root .` after writing the contract. Wrappers contain +pointers, not another map. Cursor discovers `.agents/skills` and its native skills, +so do not add duplicate names there. Run `mstack-policy check --root .` before +committing. A structural pass does not prove app behavior. Drive the app using +the contract and retain evidence. + +## Work independently + +During initial adoption or migration, create the isolated task worktree first, +then finish the canonical contract before running preflight. Do not weaken the +checker or mark an unfinished contract as verified to bypass this bootstrap step. + +Give each active writer a dedicated branch and linked worktree. Read-only agents +may share an immutable snapshot. Worktrees share Git refs; only the branch owner +may rewrite or push it. Hand off explicitly before switching writers. Use separate +ports, profiles, and data stores. If a resource cannot be isolated, name its owner +and serialize use through the existing project mechanism. + +Before writing, fetch the target and run `mstack-policy preflight --root .` with +`--harness`, `--workflow`, `--revision`, and `--base-ref origin/`. +Compare the declared revision with the installed plugin or package. The command +cannot inspect every plugin store or prove that another agent is idle. + +## Verify and hand off + +Commit the implementation, then run `mstack-policy record --root .` with the +preflight flags, `--feature `, and `--evidence ` for each +artifact. It executes policy commands and records exits, logs, head, base, +workflow declaration, feature, and artifact digests under `.harness/runs/`. +PASS means those commands passed on unchanged source. Runtime review remains +required. Explain what artifacts prove and disclose unreachable paths. Unit tests +alone cannot justify an app verification claim. + +Only one command-recording run owns a worktree at a time. If a crashed process +leaves `mstack-verification.lock` in its worktree Git directory, inspect its +`owner.json` and confirm that process and its children have stopped before +removing that lock. A retry does not take over an existing owner automatically. + +Hand off the branch, commit, base, PR when present, unfinished work, and receipt. +The receiver compares Git state and runs `mstack-policy check --receipt `. +Evidence and receipts travel together. Private Harness history is optional. +Remove completed plans and scratch work. Preserve review evidence until accepted. + +## Integrate through the forge + +Use PRs against the configured protected branch. Configure required CI checks +and reviews in the forge; policy JSON does not enable protection. CI runs the +repository verification commands as well as `node .harness/check.mjs`. Integrate the +current target and repeat affected checks. Use a merge queue or strict up-to-date +requirement to close the verification-to-merge race. Merge, deploy, or publish +only when authorized. Follow project release rules; npm is not required here. + +The init command exports the structural checker to `.harness/check.mjs` and +`.harness/harness-policy-lib.mjs`. Commit both and run `node .harness/check.mjs` +from the repository root in CI, without downloading a workflow at check time. +Review checker upgrades with policy changes. Keep forge protections on the +workflow and checker files so a PR cannot silently remove its own gate. diff --git a/scripts/skill-integrity.test.mjs b/scripts/skill-integrity.test.mjs index 3db4647..31f48a1 100644 --- a/scripts/skill-integrity.test.mjs +++ b/scripts/skill-integrity.test.mjs @@ -52,11 +52,10 @@ test("meta-agent points to an existing complete entry skill", () => { test("portable skill files contain no broken replacement artifacts", () => { const forbidden = [ /prethe current harness/i, - /(?:poteto-mode|setup-pstack|poteto-agent)/i, + /(?:setup-pstack|poteto-agent)/i, /worker type:\s*generalPurpose/i, /Comment Sicko/i, /Task subagent/i, - /cursor/i, /application support\/the current harness/i, ]; const skillDirectories = readdirSync(skillsRoot, { withFileTypes: true }) @@ -73,6 +72,10 @@ test("portable skill files contain no broken replacement artifacts", () => { for (const path of files) { const content = readFileSync(path, "utf8"); const normalized = content.replaceAll("`", "").replace(/\s+/g, " "); + const compatibilityReference = path === join(skillsRoot, "create-verification-skill", "references", "harness-paths.md"); + const creator = path === join(skillsRoot, "create-verification-skill", "SKILL.md"); + if (!compatibilityReference) assert.doesNotMatch(normalized, /poteto-mode/i, path); + if (!compatibilityReference && !creator) assert.doesNotMatch(normalized, /cursor/i, path); for (const pattern of forbidden) { assert.doesNotMatch(normalized, pattern, `${path} contains ${pattern}`); } diff --git a/scripts/sync-upstream.mjs b/scripts/sync-upstream.mjs index 22e24d3..6294514 100644 --- a/scripts/sync-upstream.mjs +++ b/scripts/sync-upstream.mjs @@ -457,6 +457,7 @@ const manifest = { source: pstack.repository, commit: pstack.commit, ...(previousManifest?.canonicalSkills ? { canonicalSkills: previousManifest.canonicalSkills } : {}), + ...(previousManifest?.adaptedArtifacts ? { adaptedArtifacts: previousManifest.adaptedArtifacts } : {}), artifacts: Object.fromEntries( [...grouped.keys()].map((name) => [ name, diff --git a/scripts/validate.mjs b/scripts/validate.mjs index ec3a843..9013768 100644 --- a/scripts/validate.mjs +++ b/scripts/validate.mjs @@ -59,7 +59,6 @@ if (JSON.stringify(actual) !== JSON.stringify(EXPECTED_SKILLS)) { } const forbidden = [ - /\.cursor[\\/]skills/i, /disable-model-invocation:/i, /\bTask tool\b/i, /\brun_in_background\b/i, @@ -145,6 +144,10 @@ function walk(directory) { for (const file of walk(skillsRoot)) { const content = readFileSync(file, "utf8"); + const compatibilityReference = file === join(skillsRoot, "create-verification-skill", "references", "harness-paths.md"); + if (!compatibilityReference && /\.cursor[\\/]skills/i.test(content)) { + errors.push(`${relative(repoRoot, file)} contains a native skill path outside the compatibility reference`); + } if (file.endsWith(".md") && content.includes(String.fromCodePoint(0xfffd))) { errors.push(`${relative(repoRoot, file)} contains a replacement character`); } diff --git a/skills/create-verification-skill/SKILL.md b/skills/create-verification-skill/SKILL.md index 338b1ca..c07a468 100644 --- a/skills/create-verification-skill/SKILL.md +++ b/skills/create-verification-skill/SKILL.md @@ -1,18 +1,25 @@ --- name: create-verification-skill -description: Generate a durable project-local verification skill that launches and drives the real app, captures evidence, and cleans up safely. Use when the user explicitly asks to create or bootstrap a reusable verification or control workflow, not for a one-time verification pass. +description: Create or migrate a shared project verification contract and discoverable skill wrappers, prove one real app path, and retain evidence. Use for a reusable verification or control workflow, not a one-time verification pass. license: MIT --- # Create a verification skill -Create a project-local skill that another agent can read cold and use to prove -the real application behavior. Do not write a generic test plan. Record the -commands, stable handles, health checks, evidence, and cleanup that work in this -repository. +Create one verification contract that agents in different Harnesses can use to +prove the same application behavior. Store project facts in +`.harness/verify//`, with thin `verify-` skill wrappers for discovery. +Record commands and observations from this repository, not a generic test plan. ## 1. Interview the repository +Read `AGENTS.md`, `.harness/policy.json`, and `.harness/workflow.md` when present. +Follow the shared project workflow before editing. Inventory canonical contracts +and existing `verify-*` or control skills across every project skill root in +[harness-paths.md](./references/harness-paths.md). Follow each wrapper's +`metadata.verification-contract` path from the repository root. Several wrappers +pointing to one contract are one target, not competing verification definitions. + Answer what you can from code and documentation before asking the user: - **Surface:** Identify the primary user surface and any secondary surfaces. @@ -29,16 +36,22 @@ Answer what you can from code and documentation before asking the user: If the checkout does not run as documented, fix or precisely report that base problem before recording a workflow against it. -## 2. Choose the project skill root +## 2. Write or migrate the canonical contract + +Reuse the app ID and canonical path already registered in the project policy. +For a new app, use `.harness/verify//contract.md`. Keep its feature map in +`features/` and its owned scripts in `helpers/` beside the contract. -Use the current harness's project-level skill directory from -[harness-paths.md](./references/harness-paths.md). Name the generated skill -`verify-` and create `/verify-/SKILL.md`. +For existing verification definitions, follow +[the migration procedure](./references/harness-paths.md). +Reconcile conflicting instructions against the running app before choosing a +canonical version. A copied legacy map is not evidence of correct behavior. -The generated frontmatter must contain `name` and a precise `description`. Its -body must include: +The contract must include these H2 sections: - **Launch:** Exact start command, readiness signal, ownership marker, and teardown. +- **Isolation:** Instance-specific ports, profiles, and data directories, or an + explicit exclusive-use procedure for resources that cannot be separated. - **Doctor:** A read-only check that confirms the right instance, build, port, data directory, and auth are healthy. - **Drive:** Real commands and stable selectors from this repository. Prefer @@ -51,6 +64,13 @@ body must include: never proof artifacts. Never kill by process name alone. - **Helpers:** Explain every shipped helper and its exact invocation. +State the working directory for commands and resolve helper paths from the +repository root. Keep app facts, selectors, fixtures, assertions, and cleanup in +this directory. Describe required driver capabilities without naming the host +agent's browser or desktop tool. A repository-owned Playwright, HTTP, PTY, or +control script is portable and may appear by its exact name. If a capability is +unavailable, report that path as blocked instead of substituting a weaker check. + ## 3. Seed the feature map Create `features/README.md` and one file for each of the three to five most @@ -59,7 +79,19 @@ important user-visible features. Use the contract in Each feature records every user entry point, exact driving steps, observable end state, and known traps. -## 4. Prove the generated skill +Do not copy the map into a Harness skill root or an automation directory. Other +consumers must reference this index and load its linked feature files. + +## 4. Render the discovery wrappers + +Use [the wrapper rules](./references/harness-paths.md). Each +wrapper contains a repository-relative `metadata.verification-contract` pointer +and capability adaptation only. Launch commands, feature facts, evidence +requirements, and helpers stay in the canonical directory. Cursor discovers +`.agents/skills/`, so a shared Codex and Cursor project needs one neutral wrapper +there instead of two same-name definitions. + +## 5. Prove the generated workflow Run the generated instructions end to end: @@ -69,12 +101,17 @@ Run the generated instructions end to end: 4. Capture the named evidence and verify any side effect. 5. Clean up and confirm the evidence still exists. +Start from each generated wrapper and confirm it resolves the same contract. +Run the project policy check when a policy exists. A static wrapper check proves +discovery structure, not another Harness's runtime capabilities. + Fix every failed instruction and repeat its affected step. A workflow that has not driven the real application once is a draft, not a deliverable. After every failed iteration, run the generated cleanup, confirm its owned processes, ports, profiles, and scratch state are released, then retry. -## 5. Hand off +## 6. Hand off -Report the generated path, the feature proved, the evidence paths, and any -remaining surface that could not be exercised. Do not claim untested paths. +Report the canonical path, generated wrappers, the feature proved, evidence +paths, and any surface that could not be exercised. For migrations, account for +every retired definition and updated consumer. Do not claim untested paths. diff --git a/skills/create-verification-skill/references/feature-map-example/README.md b/skills/create-verification-skill/references/feature-map-example/README.md index 4865069..3d900a1 100644 --- a/skills/create-verification-skill/references/feature-map-example/README.md +++ b/skills/create-verification-skill/references/feature-map-example/README.md @@ -1,9 +1,10 @@ # Notes verification map -This worked example describes a fictional Notes app and its `control-notes` -driver. The commands show the precision a real feature map needs; they are -not tools shipped by mstack. Replace the app, driver, paths, and seeded data -when writing your project's map. +This worked example describes a fictional Notes app and its repository-owned +`control-notes` driver. The commands are not tools shipped by mstack. Replace +the app, driver, paths, and seeded data with observed project behavior. Save the +real index at `.harness/verify//features/README.md`, beside its feature files. +Every Harness wrapper and automation reads that same index. ## Baseline preconditions @@ -28,11 +29,14 @@ exactly these four H2 sections: 1. `Sub-features` 2. `How to get to it (user POV)` -3. `Driving it with ` +3. `Driving it with ` 4. `Gotchas` Keep implementation details out. Record user paths, stable handles, required -state, literal commands, and observable proof. +state, literal commands, and observable proof. Name a repository-owned driver +or a capability such as a browser with accessible selectors. Keep installed +Harness tool names out of the map. Record which driver actually ran in the +evidence instead. ## Features diff --git a/skills/create-verification-skill/references/harness-paths.md b/skills/create-verification-skill/references/harness-paths.md index a6e286a..3a3ab78 100644 --- a/skills/create-verification-skill/references/harness-paths.md +++ b/skills/create-verification-skill/references/harness-paths.md @@ -1,26 +1,92 @@ -# Project skill paths and drivers +# Verification paths and drivers -Use the path for the active harness. Do not generate copies for every harness -unless the user explicitly wants project-local copies for all supported -harnesses. +Project facts live once in `.harness/verify//contract.md`, `features/`, and +`helpers/`. A project skill root contains only a `verify-/SKILL.md` wrapper. +Workflow skills such as `meta-mode` and `poteto-mode` stay in their installed +plugin or user directories. -| Harness | Project skill root | Typical user-level root | +## Render wrappers + +When `.harness/policy.json` exists, run `mstack-policy wrappers --root ` +using the project's pinned mstack package. The command renders wrappers for the +Harnesses declared in the policy. Run `mstack-policy check --root ` after +rendering. + +Without a policy, run `mstack-policy wrappers --root --app --harness +` after writing the canonical contract. Repeat `--harness` for each +requested Harness. This creates discovery wrappers without adopting a team +policy or requiring a pstack installation. + +Resolve `mstack-policy` from the project's exact mstack package version, using +`npx --package @3metajun/mstack@ mstack-policy` when needed. For an +unreleased development checkout, use +`node /scripts/check-harness-policy.mjs`. Do not assume an older +installed package provides this command. + +The wrapper frontmatter has a `name`, a precise `description`, and this pointer: + +```yaml +metadata: + verification-contract: .harness/verify/web/contract.md +``` + +Resolve the pointer from the repository root. The wrapper reads the shared +project workflow when present, the contract, and the relevant feature files. +It selects an available driver that satisfies the contract. It contains no +independent commands, selectors, fixtures, feature map, or evidence rules. +`compatibility` and a Harness name in prose are documentation, not a discovery +filter. + +| Harness | Project wrapper root | Typical user skill root | | --- | --- | --- | | Codex | `.agents/skills/` | `~/.agents/skills/` | +| Cursor or Grok Bot | `.cursor/skills/`, or the shared `.agents/skills/` wrapper when Codex is also configured | Installed pstack or mstack plugin | | Claude Code | `.claude/skills/` | `~/.claude/skills/` | | OpenCode | `.opencode/skills/` | `~/.config/opencode/skills/` | | pi | `.pi/skills/` | `~/.pi/agent/skills/` | -Driver names vary by installation. Discover capabilities before writing the -workflow: +Cursor reads `.agents/skills/` as well as `.cursor/skills/`. When both Codex and +Cursor or Grok Bot are configured, generate only the shared `.agents` wrapper +for that app. The other roots receive the same neutral wrapper text as needed. +Do not rely on directory precedence to choose between conflicting definitions. + +## Choose a driver + +Discover capabilities before proving the contract: + +- For a web UI, use the attached browser or the repository's Playwright or + Cypress setup when it supports the specified user actions and observations. +- For a desktop app, use the host's desktop-control capability or the + repository's automation harness. +- For a CLI or TUI, use a PTY, expect script, or isolated terminal session. +- For an API or service, use the repository's integration client or an HTTP + client with structured assertions. + +Record the actual driver and any missing capabilities in the run evidence. +Keep tool-specific setup in its installed driver skill. A unit test or HTTP +response cannot replace a required UI interaction. + +## Migrate existing verification -- For a web UI, prefer the harness's attached collaborative browser. If none is - available, use the repository's Playwright or Cypress setup. -- For a desktop app, use the host's desktop-control capability or a - repository-provided automation harness. -- For a CLI or TUI, use a PTY, expect script, or an isolated terminal session. -- For an API or service, use the repository's integration client or a plain - HTTP client with structured assertions. +1. Inventory every `verify-*` definition and feature map in project skill roots, + automation configuration, and repository documentation. Resolve wrappers + before counting targets. +2. Compare their launch steps, feature coverage, helpers, evidence, and cleanup. + Check differing claims against source and the running app. Report unresolved + differences instead of choosing authority by directory location. +3. Move the reconciled contract, map, fixtures, and helpers into the canonical + app directory. Rewrite relative links and helper invocations for that path. +4. Point all consumers, including maintenance automations and Benny's + `control.feature_map_path`, at the canonical contract or feature index. +5. Prove the moved instructions and each changed helper against the app. Keep + the resulting artifacts and account for any untested feature. +6. Replace the retired definitions with generated wrappers and remove duplicate + maps in the same PR. Review each deletion. The CLI diagnoses conflicts and + does not choose, merge, or silently delete legacy project facts. +7. Run the project policy check and review the diff for stale paths. Retain no + alternate active definition after migration. -Record the capability actually present. Do not name a tool that the next agent -cannot discover or invoke from the same harness. +Plan files and run artifacts are scratch. Delete completed plans. Keep evidence +in the contract's retained artifact location and run receipts in ignored +`.harness/runs/`. Publish required review artifacts through the team's existing +CI or artifact system. diff --git a/skills/maintain-verification-skill/SKILL.md b/skills/maintain-verification-skill/SKILL.md index d4bd9c7..f5fbbfb 100644 --- a/skills/maintain-verification-skill/SKILL.md +++ b/skills/maintain-verification-skill/SKILL.md @@ -1,38 +1,84 @@ --- name: maintain-verification-skill -description: "Periodic pass that keeps a project's verification skill and feature map honest: parallel source readers per feature, one live session driving every feature, at most one PR of proven corrections. Use for /maintain-verification-skill or \"audit the verify skill\"." +description: "Audit a project's canonical verification contract and feature map with parallel source readers and one live coordinator, then ship proven corrections. Use for /maintain-verification-skill or an audit of a reusable verification workflow." --- -# Maintain a verification skill +# Maintain a verification contract -A feature map rots the moment the app changes. This skill is the upkeep loop for a skill generated by `/create-verification-skill` (or any project-local verification skill with a feature map). The unit of rigor is the feature, not every sentence: cover every feature file from source and exercise every feature live, without terminalising every bullet. +Cover every mapped feature from source and exercise every feature live. Update +one canonical definition regardless of which Harness runs this pass. ## Outcomes -Pick one, and say which: +Report one outcome: -- **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. -- **changed** — one PR ships proven doc, harness, or map corrections. -- **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. +- `clean`: Every feature has source and live coverage, with no corrections to ship. +- `changed`: One PR contains proven contract, helper, or map corrections. +- `blocked`: Coverage or delivery could not finish. Name the missing prerequisite. ## Edit scope -Only edit the verification skill's own directory (its SKILL.md, features/, and any harness scripts it owns). Never edit product code during a run: a behavior the map describes that the app no longer does is either doc drift (fix the map) or a product regression (report it, don't paper over it in docs). +Edit the target's canonical `.harness/verify//` directory, including its +contract, feature map, and owned helpers. Regenerate affected discovery wrappers +with `mstack-policy wrappers` when their pointers or supported Harnesses change. +Keep project facts out of wrapper directories. Never edit product code during +this pass. An incorrect map is doc drift. A product regression is a finding to +report, not a reason to redefine expected behavior. ## Pass -0. **Locate the target.** Find the verification skill to maintain: the project-local skill whose body has launch/drive sections and a feature map (usually `.harness/skills/verify-*/`). Several candidates → ask which one; none → stop and point at `/create-verification-skill` instead of inventing a target. - -1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory. - -2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe. - -3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. - -4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Exercise every feature at least once, and hold three invariants the whole pass, whatever the failure: (1) never drive an instance you haven't health-checked since it last did something surprising — doctor before first drive, doctor on each fresh session where sessions are the unit, doctor again after any failed drive, and where doctor can't see the failure (a wedged UI state on a healthy process), reset to a known state or relaunch rather than hoping; (2) evidence captured so far survives every cleanup, checked at its named location, not assumed; (3) nothing a drive started outlives that drive's usefulness — failed-iteration residue is cleaned whether the session is stuck, exited, or shared (for a shared instance, clean the residue, not the instance). A doctor failure caused by skill drift is drift: fix it under edit scope and retry once — restart whatever the fix invalidated, nothing more — before calling the pass `blocked`. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). - -5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. - -6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. - -Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. +0. **Locate the target.** Read `AGENTS.md` and `.harness/workflow.md` when present. + Use `.harness/policy.json` to find registered apps, then read their canonical + contracts and feature indexes. Resolve a discovered wrapper's + `metadata.verification-contract` from the repository root. Wrappers with the + same target are one app. Without a policy, inspect `.harness/verify/` directly. + Infer the target from the request when possible. Ask only when apps remain + ambiguous. For legacy definitions, use `/create-verification-skill`'s + migration procedure before claiming shared maintenance. When no target + exists, report that creation is needed. + +1. **Check the index.** Read the feature README and compare its links with the + sibling files. Fix missing, duplicate, and dead entries. + +2. **Read source in parallel.** Assign one read-only subagent per feature. Each + returns a feature summary, source entry points, likely drift with citations, + and one live recipe. Children never drive the app or edit files. + +3. **Reconcile.** Require a result for every feature. Combine overlapping + recipes into as few app states as practical. Check cited drift against source. + Inspect recent changes for unmapped user paths, citing a concrete source path + for each addition. + +4. **Run the live pass.** The coordinator owns all driving. Follow the contract's + launch model, whether one long-lived instance or a fresh CLI session per + drive. Exercise every feature even when source looks unchanged. Apply these + rules throughout the pass: + + - Run Doctor before the first drive, for every fresh session, and after a + failed drive. Reset or relaunch when process health cannot explain a stuck + UI state. + - Check captured evidence at its named location after cleanup. + - Clean failed-iteration residue before retrying. Clean only owned resources. + For a shared instance, remove the run's residue without stopping the instance. + - If Doctor fails because instructions drifted, correct them within scope and + retry once. Restart only resources the correction invalidates. + - A `verified-unreachable` result requires the attempted route and a concrete + missing prerequisite such as auth, entitlement, OS, or external state. + Record a missing prerequisite in the map. It does not count as live proof. + - Re-drive every corrected helper before shipping. Perform final teardown + after the last drive and retain the evidence. + +5. **Triage.** Correct wrong or missing user descriptions. Fix gaps in owned + helpers, document exact invocations, and keep executable scripts runnable. + Report broken product behavior separately. Do not weaken the expected result + to turn a failed feature green. + +6. **Check and deliver.** Run `mstack-policy check --root ` when a policy + exists. Re-read every changed file. For `changed`, open one PR of proven + corrections through `/file-pr`. For `clean` or `blocked`, report coverage and + the outcome without opening a PR. Follow the project's receipt and evidence + requirements. A passing policy check does not replace the live pass. + +Keep run notes in scratch, including covered features, unreachable prerequisites, +confirmed drift, and the outcome. Do not commit temporary notes or copies of the +map. Delete any completed plan. diff --git a/skills/meta-mode/SKILL.md b/skills/meta-mode/SKILL.md index 1e0ed45..91ea7af 100644 --- a/skills/meta-mode/SKILL.md +++ b/skills/meta-mode/SKILL.md @@ -8,6 +8,11 @@ description: "Route a non-trivial engineering task through a verifiable mstack w ## Apply the mode 1. State the requested result in one sentence. + When the repository has `.harness/policy.json`, read `AGENTS.md` and + `.harness/workflow.md` before choosing execution steps. Use that shared + workflow for isolation, canonical verification discovery, and evidence. + Repository policy describes project requirements; it does not change the + user's instructions or grant permission for external actions. 2. Choose the smallest matching playbook from `playbooks/`. 3. Read that playbook and every principle it names before you act. 4. Write a short todo list whose first entries are the playbook steps. diff --git a/skills/setup-mstack/SKILL.md b/skills/setup-mstack/SKILL.md index da0686a..d45d82c 100644 --- a/skills/setup-mstack/SKILL.md +++ b/skills/setup-mstack/SKILL.md @@ -76,3 +76,22 @@ its target Harness or is `inherit-parent` or `auto`. Preserve choices for other Harnesses when their model catalogs are unavailable. When working in the mstack repository, run `node scripts/model-config.mjs --file ` to check the shape. Report the roles and selected values to the user. + +## Discover project verification + +Read `AGENTS.md` and `.harness/workflow.md` when present. Inspect the apps in +`.harness/policy.json`, canonical `.harness/verify/` contracts, and project +`verify-*` wrappers. Resolve `metadata.verification-contract` from the repository +root. Multiple wrappers targeting one contract are one verification workflow. + +If canonical verification exists, report its path and follow the project's +workflow. If legacy maps disagree or duplicate canonical facts, report the +conflict and use `/create-verification-skill`'s migration procedure when the +user requested verification setup. Do not generate another copy for this +Harness. + +If neither a reusable verification contract nor an existing repository driver +exists, offer once to run `/create-verification-skill`. Create it when the user +has already requested that work or accepts the offer. Model configuration alone +does not require launching the app or adopting a team policy. Mixed-Harness +adoption is a separate project change described in the mstack guide.