diff --git a/.changeset/verify-report-skipped-dimensions.md b/.changeset/verify-report-skipped-dimensions.md new file mode 100644 index 0000000000..540bf6ca36 --- /dev/null +++ b/.changeset/verify-report-skipped-dimensions.md @@ -0,0 +1,5 @@ +--- +"@fission-ai/openspec": patch +--- + +Stop `/opsx:verify` from reporting skipped checks as passing. Task completion now uses the schema-aware `tasks` and `progress` fields returned by apply instructions, while absent spec or design inputs are mapped to every check they prevent. Apply instructions aggregate every file matched by the configured task path or glob, regardless of the tracked artifact ID. Verification stays advisory and does not require optional or intentionally omitted artifacts. The scorecard identifies each skipped check, and the final assessment does not claim archive readiness when any check did not run. diff --git a/docs-lab/reference/cli.md b/docs-lab/reference/cli.md index b09393675e..b319c0a2fc 100644 --- a/docs-lab/reference/cli.md +++ b/docs-lab/reference/cli.md @@ -1288,7 +1288,9 @@ With `--json`, each form returns one object. The artifact form starts: ... ``` -and continues with `outputPath`, `existingOutputPaths`, the full `instruction` and `template` strings, `dependencies`, `unlocks`, and `root`. The `apply` form carries `contextFiles`, `progress`, `tasks`, `state` (`blocked`, `ready`, `all_done`), and `instruction`. +and continues with `outputPath`, `existingOutputPaths`, the full `instruction` and `template` strings, `dependencies`, `unlocks`, and `root`. The `apply` form carries `contextFiles`, `progress`, `tasks`, `taskTrackingConfigured`, `state` (`blocked`, `ready`, `all_done`), and `instruction`. + +`taskTrackingConfigured` is always a boolean: `true` when the schema sets a non-null [`apply.tracks`](schemas/schema-yaml.md#tracks), even if no file matches, and `false` otherwise. If a matched tracking file cannot be read, `unavailableTrackingFiles` contains its absolute `path` and error `reason`. This field is omitted when every matched file is readable. Readable files still contribute to `tasks` and `progress`, but `state` cannot be `all_done` until every matched file is read. **Exit codes** diff --git a/docs-lab/reference/schemas/schema-yaml.md b/docs-lab/reference/schemas/schema-yaml.md index bebc06a33e..08a06c92a0 100644 --- a/docs-lab/reference/schemas/schema-yaml.md +++ b/docs-lab/reference/schemas/schema-yaml.md @@ -127,7 +127,7 @@ OpenSpec rejects absolute paths and paths containing a `..` segment. | Field | Contract | |---|---| | `requires` | **Required.** A non-empty list of artifacts that must exist before apply instructions become ready. | -| `tracks` | An optional relative path to a Markdown task file in the change folder. Default: `null`. | +| `tracks` | An optional relative path or glob for Markdown task files in the change folder. Default: `null`. | | `instruction` | Optional guidance sent to the agent when apply is ready. OpenSpec uses built-in guidance by default. | Artifact `requires` controls planning order. `apply.requires` controls when apply instructions become ready. @@ -140,7 +140,9 @@ The path starts from the change folder. For a change named `add-auth`, `tracks: openspec/changes/add-auth/tasks.md ``` -Apply stays blocked if that file is missing or contains no checkbox with task text. OpenSpec counts these checkbox forms: +A glob such as `tracks: "**/tasks.md"` reads every matching file, such as `backend/tasks.md` and `frontend/tasks.md`. OpenSpec combines their tasks and progress. Use the same value for an artifact's `generates` field so status and list track the same files. + +Apply stays blocked if no file matches or the matched files contain no checkbox with task text. OpenSpec counts these checkbox forms: ```markdown - [ ] Pending task @@ -153,11 +155,13 @@ Apply stays blocked if that file is missing or contains no checkbox with task te Any Markdown list marker works: `-`, `*`, `+`, or a number of up to nine digits followed by `.` or `)`. Leading spaces are allowed. The [tasks.md section of the spec-driven page](spec-driven/index.md#tasksmd) defines the stricter format produced by the default schema. -The tracked file drives the apply state: +The tracked files drive the apply state: + +- **`blocked`**: no file matches, or no readable file has a checkbox with task text. +- **`ready`**: at least one task is pending, or a matched file could not be read while another provides tasks. +- **`all_done`**: every tracked task is checked and every matched file was read. -- **`blocked`**: the file is missing, or no checkbox has task text. -- **`ready`**: at least one tracked task is pending. -- **`all_done`**: every tracked task is checked. +If a matched file cannot be read, apply keeps the tasks and progress from readable files but does not mark the change `all_done`. [Apply JSON output](../cli.md#openspec-instructions) identifies each unavailable file and the reason. OpenSpec rejects absolute paths and paths containing a `..` segment. diff --git a/openspec/specs/opsx-verify-skill/spec.md b/openspec/specs/opsx-verify-skill/spec.md index 91562c0e55..740b5002c8 100644 --- a/openspec/specs/opsx-verify-skill/spec.md +++ b/openspec/specs/opsx-verify-skill/spec.md @@ -15,22 +15,37 @@ The system SHALL provide an `/opsx:verify` skill that validates implementation a #### Scenario: Verify without change name - **WHEN** agent executes `/opsx:verify` without a change name - **THEN** the agent infers the change from conversation context, or auto-selects it when only one active change exists -- **AND** when ambiguous, prompts user to select from available changes, showing only changes that have implementation tasks +- **AND** when ambiguous, prompts user to select from all active changes, including changes with no tracked tasks - **AND** announces which change was selected and how to override -#### Scenario: Change has no tasks -- **WHEN** selected change has no tasks.md or tasks are empty -- **THEN** the agent reports "No tasks to verify" -- **AND** suggests running `/opsx:continue` to create tasks +#### Scenario: Change has no task descriptions +- **WHEN** the schema configures task tracking but the structured task list provides no usable task descriptions, even if task progress reports nonzero totals +- **THEN** the agent reports Task Completion as not verified with the reason +- **AND** continues checks supported by the remaining artifacts + +#### Scenario: Schema has no task tracking +- **WHEN** the schema does not configure `apply.tracks` +- **THEN** apply instructions report `taskTrackingConfigured: false` +- **AND** the agent reports Task Completion as not applicable, not as skipped or failed +- **AND** continues the checks that apply to the schema ### Requirement: Completeness Verification The agent SHALL verify that all required work has been completed. #### Scenario: Task completion check - **WHEN** verifying completeness -- **THEN** the agent reads tasks.md -- **AND** counts tasks marked `- [x]` (complete) vs `- [ ]` (incomplete) +- **THEN** the agent uses the top-level `tasks` and `progress` from apply instructions +- **AND** apply instructions aggregate every concrete file matched by the active schema's `apply.tracks`, regardless of the tracked artifact's ID +- **AND** reports complete and total task counts from `progress` - **AND** reports completion status with specific incomplete tasks listed +- **AND** reports remaining checkboxes without descriptions when `progress.remaining` exceeds the listed incomplete tasks + +#### Scenario: Tracking evidence becomes unavailable +- **WHEN** one or more files matched by `apply.tracks` cannot be read after resolution +- **THEN** apply instructions include every unavailable path and reason +- **AND** preserve tasks and progress from readable tracking files +- **AND** do not report `all_done` +- **AND** the agent marks Task Completion as not verified from partial evidence #### Scenario: Spec coverage check - **WHEN** verifying completeness @@ -42,7 +57,8 @@ The agent SHALL verify that all required work has been completed. #### Scenario: All tasks complete - **WHEN** all tasks are marked complete - **THEN** report "Tasks: N/N complete" -- **AND** mark completeness dimension as passed +- **AND** mark Task Completion as passed only when task descriptions are available +- **AND** mark the completeness dimension as passed only when all applicable checks ran and passed #### Scenario: Incomplete tasks found - **WHEN** some tasks are incomplete @@ -98,7 +114,7 @@ The agent SHALL verify that implementation is sensible and follows design decisi - **WHEN** verifying coherence - **AND** no design.md exists - **THEN** skip design adherence check -- **AND** note "No design.md to verify against" +- **AND** report "Design Adherence: Not verified (No design.md to verify against)" #### Scenario: Design decision followed - **WHEN** implementation follows a design decision @@ -113,8 +129,10 @@ The agent SHALL verify that implementation is sensible and follows design decisi #### Scenario: Code pattern consistency - **WHEN** verifying coherence +- **AND** available artifacts support identifying implementation changes beyond a tasks-only check - **THEN** check if new code follows existing project patterns - **AND** flag any significant deviations as suggestions +- **AND** report Code Pattern Consistency as not verified if implementation changes cannot be identified ### Requirement: Verification Report Format The agent SHALL produce a structured, prioritized report. @@ -132,6 +150,8 @@ The agent SHALL produce a structured, prioritized report. | Correctness | X/Y | | Coherence | Followed | ``` +- **AND** report `Not verified ()` for every skipped or partially verified check in its dimension's status +- **AND** never count a skipped check as passing #### Scenario: Issue prioritization - **WHEN** issues are found @@ -147,7 +167,7 @@ The agent SHALL produce a structured, prioritized report. - **AND** avoid vague suggestions like "consider reviewing" #### Scenario: All checks pass -- **WHEN** no issues found across all dimensions +- **WHEN** every applicable check ran and no issues were found across all dimensions - **THEN** display: ```text All checks passed. Ready for archive. @@ -160,15 +180,31 @@ The agent SHALL produce a structured, prioritized report. X critical issue(s) found. Fix before archiving. ``` - **AND** do NOT suggest running archive +- **AND** name every skipped check and its reason, if any -#### Scenario: Only warnings/suggestions -- **WHEN** no CRITICAL issues but warnings exist +#### Scenario: Only warnings +- **WHEN** every applicable check ran and no CRITICAL issues but warnings exist - **THEN** display: ```text No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements). ``` +#### Scenario: Only suggestions +- **WHEN** every applicable check ran and only suggestions exist +- **THEN** report "No critical issues or warnings. Z suggestion(s) to consider. Ready for archive (with noted improvements)." + +#### Scenario: Checks skipped +- **WHEN** any check was skipped or partially verified and no CRITICAL issues exist +- **THEN** report "No critical issues found in the checks that ran" +- **AND** name every unverified check and its reason +- **AND** include the warning count when nonzero +- **AND** do not claim archive readiness + +#### Scenario: Suggestions in final assessment +- **WHEN** suggestions exist +- **THEN** include their count in the final assessment, including assessments with critical issues or skipped checks + ### Requirement: Flexible Artifact Handling The agent SHALL gracefully handle changes with varying artifact completeness. @@ -188,3 +224,16 @@ The agent SHALL gracefully handle changes with varying artifact completeness. - **WHEN** change has proposal, design, specs, and tasks - **THEN** perform all verification checks - **AND** cross-reference artifacts for consistency + +#### Scenario: Unusable or partial artifact evidence +- **WHEN** an artifact cannot be read or lacks usable requirements, scenarios, or design decisions +- **THEN** mark each affected check as not verified with its reason +- **AND** continue checks supported by the remaining evidence without treating partial coverage as a fully verified check + +#### Scenario: Intentional artifact omissions +- **WHEN** a check has no supporting artifacts because the schema omits task tracking or optional artifacts, or the change declares `skip_specs: true` +- **THEN** report the corresponding checks as not applicable and explain why +- **AND** exclude not-applicable checks from skipped-check counts and readiness assessment +- **AND** do not require or create optional or intentionally skipped artifacts to obtain a passing report +- **AND** treat verification as advisory: not verified describes missing evidence for an applicable check, not a new archive gate +- **AND** leave archive checks and user-confirmation behavior unchanged diff --git a/skills/openspec-verify-change/SKILL.md b/skills/openspec-verify-change/SKILL.md index 355febc9b2..d5f9d293f8 100644 --- a/skills/openspec-verify-change/SKILL.md +++ b/skills/openspec-verify-change/SKILL.md @@ -35,7 +35,7 @@ In both branches, never create the root as a side effect: do not run `openspec i - Auto-select if only one active change exists - If ambiguous, run `openspec list --json` to get available changes and ask the user to select one - When prompting, show changes that have implementation tasks (tasks artifact exists). + When prompting, show all active changes returned by the list, including changes with `status: "no-tasks"`. Include the schema used for each change if available. Mark changes with incomplete tasks as "(In Progress)". @@ -56,7 +56,9 @@ In both branches, never create the root as a side effect: do not run `openspec i openspec instructions apply --change "" --json ``` - This returns the change directory and `contextFiles` (artifact ID -> array of concrete file paths). Read all available artifacts from `contextFiles`. + This returns the change directory, `contextFiles` (artifact ID -> array of concrete file paths), `taskTrackingConfigured`, and top-level `tasks` and `progress` aggregated from every concrete file matched by the schema's `apply.tracks` configuration that could be read. Read all available artifacts from `contextFiles`. + + Treat apply `state` and `instruction` as context, not a verification verdict. Do not implement tasks or archive the change during verification. 4. **Initialize verification report structure** @@ -67,19 +69,29 @@ In both branches, never create the root as a side effect: do not run `openspec i Each dimension can have CRITICAL, WARNING, or SUGGESTION issues. + Verification is advisory. Respect intentional omissions such as `skip_specs: true`, optional design documents, and schemas without task tracking. Do not require or invent optional or intentionally omitted artifacts to obtain a clean report. `Not verified` describes a limit of this report, not a new archive prerequisite. Archive retains its own checks and user-confirmation behavior. + + Mark checks the schema does not define, or artifacts the status reports as intentionally skipped, as **Not applicable**. Exclude them from skipped-check counts and the archive-readiness assessment. Reserve **Not verified** for applicable checks whose evidence is missing or unusable. + + If only task evidence is available for applicable checks, verify task completion only and mark the remaining applicable checks, including **Code Pattern Consistency**, as not verified with the reason "Only task evidence available". + + If artifacts cannot be read or contain no usable requirements, scenarios, or design decisions, mark the affected checks as not verified with the specific reason. Continue checks supported by the remaining evidence, but a partially checked input set is not a fully verified check. Missing requirements affect Spec Coverage and Requirement Implementation Mapping; missing scenarios affect Scenario Coverage; missing design decisions affect Design Adherence. + 5. **Verify Completeness** **Task Completion**: - - If `contextFiles.tasks` exists, read every file path in it - - Parse checkboxes: complete means the box holds only `x`/`X`, ignoring - spacing (`- [ x]` is complete); every other marker is incomplete - (`- [ ]`, `- []`, and unfamiliar ones such as `- [~]` or `- [-]`) - - Count complete vs total tasks - - If incomplete tasks exist: - - Add CRITICAL issue for each incomplete task + - If `taskTrackingConfigured` is false, report **Task Completion** as not applicable. Do not treat empty `tasks` as missing evidence. + - Otherwise, use the top-level `tasks` and `progress` fields. They already aggregate every readable concrete file matched by `apply.tracks`, regardless of the tracked artifact's ID; do not infer tracking from a `contextFiles` key. + - If `unavailableTrackingFiles` is nonempty, mark **Task Completion** as not verified and include every unavailable path and reason. Continue using any readable task evidence, but do not infer completion from the partial `tasks` and `progress` fields. + - If `taskTrackingConfigured` is true and `tasks` is empty, mark **Task Completion** as not verified and record the reason from apply `state` and `instruction`. Nonzero totals alone do not establish evaluable task descriptions. + - Report complete vs total tasks from `progress`. + - If `progress.remaining` is greater than 0: + - Add CRITICAL issue for each listed incomplete task. If the remaining count exceeds the listed incomplete tasks, also report the incomplete checkboxes without descriptions and recommend adding descriptions and completing them. Do not infer completion from the listed tasks alone. - Recommendation: "Complete task: " or "Mark as done if already implemented" **Spec Coverage**: + - If status marks the spec artifact skipped by `skip_specs: true`, or the schema defines no spec artifact, report the spec-dependent checks as not applicable. + - Otherwise, `contextFiles` is keyed by artifact id, and artifact ids come from the active schema. If `contextFiles.specs` is absent or empty, mark **Spec Coverage**, **Requirement Implementation Mapping**, and **Scenario Coverage** as not verified; do not treat any of them as clean. - If delta specs exist in `contextFiles.specs`: - Extract all requirements (marked with "### Requirement:") - For each requirement: @@ -111,16 +123,18 @@ In both branches, never create the root as a side effect: do not run `openspec i 7. **Verify Coherence** **Design Adherence**: + - If the schema defines no design artifact, report **Design Adherence** as not applicable. - If `contextFiles.design` exists: - Extract key decisions (look for sections like "Decision:", "Approach:", "Architecture:") - Verify implementation follows those decisions - If contradiction detected: - Add WARNING: "Design decision not followed: " - Recommendation: "Update implementation or revise design.md to match reality" - - If no design.md: Skip design adherence check, note "No design.md to verify against" + - Otherwise, if `contextFiles.design` is absent or empty: mark **Design Adherence** as not verified. With other supporting artifacts, **Code Pattern Consistency** still runs; the task-only case remains limited to task completion. **Code Pattern Consistency**: - - Review new code for consistency with project patterns + - If implementation changes cannot be identified, mark **Code Pattern Consistency** as not verified and explain the missing evidence. + - Otherwise, review new code for consistency with project patterns - Check file naming, directory structure, coding style - If significant deviations found: - Add SUGGESTION: "Code pattern deviation:
" @@ -140,6 +154,8 @@ In both branches, never create the root as a side effect: do not run `openspec i | Coherence | Followed/Issues | ``` + In each Status cell, report the results of checks that ran and `Not verified ()` for every skipped check. If all checks in a dimension were skipped, start the cell with `Not verified`. Never score a skipped check as passing. Treat every not verified or partially verified check as skipped in the final assessment. + **Issues by Priority**: 1. **CRITICAL** (Must fix before archive): @@ -158,9 +174,12 @@ In both branches, never create the root as a side effect: do not run `openspec i - Each with specific recommendation **Final Assessment**: - - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." - - If only warnings: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." - - If all clear: "All checks passed. Ready for archive." + - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." If any check was skipped, also name every skipped check and its reason. + - If no CRITICAL issues, one or more warnings, and no checks were skipped: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." + - If only suggestions and no checks were skipped: "No critical issues or warnings. Z suggestion(s) to consider. Ready for archive (with noted improvements)." + - If no issues and no checks were skipped: "All checks passed. Ready for archive." + - If any check was skipped and there are no CRITICAL issues: do not claim readiness. Say "No critical issues found in the checks that ran. not verified: ." Include the warning count when nonzero. + - Include the suggestion count when nonzero in every final assessment. **Verification Heuristics** @@ -170,13 +189,6 @@ In both branches, never create the root as a side effect: do not run `openspec i - **False Positives**: When uncertain, prefer SUGGESTION over WARNING, WARNING over CRITICAL - **Actionability**: Every issue must have a specific recommendation with file/line references where applicable -**Graceful Degradation** - -- If only tasks.md exists: verify task completion only, skip spec/design checks -- If tasks + specs exist: verify completeness and correctness, skip design -- If full artifacts: verify all three dimensions -- Always note which checks were skipped and why - **Output Format** Use clear markdown with: diff --git a/src/commands/workflow/instructions.ts b/src/commands/workflow/instructions.ts index a585e864ff..aed6418338 100644 --- a/src/commands/workflow/instructions.ts +++ b/src/commands/workflow/instructions.ts @@ -12,7 +12,6 @@ import { loadChangeContext, generateInstructions, resolveSchema, - resolveArtifactOutputPath, resolveArtifactOutputs, type ArtifactInstructions, } from '../../core/artifact-graph/index.js'; @@ -569,15 +568,27 @@ export async function generateApplyInstructions( } } - // Parse tasks if tracking file exists + // Parse every concrete file matched by apply.tracks. A tracking path may be + // a glob owned by an artifact with any ID, so treating it as one literal + // path loses task evidence for valid custom schemas. let parsedTasks: ParsedTask[] = []; + const unavailableTrackingFiles: Array<{ path: string; reason: string }> = []; let tracksFileExists = false; if (tracksFile) { - const tracksPath = resolveArtifactOutputPath(changeDir, tracksFile); - tracksFileExists = fs.existsSync(tracksPath); - if (tracksFileExists) { - const tasksContent = await fs.promises.readFile(tracksPath, 'utf-8'); - parsedTasks = parseTaskLines(tasksContent); + const tracksPaths = resolveArtifactOutputs(changeDir, tracksFile); + tracksFileExists = tracksPaths.length > 0; + for (const tracksPath of tracksPaths) { + try { + const tasksContent = await fs.promises.readFile(tracksPath, 'utf-8'); + parsedTasks.push(...parseTaskLines(tasksContent)); + } catch (error) { + const code = (error as NodeJS.ErrnoException)?.code; + const message = error instanceof Error ? error.message : String(error); + unavailableTrackingFiles.push({ + path: tracksPath, + reason: code && !message.includes(code) ? `${code}: ${message}` : message, + }); + } } } const tasks = toTaskItems(parsedTasks); @@ -615,6 +626,9 @@ export async function generateApplyInstructions( instruction = `The ${tracksFilename} file is missing and must be created.` + `\n${describeArtifactRemedy(changeName, findArtifactIdFor(schema, tracksFile))}`; + } else if (tracksFile && unavailableTrackingFiles.length > 0 && tasks.length === 0) { + state = 'blocked'; + instruction = 'No readable task descriptions are available.'; } else if (tracksFile && tracksFileExists && tasks.length === 0) { // Tracking file exists but lists nothing an agent can work on: either no // checkboxes at all, or only checkboxes with no text after them. @@ -623,7 +637,12 @@ export async function generateApplyInstructions( instruction = `The ${tracksFilename} file exists but contains no tasks to work on.` + `\nAdd tasks to ${tracksFilename}, or rebuild it: ${describeArtifactRemedy(changeName, findArtifactIdFor(schema, tracksFile))}`; - } else if (tracksFile && remaining === 0 && total > 0) { + } else if ( + tracksFile && + unavailableTrackingFiles.length === 0 && + remaining === 0 && + total > 0 + ) { state = 'all_done'; instruction = 'All tasks are complete! This change is ready to be archived.\nConsider running tests and reviewing the changes before archiving.'; } else if (!tracksFile) { @@ -635,6 +654,13 @@ export async function generateApplyInstructions( instruction = schemaInstruction?.trim() ?? 'Read context files, work through pending tasks, mark complete as you go.\nPause if you hit blockers or need clarification.'; } + if (unavailableTrackingFiles.length > 0) { + const unavailableDetails = unavailableTrackingFiles + .map((file) => `- ${file.path}: ${file.reason}`) + .join('\n'); + instruction += `\nTask completion is not verified because tracking evidence was unavailable:\n${unavailableDetails}`; + } + const warnings = await collectApplyWarnings({ state, schema, @@ -650,6 +676,8 @@ export async function generateApplyInstructions( contextFiles, progress: { total, complete, remaining }, tasks, + taskTrackingConfigured: tracksFile !== null, + ...(unavailableTrackingFiles.length > 0 ? { unavailableTrackingFiles } : {}), state, missingArtifacts: missingArtifacts.length > 0 ? missingArtifacts : undefined, ...(missingPrerequisites.length > 0 ? { missingPrerequisites } : {}), diff --git a/src/commands/workflow/shared.ts b/src/commands/workflow/shared.ts index 0b0075ddbc..e3e93bc73a 100644 --- a/src/commands/workflow/shared.ts +++ b/src/commands/workflow/shared.ts @@ -45,6 +45,11 @@ export interface ApplyInstructions { remaining: number; }; tasks: TaskItem[]; + taskTrackingConfigured: boolean; + unavailableTrackingFiles?: Array<{ + path: string; + reason: string; + }>; state: 'blocked' | 'all_done' | 'ready'; missingArtifacts?: string[]; /** diff --git a/src/core/templates/workflows/verify-change.ts b/src/core/templates/workflows/verify-change.ts index 8bc85edfb6..92c4b0cc2b 100644 --- a/src/core/templates/workflows/verify-change.ts +++ b/src/core/templates/workflows/verify-change.ts @@ -29,7 +29,7 @@ ${PROJECT_ROOT_GUARD} - Auto-select if only one active change exists - If ambiguous, run \`openspec list --json\` to get available changes and ask the user to select one - When prompting, show changes that have implementation tasks (tasks artifact exists). + When prompting, show all active changes returned by the list, including changes with \`status: "no-tasks"\`. Include the schema used for each change if available. Mark changes with incomplete tasks as "(In Progress)". @@ -50,7 +50,9 @@ ${PROJECT_ROOT_GUARD} openspec instructions apply --change "" --json \`\`\` - This returns the change directory and \`contextFiles\` (artifact ID -> array of concrete file paths). Read all available artifacts from \`contextFiles\`. + This returns the change directory, \`contextFiles\` (artifact ID -> array of concrete file paths), \`taskTrackingConfigured\`, and top-level \`tasks\` and \`progress\` aggregated from every concrete file matched by the schema's \`apply.tracks\` configuration that could be read. Read all available artifacts from \`contextFiles\`. + + Treat apply \`state\` and \`instruction\` as context, not a verification verdict. Do not implement tasks or archive the change during verification. 4. **Initialize verification report structure** @@ -61,19 +63,29 @@ ${PROJECT_ROOT_GUARD} Each dimension can have CRITICAL, WARNING, or SUGGESTION issues. + Verification is advisory. Respect intentional omissions such as \`skip_specs: true\`, optional design documents, and schemas without task tracking. Do not require or invent optional or intentionally omitted artifacts to obtain a clean report. \`Not verified\` describes a limit of this report, not a new archive prerequisite. Archive retains its own checks and user-confirmation behavior. + + Mark checks the schema does not define, or artifacts the status reports as intentionally skipped, as **Not applicable**. Exclude them from skipped-check counts and the archive-readiness assessment. Reserve **Not verified** for applicable checks whose evidence is missing or unusable. + + If only task evidence is available for applicable checks, verify task completion only and mark the remaining applicable checks, including **Code Pattern Consistency**, as not verified with the reason "Only task evidence available". + + If artifacts cannot be read or contain no usable requirements, scenarios, or design decisions, mark the affected checks as not verified with the specific reason. Continue checks supported by the remaining evidence, but a partially checked input set is not a fully verified check. Missing requirements affect Spec Coverage and Requirement Implementation Mapping; missing scenarios affect Scenario Coverage; missing design decisions affect Design Adherence. + 5. **Verify Completeness** **Task Completion**: - - If \`contextFiles.tasks\` exists, read every file path in it - - Parse checkboxes: complete means the box holds only \`x\`/\`X\`, ignoring - spacing (\`- [ x]\` is complete); every other marker is incomplete - (\`- [ ]\`, \`- []\`, and unfamiliar ones such as \`- [~]\` or \`- [-]\`) - - Count complete vs total tasks - - If incomplete tasks exist: - - Add CRITICAL issue for each incomplete task + - If \`taskTrackingConfigured\` is false, report **Task Completion** as not applicable. Do not treat empty \`tasks\` as missing evidence. + - Otherwise, use the top-level \`tasks\` and \`progress\` fields. They already aggregate every readable concrete file matched by \`apply.tracks\`, regardless of the tracked artifact's ID; do not infer tracking from a \`contextFiles\` key. + - If \`unavailableTrackingFiles\` is nonempty, mark **Task Completion** as not verified and include every unavailable path and reason. Continue using any readable task evidence, but do not infer completion from the partial \`tasks\` and \`progress\` fields. + - If \`taskTrackingConfigured\` is true and \`tasks\` is empty, mark **Task Completion** as not verified and record the reason from apply \`state\` and \`instruction\`. Nonzero totals alone do not establish evaluable task descriptions. + - Report complete vs total tasks from \`progress\`. + - If \`progress.remaining\` is greater than 0: + - Add CRITICAL issue for each listed incomplete task. If the remaining count exceeds the listed incomplete tasks, also report the incomplete checkboxes without descriptions and recommend adding descriptions and completing them. Do not infer completion from the listed tasks alone. - Recommendation: "Complete task: " or "Mark as done if already implemented" **Spec Coverage**: + - If status marks the spec artifact skipped by \`skip_specs: true\`, or the schema defines no spec artifact, report the spec-dependent checks as not applicable. + - Otherwise, \`contextFiles\` is keyed by artifact id, and artifact ids come from the active schema. If \`contextFiles.specs\` is absent or empty, mark **Spec Coverage**, **Requirement Implementation Mapping**, and **Scenario Coverage** as not verified; do not treat any of them as clean. - If delta specs exist in \`contextFiles.specs\`: - Extract all requirements (marked with "### Requirement:") - For each requirement: @@ -105,16 +117,18 @@ ${PROJECT_ROOT_GUARD} 7. **Verify Coherence** **Design Adherence**: + - If the schema defines no design artifact, report **Design Adherence** as not applicable. - If \`contextFiles.design\` exists: - Extract key decisions (look for sections like "Decision:", "Approach:", "Architecture:") - Verify implementation follows those decisions - If contradiction detected: - Add WARNING: "Design decision not followed: " - Recommendation: "Update implementation or revise design.md to match reality" - - If no design.md: Skip design adherence check, note "No design.md to verify against" + - Otherwise, if \`contextFiles.design\` is absent or empty: mark **Design Adherence** as not verified. With other supporting artifacts, **Code Pattern Consistency** still runs; the task-only case remains limited to task completion. **Code Pattern Consistency**: - - Review new code for consistency with project patterns + - If implementation changes cannot be identified, mark **Code Pattern Consistency** as not verified and explain the missing evidence. + - Otherwise, review new code for consistency with project patterns - Check file naming, directory structure, coding style - If significant deviations found: - Add SUGGESTION: "Code pattern deviation:
" @@ -134,6 +148,8 @@ ${PROJECT_ROOT_GUARD} | Coherence | Followed/Issues | \`\`\` + In each Status cell, report the results of checks that ran and \`Not verified ()\` for every skipped check. If all checks in a dimension were skipped, start the cell with \`Not verified\`. Never score a skipped check as passing. Treat every not verified or partially verified check as skipped in the final assessment. + **Issues by Priority**: 1. **CRITICAL** (Must fix before archive): @@ -152,9 +168,12 @@ ${PROJECT_ROOT_GUARD} - Each with specific recommendation **Final Assessment**: - - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." - - If only warnings: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." - - If all clear: "All checks passed. Ready for archive." + - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." If any check was skipped, also name every skipped check and its reason. + - If no CRITICAL issues, one or more warnings, and no checks were skipped: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." + - If only suggestions and no checks were skipped: "No critical issues or warnings. Z suggestion(s) to consider. Ready for archive (with noted improvements)." + - If no issues and no checks were skipped: "All checks passed. Ready for archive." + - If any check was skipped and there are no CRITICAL issues: do not claim readiness. Say "No critical issues found in the checks that ran. not verified: ." Include the warning count when nonzero. + - Include the suggestion count when nonzero in every final assessment. **Verification Heuristics** @@ -164,13 +183,6 @@ ${PROJECT_ROOT_GUARD} - **False Positives**: When uncertain, prefer SUGGESTION over WARNING, WARNING over CRITICAL - **Actionability**: Every issue must have a specific recommendation with file/line references where applicable -**Graceful Degradation** - -- If only tasks.md exists: verify task completion only, skip spec/design checks -- If tasks + specs exist: verify completeness and correctness, skip design -- If full artifacts: verify all three dimensions -- Always note which checks were skipped and why - **Output Format** Use clear markdown with: @@ -208,7 +220,7 @@ ${PROJECT_ROOT_GUARD} - Auto-select if only one active change exists - If ambiguous, run \`openspec list --json\` to get available changes and ask the user to select one - When prompting, show changes that have implementation tasks (tasks artifact exists). + When prompting, show all active changes returned by the list, including changes with \`status: "no-tasks"\`. Include the schema used for each change if available. Mark changes with incomplete tasks as "(In Progress)". @@ -229,7 +241,9 @@ ${PROJECT_ROOT_GUARD} openspec instructions apply --change "" --json \`\`\` - This returns the change directory and \`contextFiles\` (artifact ID -> array of concrete file paths). Read all available artifacts from \`contextFiles\`. + This returns the change directory, \`contextFiles\` (artifact ID -> array of concrete file paths), \`taskTrackingConfigured\`, and top-level \`tasks\` and \`progress\` aggregated from every concrete file matched by the schema's \`apply.tracks\` configuration that could be read. Read all available artifacts from \`contextFiles\`. + + Treat apply \`state\` and \`instruction\` as context, not a verification verdict. Do not implement tasks or archive the change during verification. 4. **Initialize verification report structure** @@ -240,19 +254,28 @@ ${PROJECT_ROOT_GUARD} Each dimension can have CRITICAL, WARNING, or SUGGESTION issues. + Verification is advisory. Respect intentional omissions such as \`skip_specs: true\`, optional design documents, and schemas without task tracking. Do not require or invent optional or intentionally omitted artifacts to obtain a clean report. \`Not verified\` describes a limit of this report, not a new archive prerequisite. Archive retains its own checks and user-confirmation behavior. + Mark checks the schema does not define, or artifacts the status reports as intentionally skipped, as **Not applicable**. Exclude them from skipped-check counts and the archive-readiness assessment. Reserve **Not verified** for applicable checks whose evidence is missing or unusable. + + If only task evidence is available for applicable checks, verify task completion only and mark the remaining applicable checks, including **Code Pattern Consistency**, as not verified with the reason "Only task evidence available". + + If artifacts cannot be read or contain no usable requirements, scenarios, or design decisions, mark the affected checks as not verified with the specific reason. Continue checks supported by the remaining evidence, but a partially checked input set is not a fully verified check. Missing requirements affect Spec Coverage and Requirement Implementation Mapping; missing scenarios affect Scenario Coverage; missing design decisions affect Design Adherence. + 5. **Verify Completeness** **Task Completion**: - - If \`contextFiles.tasks\` exists, read every file path in it - - Parse checkboxes: complete means the box holds only \`x\`/\`X\`, ignoring - spacing (\`- [ x]\` is complete); every other marker is incomplete - (\`- [ ]\`, \`- []\`, and unfamiliar ones such as \`- [~]\` or \`- [-]\`) - - Count complete vs total tasks - - If incomplete tasks exist: - - Add CRITICAL issue for each incomplete task + - If \`taskTrackingConfigured\` is false, report **Task Completion** as not applicable. Do not treat empty \`tasks\` as missing evidence. + - Otherwise, use the top-level \`tasks\` and \`progress\` fields. They already aggregate every readable concrete file matched by \`apply.tracks\`, regardless of the tracked artifact's ID; do not infer tracking from a \`contextFiles\` key. + - If \`unavailableTrackingFiles\` is nonempty, mark **Task Completion** as not verified and include every unavailable path and reason. Continue using any readable task evidence, but do not infer completion from the partial \`tasks\` and \`progress\` fields. + - If \`taskTrackingConfigured\` is true and \`tasks\` is empty, mark **Task Completion** as not verified and record the reason from apply \`state\` and \`instruction\`. Nonzero totals alone do not establish evaluable task descriptions. + - Report complete vs total tasks from \`progress\`. + - If \`progress.remaining\` is greater than 0: + - Add CRITICAL issue for each listed incomplete task. If the remaining count exceeds the listed incomplete tasks, also report the incomplete checkboxes without descriptions and recommend adding descriptions and completing them. Do not infer completion from the listed tasks alone. - Recommendation: "Complete task: " or "Mark as done if already implemented" **Spec Coverage**: + - If status marks the spec artifact skipped by \`skip_specs: true\`, or the schema defines no spec artifact, report the spec-dependent checks as not applicable. + - Otherwise, \`contextFiles\` is keyed by artifact id, and artifact ids come from the active schema. If \`contextFiles.specs\` is absent or empty, mark **Spec Coverage**, **Requirement Implementation Mapping**, and **Scenario Coverage** as not verified; do not treat any of them as clean. - If delta specs exist in \`contextFiles.specs\`: - Extract all requirements (marked with "### Requirement:") - For each requirement: @@ -284,16 +307,18 @@ ${PROJECT_ROOT_GUARD} 7. **Verify Coherence** **Design Adherence**: + - If the schema defines no design artifact, report **Design Adherence** as not applicable. - If \`contextFiles.design\` exists: - Extract key decisions (look for sections like "Decision:", "Approach:", "Architecture:") - Verify implementation follows those decisions - If contradiction detected: - Add WARNING: "Design decision not followed: " - Recommendation: "Update implementation or revise design.md to match reality" - - If no design.md: Skip design adherence check, note "No design.md to verify against" + - Otherwise, if \`contextFiles.design\` is absent or empty: mark **Design Adherence** as not verified. With other supporting artifacts, **Code Pattern Consistency** still runs; the task-only case remains limited to task completion. **Code Pattern Consistency**: - - Review new code for consistency with project patterns + - If implementation changes cannot be identified, mark **Code Pattern Consistency** as not verified and explain the missing evidence. + - Otherwise, review new code for consistency with project patterns - Check file naming, directory structure, coding style - If significant deviations found: - Add SUGGESTION: "Code pattern deviation:
" @@ -313,6 +338,8 @@ ${PROJECT_ROOT_GUARD} | Coherence | Followed/Issues | \`\`\` + In each Status cell, report the results of checks that ran and \`Not verified ()\` for every skipped check. If all checks in a dimension were skipped, start the cell with \`Not verified\`. Never score a skipped check as passing. Treat every not verified or partially verified check as skipped in the final assessment. + **Issues by Priority**: 1. **CRITICAL** (Must fix before archive): @@ -331,9 +358,12 @@ ${PROJECT_ROOT_GUARD} - Each with specific recommendation **Final Assessment**: - - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." - - If only warnings: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." - - If all clear: "All checks passed. Ready for archive." + - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." If any check was skipped, also name every skipped check and its reason. + - If no CRITICAL issues, one or more warnings, and no checks were skipped: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." + - If only suggestions and no checks were skipped: "No critical issues or warnings. Z suggestion(s) to consider. Ready for archive (with noted improvements)." + - If no issues and no checks were skipped: "All checks passed. Ready for archive." + - If any check was skipped and there are no CRITICAL issues: do not claim readiness. Say "No critical issues found in the checks that ran. not verified: ." Include the warning count when nonzero. + - Include the suggestion count when nonzero in every final assessment. **Verification Heuristics** @@ -343,13 +373,6 @@ ${PROJECT_ROOT_GUARD} - **False Positives**: When uncertain, prefer SUGGESTION over WARNING, WARNING over CRITICAL - **Actionability**: Every issue must have a specific recommendation with file/line references where applicable -**Graceful Degradation** - -- If only tasks.md exists: verify task completion only, skip spec/design checks -- If tasks + specs exist: verify completeness and correctness, skip design -- If full artifacts: verify all three dimensions -- Always note which checks were skipped and why - **Output Format** Use clear markdown with: diff --git a/test/commands/apply-instructions-tasks.test.ts b/test/commands/apply-instructions-tasks.test.ts index 38f482803e..f11e76aefb 100644 --- a/test/commands/apply-instructions-tasks.test.ts +++ b/test/commands/apply-instructions-tasks.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'node:fs'; import * as path from 'node:path'; import * as os from 'node:os'; @@ -27,6 +27,7 @@ describe('generateApplyInstructions task list', () => { }); afterEach(() => { + vi.restoreAllMocks(); fs.rmSync(tempDir, { recursive: true, force: true }); }); @@ -34,6 +35,162 @@ describe('generateApplyInstructions task list', () => { fs.writeFileSync(path.join(changeDir, 'tasks.md'), content); } + function writeGlobTasksSchema(): void { + const schemaDir = path.join(tempDir, 'openspec', 'schemas', 'glob-tasks'); + fs.mkdirSync(schemaDir, { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: glob-tasks +version: 1 +artifacts: + - id: implementation + generates: "**/tasks.md" + description: Implementation checklists + template: tasks.md + requires: [] +apply: + requires: [implementation] + tracks: "**/tasks.md" +` + ); + fs.writeFileSync(path.join(changeDir, '.openspec.yaml'), 'schema: glob-tasks\n'); + } + + it.each([true, false])('resolves custom tracking configuration (enabled: %s)', async (tracked) => { + const schemaDir = path.join(tempDir, 'openspec', 'schemas', 'custom'); + fs.mkdirSync(schemaDir, { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: custom +version: 1 +artifacts: + - id: implementation + generates: checklist.md + description: Implementation checklist + template: checklist.md + requires: [] +apply: + requires: [implementation] +${tracked ? ' tracks: checklist.md\n' : ''}` + ); + fs.writeFileSync(path.join(changeDir, '.openspec.yaml'), 'schema: custom\n'); + const checklist = path.join(changeDir, 'checklist.md'); + fs.writeFileSync(checklist, '- [x] Finished task\n- [ ] Pending task\n'); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + + expect(instructions.contextFiles).toEqual({ implementation: [fs.realpathSync.native(checklist)] }); + expect(instructions.taskTrackingConfigured).toBe(tracked); + expect(instructions.tasks).toEqual(tracked ? [ + { id: '1', description: 'Finished task', done: true }, + { id: '2', description: 'Pending task', done: false }, + ] : []); + expect(instructions.progress).toEqual(tracked + ? { total: 2, complete: 1, remaining: 1 } + : { total: 0, complete: 0, remaining: 0 }); + expect(instructions.state).toBe('ready'); + }); + + it.each(['missing', 'empty'])('returns no task evidence for a %s tracking file', async (kind) => { + if (kind === 'empty') writeTasks(''); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + + expect(instructions.tasks).toEqual([]); + expect(instructions.progress).toEqual({ total: 0, complete: 0, remaining: 0 }); + expect(instructions.state).toBe('blocked'); + expect(instructions.instruction).toContain(kind === 'missing' ? 'Missing artifacts: tasks' : 'contains no tasks'); + }); + + it('aggregates a tracking glob owned by an artifact not named tasks', async () => { + writeGlobTasksSchema(); + const backendTasks = path.join(changeDir, 'backend', 'tasks.md'); + const frontendTasks = path.join(changeDir, 'frontend', 'tasks.md'); + fs.mkdirSync(path.dirname(backendTasks), { recursive: true }); + fs.mkdirSync(path.dirname(frontendTasks), { recursive: true }); + fs.writeFileSync(backendTasks, '- [x] Finished backend task\n'); + fs.writeFileSync(frontendTasks, '- [x] Finished frontend task\n- [ ] Pending frontend task\n'); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + const listProgress = await getTaskProgressForChange( + path.join(tempDir, 'openspec', 'changes'), + 'my-change', + tempDir + ); + + expect(instructions.contextFiles.implementation).toEqual([ + fs.realpathSync.native(backendTasks), + fs.realpathSync.native(frontendTasks), + ]); + expect(instructions.tasks).toEqual([ + { id: '1', description: 'Finished backend task', done: true }, + { id: '2', description: 'Finished frontend task', done: true }, + { id: '3', description: 'Pending frontend task', done: false }, + ]); + expect(instructions.progress).toEqual({ total: 3, complete: 2, remaining: 1 }); + expect(instructions.state).toBe('ready'); + expect(listProgress).toEqual({ total: 3, completed: 2 }); + }); + + it('retains partial task evidence when a tracked file is unreadable', async () => { + writeGlobTasksSchema(); + const backendTasks = path.join(changeDir, 'backend', 'tasks.md'); + const frontendTasks = path.join(changeDir, 'frontend', 'tasks.md'); + fs.mkdirSync(path.dirname(backendTasks), { recursive: true }); + fs.mkdirSync(path.dirname(frontendTasks), { recursive: true }); + fs.writeFileSync(backendTasks, '- [x] Finished backend task\n'); + fs.writeFileSync(frontendTasks, '- [x] Finished frontend task\n'); + vi.spyOn(fs.promises, 'readFile').mockRejectedValueOnce( + Object.assign(new Error('permission denied'), { code: 'EACCES' }) + ); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + + expect(instructions.tasks).toEqual([ + { id: '1', description: 'Finished frontend task', done: true }, + ]); + expect(instructions.progress).toEqual({ total: 1, complete: 1, remaining: 0 }); + expect(instructions.unavailableTrackingFiles).toEqual([ + { path: fs.realpathSync.native(backendTasks), reason: 'EACCES: permission denied' }, + ]); + expect(instructions.state).toBe('ready'); + expect(instructions.instruction).toContain('Task completion is not verified'); + expect(instructions.instruction).toContain(fs.realpathSync.native(backendTasks)); + }); + + it('reports tracking evidence that disappears after glob resolution', async () => { + writeTasks('- [x] Finished task\n'); + const tasksPath = fs.realpathSync.native(path.join(changeDir, 'tasks.md')); + vi.spyOn(fs.promises, 'readFile').mockRejectedValueOnce( + Object.assign(new Error('no such file or directory'), { code: 'ENOENT' }) + ); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + + expect(instructions.tasks).toEqual([]); + expect(instructions.progress).toEqual({ total: 0, complete: 0, remaining: 0 }); + expect(instructions.unavailableTrackingFiles).toEqual([ + { path: tasksPath, reason: 'ENOENT: no such file or directory' }, + ]); + expect(instructions.state).toBe('blocked'); + expect(instructions.instruction).toContain('Task completion is not verified'); + expect(instructions.instruction).toContain(tasksPath); + }); + + it('returns existing spec and design paths even when their files contain no evidence', async () => { + writeTasks('- [x] Finished task\n'); + const spec = path.join(changeDir, 'specs', 'demo', 'spec.md'); + const design = path.join(changeDir, 'design.md'); + fs.writeFileSync(spec, ''); + fs.writeFileSync(design, ''); + + const instructions = await generateApplyInstructions(tempDir, 'my-change'); + + expect(instructions.contextFiles.specs).toEqual([fs.realpathSync.native(spec)]); + expect(instructions.contextFiles.design).toEqual([fs.realpathSync.native(design)]); + expect(instructions.state).toBe('all_done'); + }); + it('lists indented sub-tasks alongside their parents', async () => { writeTasks( [ @@ -84,6 +241,7 @@ describe('generateApplyInstructions task list', () => { // As before the shared parser: apply points at regenerating the file // rather than listing a blank row an agent cannot act on. expect(instructions.tasks).toEqual([]); + expect(instructions.progress).toEqual({ total: 1, complete: 1, remaining: 0 }); expect(instructions.state).toBe('blocked'); expect(instructions.instruction).toContain('contains no tasks'); }); diff --git a/test/core/templates/skill-templates-parity.test.ts b/test/core/templates/skill-templates-parity.test.ts index 4157a5d9d0..03fda1ae79 100644 --- a/test/core/templates/skill-templates-parity.test.ts +++ b/test/core/templates/skill-templates-parity.test.ts @@ -91,11 +91,11 @@ const EXPECTED_FUNCTION_HASHES: Record = { getArchiveChangeSkillTemplate: '71715f9d5899498942af03e182e6d1ac2c95952dde967950c2a9161084a53a8b', getBulkArchiveChangeSkillTemplate: '2a6ec08fea0f942158b4abe9c8d1af9622038e0c4dc684e7c73dad2fb8379a54', getOpsxSyncCommandTemplate: '60550b7bb9829421656d6324a9e4c951bc912f48f88882d1a07ce7f78397a5e7', - getVerifyChangeSkillTemplate: '2e069a277dac23818b13bb50b66e806ab405bc3b7f535400e1ebf81b84153699', + getVerifyChangeSkillTemplate: '40d955c703bd319c3d7ecbf8c6da3a04d7c85746587844ecc60656a89a2f47b3', getOpsxArchiveCommandTemplate: '3d2a330b46043fbb9f220831aa42ebbb62f411e9597b2bb491ad1ac1fa2d0873', getOpsxOnboardCommandTemplate: '0cf66e164c0e14c916c6d1ebb5d80ded07d7fb8e55d4eb34eba43e8ca9c28558', getOpsxBulkArchiveCommandTemplate: '4e2e39c4d634074f4a1ed67f076d5c4d0ead8b998f4d75218c33cdc6173719be', - getOpsxVerifyCommandTemplate: '938f52f20fb9a3b811ea47314baac1034cd550e8ab363ae878ccba4b6329348f', + getOpsxVerifyCommandTemplate: '9c8219820ee83a72d180a94c7504457a5fa54670386be266f8247cccd5c68fce', getOpsxProposeSkillTemplate: '1aa2f2eb9c8cbc4dcab9d777bf8832b92ca04f9ef91d0494f1224a566aefdfe8', getOpsxProposeCommandTemplate: '3b7090ce5e79e879ab9b5bdaf4ff2b52e3c02211f71188838772d36ac337f96c', getFeedbackSkillTemplate: 'dabeb5e825b9349abc8156c3e7b8608f27987912a6d9bf47ef29addde6138133', @@ -112,7 +112,7 @@ const EXPECTED_GENERATED_SKILL_CONTENT_HASHES: Record = { 'openspec-sync-specs': '3909936a236a21a9a6d5bf495f90b396b3b68fc9220d7b2c1894668653beb2e4', 'openspec-archive-change': 'd01d9eeb06223ee89708b7963e82c5ebc11719c5b2dc62d4abb268ee016fcb7b', 'openspec-bulk-archive-change': '10f050ad5ef77084dc55a202427988b23903ee122f00985238a4eb9354a5dc3c', - 'openspec-verify-change': 'ad8a3098bd27d852721687c47a12db7107ed8b8dfc7f071406bb19961652e7ee', + 'openspec-verify-change': '1a9493b623da4f399ceac0be105c7a7248db421e83eb4c8b983ba3692c4a7453', 'openspec-onboard': '6993eff867d97d485e080078f9dfb80e968e242f3b17a924eeb077715fd548fa', 'openspec-propose': '66e3395adf9f2d93a09e8ef1d20e4efb010e5e8d4811f2d42a9316e4d1ca5a8b', 'openspec-update-change': '5f4ea19aa732b33d87a2120ec393ee34578e70678d97e8c3bb10f988c00cb4d3', diff --git a/test/core/templates/verify-change.test.ts b/test/core/templates/verify-change.test.ts new file mode 100644 index 0000000000..05fa002949 --- /dev/null +++ b/test/core/templates/verify-change.test.ts @@ -0,0 +1,124 @@ +import { describe, expect, it } from 'vitest'; + +import { + getOpsxVerifyCommandTemplate, + getVerifyChangeSkillTemplate, +} from '../../../src/core/templates/skill-templates.js'; + +const skill = getVerifyChangeSkillTemplate(); +const command = getOpsxVerifyCommandTemplate(); + +const bodies: Array<[string, string]> = [ + ['skill', skill.instructions], + ['command', command.content], +]; + +describe('verify-change templates', () => { + it('keeps active no-task changes eligible for ambiguous selection', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('show all active changes returned by the list'); + expect(body, label).toContain('including changes with `status: "no-tasks"`'); + expect(body, label).not.toContain( + 'show changes that have implementation tasks (tasks artifact exists)' + ); + } + }); + + it('prefers schema-aware apply task fields without assuming a tasks artifact id', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('top-level `tasks` and `progress`'); + expect(body, label).toContain("schema's `apply.tracks` configuration"); + expect(body, label).toContain('aggregated from every concrete file matched'); + expect(body, label).toContain("regardless of the tracked artifact's ID"); + expect(body, label).toContain('do not infer tracking from a `contextFiles` key'); + } + }); + + it('marks partial tracking evidence as not verified', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('If `unavailableTrackingFiles` is nonempty'); + expect(body, label).toContain('include every unavailable path and reason'); + expect(body, label).toContain('do not infer completion from the partial `tasks` and `progress` fields'); + } + }); + + + it('does not lose incomplete checkboxes omitted from the task list', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('If `progress.remaining` is greater than 0'); + expect(body, label).toContain('incomplete checkboxes without descriptions'); + expect(body, label).toContain('Do not infer completion from the listed tasks alone'); + } + }); + + it('requires usable evidence rather than just existing artifact paths', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('cannot be read or contain no usable requirements, scenarios, or design decisions'); + expect(body, label).toContain('Continue checks supported by the remaining evidence'); + expect(body, label).toContain('a partially checked input set is not a fully verified check'); + expect(body, label).toContain('If implementation changes cannot be identified, mark **Code Pattern Consistency** as not verified'); + } + }); + + it('does not mistake apply readiness for verification or execute apply instructions', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('Treat apply `state` and `instruction` as context, not a verification verdict'); + expect(body, label).toContain('Do not implement tasks or archive the change during verification'); + } + }); + + it('preserves optional artifacts and the existing archive workflow', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('Verification is advisory'); + expect(body, label).toContain('`skip_specs: true`'); + expect(body, label).toContain('schemas without task tracking'); + expect(body, label).toContain('Do not require or invent optional or intentionally omitted artifacts'); + expect(body, label).toContain('Mark checks the schema does not define, or artifacts the status reports as intentionally skipped, as **Not applicable**'); + expect(body, label).toContain('Exclude them from skipped-check counts and the archive-readiness assessment'); + expect(body, label).toContain('If `taskTrackingConfigured` is false, report **Task Completion** as not applicable'); + expect(body, label).toContain('If `taskTrackingConfigured` is true and `tasks` is empty, mark **Task Completion** as not verified'); + expect(body, label).toContain('`Not verified` describes a limit of this report, not a new archive prerequisite'); + expect(body, label).toContain('Archive retains its own checks and user-confirmation behavior'); + } + }); + + it('preserves task-only verification without dropping checks supported by other artifacts', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('If only task evidence is available for applicable checks, verify task completion only'); + expect(body, label).toContain('including **Code Pattern Consistency**, as not verified'); + expect(body, label).toContain('With other supporting artifacts, **Code Pattern Consistency** still runs'); + } + }); + + it('covers warning and suggestion outcomes without claiming all checks passed', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('If no CRITICAL issues, one or more warnings, and no checks were skipped'); + expect(body, label).toContain('If only suggestions and no checks were skipped'); + expect(body, label).toContain('Include the suggestion count when nonzero'); + } + }); + + it('maps missing supporting artifacts to every check they prevent', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain( + 'mark **Spec Coverage**, **Requirement Implementation Mapping**, and **Scenario Coverage** as not verified' + ); + expect(body, label).toContain('mark **Design Adherence** as not verified'); + expect(body, label).toContain('**Code Pattern Consistency** still runs'); + } + }); + + it('never reports a skipped check as passing or archive-ready', () => { + for (const [label, body] of bodies) { + expect(body, label).toContain('`Not verified ()` for every skipped check'); + expect(body, label).toContain('Never score a skipped check as passing'); + expect(body, label).toContain('Treat every not verified or partially verified check as skipped in the final assessment'); + expect(body, label).toContain('If any check was skipped and there are no CRITICAL issues'); + expect(body, label).toContain( + 'If any check was skipped, also name every skipped check and its reason' + ); + expect(body, label).toContain('do not claim readiness'); + expect(body, label).toContain('If no issues and no checks were skipped'); + } + }); +});