Spec auto-generator emits invalid specs with missing titles and self-contradictory Exclusions

[caoergou]

Problem

The automated spec generator used by owloop go can produce specs that are invalid or self-defeating, which causes the autonomous loop to stall or waste iterations.

Observed in a real refactor run on xyz-data-website:

• Generated specs were missing the # Title / front-matter title, so the spec table showed truncated filenames instead of readable titles.
• The ## Exclusions section listed the very files the spec needed to modify (e.g. the target API file was excluded from modification), making the task impossible to complete.
• Acceptance criteria sometimes referenced shell commands that did not match the actual project layout (e.g. wrong paths or missing test commands).
• Ruff/test baselines written into specs did not match the actual measured baselines on the target branch, forcing the implementation agent to re-measure every time.

Root cause

The spec generator performs static investigation (reading files, grepping, counting call sites) but does not execute the acceptance criteria commands or verify consistency before writing specs to disk. As a result, specs contain plausible-looking but incorrect numbers and contradictory constraints.

Suggested fixes

1. Add a spec validation / grounding pass after the worktree is created and before the autonomous loop starts.

   • Verify every shell command in the spec is syntactically valid and executable in the worktree.
   • Measure actual ruff / pytest baselines on the unmodified target files and update the spec accordingly.
   • Ensure ## Files and ## Exclusions do not overlap.
   • Ensure the title field is present and non-empty.
   • Resolve all Depends On references to existing spec files.

2. If validation fails, stop and surface the problem instead of entering the loop.

   • Either regenerate the offending spec or ask the user for confirmation.
   • Do not let an invalid spec burn implementation iterations.

3. Keep the rule that the loop agent cannot edit specs, but give the pre-loop validator permission to reject or request regeneration.

4. Add test coverage for the spec generator that includes pathological cases (overlapping Files/Exclusions, missing titles, bad commands, stale baselines).

Related

• Meta issue: owloop go end-to-end UX issues (Meta: owloop go end-to-end UX issues discovered during refactor test #75)
• Generated HTML report is inaccurate and lacks AI summary of completed work (Generated HTML report is inaccurate and lacks AI summary of completed work #76)
• Token counter resets between spec generation and autonomous loop (Token counter resets between spec generation and autonomous loop, making total appear to decrease #72)
• Branch naming should be meaningful (Autonomous loop branch names should be meaningful instead of hash-only slugs #74)

[caoergou]

Confirmations + three more spec-gen defects from the same run (moved here from #78 during dedup):

1. Stale baselines confirmed, with a dangerous emergent behavior: 3/5 baselines were wrong (6→17, 9→11, 26→27) because they were measured in the main repo's dirty working tree while the loop ran in a worktree at HEAD. Consequence: the agent invented a per-iteration git stash / measure / stash pop dance — and in iteration 2 the stash pop was skipped when an && chain broke on ruff's exit code, nearly losing all work. Strong +1 for the "grounding pass after worktree creation": baselines must be engine-measured inside the worktree (subprocess, zero tokens), never agent-measured.
2. New — goal→AC coverage gap: the goal explicitly required mypy to pass; spec-gen even measured a mypy baseline, then emitted zero mypy ACs across all 5 specs. Validation should check that every verifiable token in the goal maps to ≥1 AC.
3. New — regeneration lost ## Files: the first generation (7 specs) failed lint and the full regeneration (5 specs) dropped the ## Files section entirely — silently disabling --workers file-disjoint scheduling and any scope-based commit enforcement. ## Files should be a linted required section, and lint failures should trigger incremental repair of the offending spec rather than full regeneration (the full regen cost ~7 min + one mid-stream API stall).
4. New — self-defeating instruction: Requirements say "list it under ## Blockers" while the harness forbids the agent from editing spec files; iteration 3 burned tokens hitting that wall, rolled back, and invented a <learning> side channel. Template and protection rules need one consistent blocker channel (e.g. .owloop/blockers/<spec>.md, or the engine writes the unguarded section on the agent's behalf).