Skip to content

Design mode sentinel checks diverge from strategist/builder output formats #1323

Description

@crqu

Summary

The design workflow inherits sentinel checks from build_workflow() without adapting them. This causes verification failures when agents produce valid output that doesn't match the inherited format expectations.

Affected components

  • factory/workflow/definitions.pybuild_workflow() lines 430-487, design_workflow() line 616
  • factory/workflow/verification.py — sentinel compilation
  • factory/agents/prompts/strategist.md — two output formats (improve vs ideation)
  • factory/agents/prompts/builder.md — commit phrasing

Problem

1. Strategist sentinel fails on existing projects

The design workflow inherits must_contain=["### Phase 1", "### Architecture"] from build_workflow() (line 435). This matches the ideation output format (strategist.md:531-537), which is correct for new projects.

But for existing projects (factory ceo /path/to/existing --mode design), the strategist's system prompt contains a second output template — the improve-mode format (strategist.md:99-146) — with headings ### Observations, ### Hypotheses, ### Anti-patterns. These headings have no ### Phase 1 or ### Architecture.

The strategist prompt contains both templates in a single file. Which one the agent follows depends on how well it picks up the prompt_template directive ("Produce a complete phased build plan. Phase 1 must be project scaffold + eval harness.") over its system prompt's improve-mode defaults. When the agent sees an existing project with experiment history, observations, and backlog, it may reasonably follow the improve-mode format — and the sentinel fails:

VERIFY FAIL: strategist: .factory/strategy/current.md missing required sentinel (### Phase 1, ### Architecture)

2. Builder sentinel is fragile

The builder sentinel checks must_contain=["commit"] (line 485) by grepping the agent's captured stdout (builder-latest.md). While the builder prompt says "Commit and open a draft PR" (builder.md:25-26), the actual stdout may use phrasing like "pushed changes" or "opened PR" without echoing the word "commit". The sentinel fails even though the work completed successfully.

3. design_workflow() never overrides inherited checks

design_workflow() modifies edges, gates, start node, and reads — but never touches post_checks on the strategist or builder nodes (lines 616-667). The sentinel checks are inherited verbatim from build_workflow().

Reproduction

# Existing project — strategist may produce improve-mode format
factory ceo /path/to/existing-project --mode design
# → VERIFY FAIL on strategist if current.md uses ### Hypotheses instead of ### Phase 1

Suggested fix

Option A — Broaden the strategist sentinel for design mode. Override post_checks in design_workflow() to accept either format:

# In design_workflow(), after inheriting from build_workflow():
wf.nodes["strategist"] = wf.nodes["strategist"].model_copy(
    update={"post_checks": [
        ArtifactCheck(
            path=".factory/strategy/current.md",
            must_exist=True,
            min_size=200,
            must_contain=["### Phase 1|### Hypotheses"],  # accept either format
        )
    ]},
)

Option B — Split strategist nodes by project type. Use gate_has_factory to route to two different strategist nodes with format-appropriate sentinels.

For the builder sentinel, replace must_contain=["commit"] with a structural check (e.g., verify a git commit or PR exists) rather than grepping agent prose.

Impact

Both surfaces where sentinel checks fire are affected:

  • SKILL.md (interactive mode) — inline bash blocks the CEO runs after each agent call
  • PostToolUse hook (headless mode) — .factory/hooks/verify-design.sh

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions