Skip to content

prompts: --core-prompt evolution loop wiring - #122

Merged
noeljackson merged 0 commit into
mainfrom
prompts/evolution-loop
May 14, 2026
Merged

noeljackson merged 0 commit into
mainfrom
prompts/evolution-loop

Conversation

@noeljackson

Copy link
Copy Markdown
Owner

Closes #86.

🤖 Generated with Claude Code

@noeljackson noeljackson mentioned this pull request May 14, 2026
13 of 15 tasks
@noeljackson
noeljackson merged this pull request into main May 14, 2026
3 checks passed
@noeljackson
noeljackson deleted the prompts/evolution-loop branch May 14, 2026 15:56
noeljackson added a commit that referenced this pull request May 15, 2026
The bench corpus is now real (#84 hit 30 tasks). This wires the
final missing piece for prompt evolution: a way to A/B candidate
prompts against the bundled baseline using process-score +
process-compare.

## What's new

- `ProcessOptions.CorePromptOverride` (processor) — a per-run
  string that, when non-empty, replaces the bundled `CorePrompt`
  for every InvestigateBatch this Process produces.
- `InvestigateBatch.CorePromptOverride` — the per-batch field
  AssemblePrompt now checks before falling back to the package
  default.
- `bench.ProcessScoreOptions.CorePromptPath` — file path; the
  scorer reads it once and threads it through.
- `benchsec process-score --core-prompt <path>` flag.

## Walkthrough

  # 1. Author a candidate
  cp internal/processor/prompts/core.md /tmp/core.candidate.md
  $EDITOR /tmp/core.candidate.md

  # 2. Baseline
  benchsec process-score --repeat 8 --seed 1 > baseline.json

  # 3. Candidate (single flag swap)
  benchsec process-score --repeat 8 --seed 1 \
      --core-prompt /tmp/core.candidate.md > candidate.json

  # 4. Compare with CI threshold
  benchsec process-compare <baseline-dir> <candidate-dir> \
      --threshold 0.02

## Documentation

`docs/prompt-evolution.md` is the full walkthrough — when to use
the loop, what to vary, what NOT to vary, how variance shapes
`--repeat` and bootstrap sizing.

## Test

`TestAssemblePromptCorePromptOverride` confirms the override
short-circuits the bundled CorePrompt at the prompt-assembly
layer. Repeat path (`ProcessScoreRepeated`) inherits via
`processScoreOnce`; no new wiring needed.

Pre-push gate: build / vet / gofmt / staticcheck / bench tests all
green.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
noeljackson added a commit that referenced this pull request May 18, 2026
The bench corpus is now real (#84 hit 30 tasks). This wires the
final missing piece for prompt evolution: a way to A/B candidate
prompts against the bundled baseline using process-score +
process-compare.

## What's new

- `ProcessOptions.CorePromptOverride` (processor) — a per-run
  string that, when non-empty, replaces the bundled `CorePrompt`
  for every InvestigateBatch this Process produces.
- `InvestigateBatch.CorePromptOverride` — the per-batch field
  AssemblePrompt now checks before falling back to the package
  default.
- `bench.ProcessScoreOptions.CorePromptPath` — file path; the
  scorer reads it once and threads it through.
- `benchsec process-score --core-prompt <path>` flag.

## Walkthrough

  # 1. Author a candidate
  cp internal/processor/prompts/core.md /tmp/core.candidate.md
  $EDITOR /tmp/core.candidate.md

  # 2. Baseline
  benchsec process-score --repeat 8 --seed 1 > baseline.json

  # 3. Candidate (single flag swap)
  benchsec process-score --repeat 8 --seed 1 \
      --core-prompt /tmp/core.candidate.md > candidate.json

  # 4. Compare with CI threshold
  benchsec process-compare <baseline-dir> <candidate-dir> \
      --threshold 0.02

## Documentation

`docs/prompt-evolution.md` is the full walkthrough — when to use
the loop, what to vary, what NOT to vary, how variance shapes
`--repeat` and bootstrap sizing.

## Test

`TestAssemblePromptCorePromptOverride` confirms the override
short-circuits the bundled CorePrompt at the prompt-assembly
layer. Repeat path (`ProcessScoreRepeated`) inherits via
`processScoreOnce`; no new wiring needed.

Pre-push gate: build / vet / gofmt / staticcheck / bench tests all
green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prompt evolution loop using the bench corpus + process-compare

1 participant