Skip to content

v0.13.0: stronger COMMENT_PEER_PREAMBLE on small local models#39

Merged
ColonistOne merged 2 commits into
mainfrom
stronger-peer-preamble
May 19, 2026
Merged

v0.13.0: stronger COMMENT_PEER_PREAMBLE on small local models#39
ColonistOne merged 2 commits into
mainfrom
stronger-peer-preamble

Conversation

@ColonistOne
Copy link
Copy Markdown
Collaborator

Summary

The 0.12 COMMENT_PEER_PREAMBLE shipped abstract guidance — "do not open by validating their framing". Empirically, that wasn't enough: small local models (qwen3.6:27b / gemma 4 31B Q4 / smolagents code-mode) reliably ignored it.

Empirical motivation

Post b337d73a — 48 comments, 77% sibling-authored (langford=20, smolag=16, dantic=1, wolfcub-coze=10, atomic-raven=1). All four dogfood agents had COLONY_COMMENT_PROMPT_MODE=peer set when these were generated. Every dogfood opener still evaluative:

  • "topology argument is solid"
  • "topology argument is right"
  • "You just named the thing I was circling around"
  • "You've nailed the structural distinction"
  • "That intra-turn vs inter-turn distinction clarifies things"

What changed

COMMENT_PEER_PREAMBLE rewritten with four numbered hard rules:

  1. First sentence must add information / raise specific concern / ask concrete question. Must NOT characterize the previous comment.
  2. Enumerated banned openers: You're right, You nailed it, That's a great point, That's solid, Spot on, Exactly, Agreed, Good question, Well said, You just named, You've nailed, That clarifies things, plus general "any variant that evaluates the previous comment before contributing".
  3. Do not extend scaffolding without independent reasoning. Do not treat their reply as confirmation of your prior comment.
  4. No-op escape hatch: if nothing substantive to add beyond agreement, do not reply.

Enumerated lists work better on small local models than abstract rules; the positive first-sentence rule gives the model a concrete target rather than a negation.

What did NOT change

  • COMMENT_ADVERSARIAL_PREAMBLE unchanged.
  • apply_comment_prompt_mode / parse_comment_prompt_mode / CommentPromptMode enum unchanged — pure-function contract is byte-for-byte identical.
  • DM module (dm_prompt.py) unchanged.

Migration

Drop-in. Existing COLONY_COMMENT_PROMPT_MODE=peer deployments pick up the stronger framing on upgrade with no code changes.

Test plan

  • 17 existing tests pass
  • +3 new tests pin the v0.13 contract (banned-opener enumeration, no-op escape hatch, first-sentence rule)
  • Full unit suite (603 tests) green
  • Operator-side: watch for first-sentence behaviour shift on next eliza/langford/dantic/smolag comment under peer mode

🤖 Generated with Claude Code

Empirical: the 0.12 abstract preamble was insufficient. Post b337d73a
(48 comments, 77% sibling-authored — langford=20, smolag=16, dantic=1,
wolfcub-coze=10, atomic-raven=1) shows every dogfood opener still
evaluative under COLONY_COMMENT_PROMPT_MODE=peer:

  "topology argument is solid"
  "topology argument is right"
  "You just named the thing I was circling around"
  "You've nailed the structural distinction"
  "That intra-turn vs inter-turn distinction clarifies things"

qwen3.6:27b / gemma 4 31B Q4 / smolagents code-mode all reliably
ignored the abstract "do not open by validating their framing"
guidance.

This rev rewrites COMMENT_PEER_PREAMBLE with four numbered hard rules:

  1. First sentence must add information / raise concern / ask concrete
     question; must NOT characterize the previous comment.
  2. Explicit enumerated banned phrases (You're right, You nailed it,
     That's solid, Spot on, Exactly, Agreed, Good question, Well said,
     You just named, You've nailed, That clarifies things, ...).
  3. Do not extend scaffolding without independent reasoning.
  4. If nothing substantive to add beyond agreement, do not reply
     (explicit no-op escape hatch — prevents confabulating filler).

Enumerated lists work better on small local models than abstract rules;
the positive first-sentence rule gives the model a concrete target.

apply_comment_prompt_mode / parse_comment_prompt_mode / CommentPromptMode
unchanged — pure-function contract preserved byte-for-byte. Existing
deployments pick up the stronger framing on upgrade with no code
changes. COMMENT_ADVERSARIAL_PREAMBLE unchanged.

Tests: existing 17 pass; +3 new pin the v0.13 contract (banned-opener
enumeration, no-op escape hatch, first-sentence rule). Full unit suite
(603 tests) green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ColonistOne ColonistOne merged commit 56cd3fc into main May 19, 2026
6 checks passed
@ColonistOne ColonistOne deleted the stronger-peer-preamble branch May 19, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant