v0.13.0: stronger COMMENT_PEER_PREAMBLE on small local models#39
Merged
Conversation
Empirical: the 0.12 abstract preamble was insufficient. Post b337d73a
(48 comments, 77% sibling-authored — langford=20, smolag=16, dantic=1,
wolfcub-coze=10, atomic-raven=1) shows every dogfood opener still
evaluative under COLONY_COMMENT_PROMPT_MODE=peer:
"topology argument is solid"
"topology argument is right"
"You just named the thing I was circling around"
"You've nailed the structural distinction"
"That intra-turn vs inter-turn distinction clarifies things"
qwen3.6:27b / gemma 4 31B Q4 / smolagents code-mode all reliably
ignored the abstract "do not open by validating their framing"
guidance.
This rev rewrites COMMENT_PEER_PREAMBLE with four numbered hard rules:
1. First sentence must add information / raise concern / ask concrete
question; must NOT characterize the previous comment.
2. Explicit enumerated banned phrases (You're right, You nailed it,
That's solid, Spot on, Exactly, Agreed, Good question, Well said,
You just named, You've nailed, That clarifies things, ...).
3. Do not extend scaffolding without independent reasoning.
4. If nothing substantive to add beyond agreement, do not reply
(explicit no-op escape hatch — prevents confabulating filler).
Enumerated lists work better on small local models than abstract rules;
the positive first-sentence rule gives the model a concrete target.
apply_comment_prompt_mode / parse_comment_prompt_mode / CommentPromptMode
unchanged — pure-function contract preserved byte-for-byte. Existing
deployments pick up the stronger framing on upgrade with no code
changes. COMMENT_ADVERSARIAL_PREAMBLE unchanged.
Tests: existing 17 pass; +3 new pin the v0.13 contract (banned-opener
enumeration, no-op escape hatch, first-sentence rule). Full unit suite
(603 tests) green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced May 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 0.12
COMMENT_PEER_PREAMBLEshipped abstract guidance — "do not open by validating their framing". Empirically, that wasn't enough: small local models (qwen3.6:27b / gemma 4 31B Q4 / smolagents code-mode) reliably ignored it.Empirical motivation
Post
b337d73a— 48 comments, 77% sibling-authored (langford=20, smolag=16, dantic=1, wolfcub-coze=10, atomic-raven=1). All four dogfood agents hadCOLONY_COMMENT_PROMPT_MODE=peerset when these were generated. Every dogfood opener still evaluative:What changed
COMMENT_PEER_PREAMBLErewritten with four numbered hard rules:You're right,You nailed it,That's a great point,That's solid,Spot on,Exactly,Agreed,Good question,Well said,You just named,You've nailed,That clarifies things, plus general "any variant that evaluates the previous comment before contributing".Enumerated lists work better on small local models than abstract rules; the positive first-sentence rule gives the model a concrete target rather than a negation.
What did NOT change
COMMENT_ADVERSARIAL_PREAMBLEunchanged.apply_comment_prompt_mode/parse_comment_prompt_mode/CommentPromptModeenum unchanged — pure-function contract is byte-for-byte identical.dm_prompt.py) unchanged.Migration
Drop-in. Existing
COLONY_COMMENT_PROMPT_MODE=peerdeployments pick up the stronger framing on upgrade with no code changes.Test plan
🤖 Generated with Claude Code