Skip to content

lab: Q14 generator-by-policy result (rejected as stated; tuned stack helps Gemma by 19 points) - #471

Merged
r3dbars merged 1 commit into
mainfrom
claude/tilde-autocomplete-strategy-c88fd9
Sep 2, 2026
Merged

r3dbars merged 1 commit into
mainfrom
claude/tilde-autocomplete-strategy-c88fd9

Conversation

@r3dbars

@r3dbars r3dbars commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Q14 ran as two paired campaigns (Gemma 4 E2B, Qwen 3.5 9B; conservative vs tuned display stack) on the certified-v2 development partition, 3,600/3,600 evaluations each, reviewed and compared with a 10,000-iteration root-clustered bootstrap.

  • Gemma: bad-when-shown 76.6% → 57.1%, improvement 0.19 [0.14, 0.25], P(>0) 100%; useful displays 73 → 199
  • Qwen: 61.1% → 30.8%, improvement 0.30 [0.26, 0.35], P(>0) 100%; useful 373 → 747
  • Guardrail (95% useful retention) held by a wide margin in both
  • Registered hypothesis (Gemma gains at least as much as Qwen) rejected: intervals do not overlap. The tuned stack's direction is generator-independent; its magnitude is not.
  • Every arm fails the bad-suggestion and latency hard gates under the 8-worker campaign runtime, so nothing promotes from this pair, as registered; the standard validation path with a Gemma control starts from here.

Docs only: Q14 result section, lab log entry, Learning Ledger entry (tuned-display-stack-helps-gemma-at-half-qwen-gain, counts re-pinned 47/40).

🤖 Generated with Claude Code

…alf of Qwen's gain; nothing promotes from the development pair

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@r3dbars
r3dbars merged commit 91c7600 into main Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant