The prose example stops asking for line breaks - #35
Merged
Merged
Conversation
dnt-eval has had this flag since the example box shipped, but it points at the near-miss corpus. The question that actually comes up is narrower and about the user's own speech: "my dictations have line breaks I did not ask for — which of the words in my box is putting them there?" Answering it meant editing the setting the app dictates with, running the hotkey, and editing it back, which changes the thing being measured while measuring it. The flag resolves the same three ways dnt-eval's does — a preset name, `none`, or literal text — through the same builder, so a preset picks up the user's own override of that file rather than the shipped copy. Nothing is written back: this reads the box for one run, and the app goes on sending whatever it was sending. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`prose.md` ended with "and paragraph breaks where the speaker changed subject".
Seeded into every new install since 0.6.0, that turned out to be an instruction
the model obeys at hesitation pauses rather than at subject changes — and in
spontaneous speech those are mostly the same pauses. A 65-second Mandarin
dictation came back in 19 fragments, 8 of them splitting a sentence in half.
Measured on real dictations, not on eval/nearmiss: that suite scores spelling
under screen context and says nothing about layout. Ten recordings replayed from
history with the screen context each originally carried, six passes per wording:
82 line breaks across 60 runs before, 0 across 60 after.
Clearing the box also gives no line breaks, and is not the fix. On two of the
five Mandarin recordings it dropped nearly all punctuation — 1.0 and 0.0 marks
per hundred characters. The prose clause earns its place; only its paragraph
half was doing harm.
A guarded variant naming the failure ("never in the middle of a sentence", "a
pause is punctuation, never a line break") produced *more* mid-sentence breaks
than the wording it was meant to fix. That is the effect already recorded in
ContextEncoder.footer, where illustrating the rule with a concrete wrong value
made substitution worse. The shipped wording does not mention line breaks as a
mistake; it just stops asking for them.
Near-miss cost: none this suite can see. The new wording and the 0.6.0 default
were run three passes each on the same day — 38/48 matched both times, regressed
2 against 3, inside the printed per-pass range. Both runs are committed as
scorecards so the grading can be re-checked without re-billing the suite. The
control doubles as the re-measurement PROMPT.md has owed since the box began
being seeded.
Existing installs keep the old sentence: the box stores a copy and is seeded
only once. Pressing Prose again replaces it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prose is the seeded default, and it was the thing inserting the line breaks people were choosing prose to avoid.
prompt/dictation-style/prose.mdended with "and paragraph breaks where the speaker changed subject". The model obeys that at hesitation pauses rather than at subject changes, and in spontaneous speech those are mostly the same pauses. The dictation that surfaced it was 65 seconds of Mandarin that came back in 19 fragments, 8 of them splitting a sentence in half.Measured on real dictations
eval/nearmissscores spelling under screen context and says nothing about layout, so it cannot see this. Ten recordings were replayed from local history with the screen context each one originally carried, six passes per wording:...and paragraph breaks where the speaker changed subject....in one unbroken paragraph. Never emit a line break.Clearing the box is not the fix, though it also gives no line breaks. On two of the five Mandarin recordings the model then dropped nearly all punctuation, down to 1.0 and 0.0 marks per hundred characters. The prose clause earns its place. Only its paragraph half was doing harm.
Naming the failure made it worse. A variant saying "never in the middle of a sentence" and "a pause is punctuation, never a line break" produced more mid-sentence breaks than the wording it was meant to fix. That is the effect already recorded in
ContextEncoder.footer, where illustrating the rule with a concrete wrong value made substitution worse. The shipped wording does not mention line breaks as a mistake; it just stops asking for them.Cost
None this suite can see. The new wording and the 0.6.0 default were run three passes each on the same day: 38 of 48 matched both times, regressed 2 against 3, inside the per-pass range the runner prints. Both runs are committed as scorecards so the grading can be re-checked without re-billing the suite, and
dnt-eval rescorereproduces every count.Also here
dnt transcribe --example, mirroring the flagdnt-evalalready has. It is what made the table above possible without editing the setting the app dictates with while measuring it.Note for anyone already on 0.6.0
The example box stores a copy of the preset's text and is seeded only once, so an existing install keeps the old sentence. Pressing Prose again in Settings replaces it.
🤖 Generated with Claude Code