Skip to content

The prose example stops asking for line breaks #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bojieli merged 2 commits into from
Sep 6, 2026
Merged

The prose example stops asking for line breaks #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a2572b
dnt: an --example flag, so a wording can be tried on real audio
bojieli Sep 6, 2026
68f7a8e
prompt: the prose example stops asking for line breaks
bojieli Sep 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions Sources/dnt/Dnt.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,18 @@ struct BackendOptions: ParsableArguments {
""")
var keyterms = false

/// Overrides what the app's example box holds, for this run only.
///
/// Nothing is written back: the point is to answer "what would this wording do to my own
/// audio" without first changing the setting the app is dictating with. `dnt-eval` has the
/// same flag against the near-miss corpus; this one points it at a real recording.
@Option(
name: .long,
help: ArgumentHelp(
"Override the example box: a preset name (prose, chat, notes), `none`, or literal "
+ "text. Defaults to whatever the app is set to."))
var example: String?

func resolveProvider() throws -> ProviderKind {
guard let provider else { return AppPreferences.provider }
guard let kind = ProviderKind(persistedValue: provider) else {
Expand All @@ -147,6 +159,21 @@ struct BackendOptions: ParsableArguments {
model ?? AppPreferences.model(for: kind)
}

/// What to send as the dictation-style example.
///
/// A preset name resolves to that preset's text — the user's own copy of it when they have
/// edited one, since the builder is the same one the app reads. Anything else is taken
/// literally, so a wording being considered can be measured before it is written to a file.
func resolveDictationExample(_ builder: PromptBuilder) throws -> String {
guard let example else { return AppPreferences.dictationExample(using: builder) }
let trimmed = example.trimmed
if trimmed.isEmpty || trimmed.lowercased() == "none" { return "" }
if let preset = DictationPreset(rawValue: trimmed.lowercased()) {
return try builder.dictationPresetText(preset)
}
return trimmed
}

func promptURL() throws -> URL {
if let prompt { return URL(fileURLWithPath: prompt) }
guard let found = PromptBuilder.findPromptDirectory() ?? Self.bundledPromptURL else {
Expand Down Expand Up @@ -206,7 +233,7 @@ struct BackendOptions: ParsableArguments {
systemInstruction: try promptBuilder().systemInstruction(
fidelity: try resolveFidelity(),
script: AppPreferences.chineseScript,
dictationExample: AppPreferences.dictationExample(using: try promptBuilder())),
dictationExample: try resolveDictationExample(try promptBuilder())),
fidelity: try resolveFidelity(),
keytermBiasing: keyterms,
typography: AppPreferences.typographySpacing)
Expand Down Expand Up @@ -239,7 +266,7 @@ struct BackendOptions: ParsableArguments {
systemInstruction: try promptBuilder().systemInstruction(
fidelity: try resolveFidelity(),
script: AppPreferences.chineseScript,
dictationExample: AppPreferences.dictationExample(using: try promptBuilder())),
dictationExample: try resolveDictationExample(try promptBuilder())),
fidelity: try resolveFidelity(),
typography: AppPreferences.typographySpacing)
return (service, resolved.source)
Expand Down
54 changes: 52 additions & 2 deletions docs/PROMPT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,8 +253,8 @@ clause added to it unconditionally would invalidate all of them at once.
> is still an answer, and it was the least predictable one available. So the numbers below describe
> the request as *measured*, which is now the request an install makes only after the box is
> cleared. Upgrading installs are untouched: whatever they were sending, they go on sending. The
> re-measurement against the seeded default is owed, and clearing the box reproduces the table's
> conditions exactly in the meantime.
> re-measurement against the seeded default was owed and is now the 2026-09-06 pair below;
> clearing the box still reproduces the table's original conditions exactly.

**What each block is allowed to do.** Both open by restating the rule they could otherwise be read
as relaxing: formatting governs how the transcript is written down, never what it says, and nothing
Expand All @@ -280,6 +280,8 @@ screen context broke it. That is the failure this contract exists to prevent, an

| Date | Change | Provider / model | runs | matched | improved | regressed |
|------|--------|------------------|------|---------|----------|-----------|
| 2026-09-06 | Prose example reworded to stop asking for paragraph breaks | **gemini** · gemini-3.6-flash | 48 | 38 | 8 | **2** |
| 2026-09-06 | The seeded prose default as it shipped in 0.6.0 (control) | **gemini** · gemini-3.6-flash | 48 | 38 | 7 | **3** |
| 2026-08-30 | Writing styles added; default request byte-identical (control) | **gemini** · gemini-3.5-flash | 48 | 38 | 8 | **2** |
| 2026-08-30 | Translation stage added; transcription request untouched (control) | **gemini** · gemini-3.5-flash | 48 | 36 | 9 | **3** |
| 2026-08-30 | Formatting blocks added; default request byte-identical (control) | **gemini** · gemini-3.5-flash | 48 | 38 | 7 | **2** |
Expand All @@ -291,6 +293,54 @@ screen context broke it. That is the failure this contract exists to prevent, an
| 2026-08-09 | Initial contract | **gemini** · gemini-3.6-flash | 15 | 15 | 0 | **0** |
| 2026-08-09 | Initial contract | openrouter · google/gemini-3.6-flash | 15 | 12 | 0 | 1 |

### 2026-09-06 — the prose example stops asking for line breaks

The previous entry closed by saying that a change to the *text* of a formatting block needs its own
measurement. This is that change, and this is that measurement.

`prompt/dictation-style/prose.md` ended with *"and paragraph breaks where the speaker changed
subject"*. Seeded into every new install since 0.6.0, it turned out to be an instruction the model
obeys at hesitation pauses rather than at subject changes — which in spontaneous speech is most
pauses. The complaint that surfaced it was a 65-second Mandarin dictation that came back in 19
fragments, 8 of them splitting a sentence in half.

**Measured on real dictations rather than on the near-miss corpus**, because the near-miss suite
scores spelling under screen context and says nothing about layout. Ten recordings were replayed
from the maintainer's own history with the screen context each one originally carried, six passes
per wording, through `dnt transcribe --example`:

| wording | runs | line breaks | mid-sentence | runs affected |
|---|---|---|---|---|
| `...and paragraph breaks where the speaker changed subject.` | 60 | 82 | 26 | 24 |
| `...in one unbroken paragraph. Never emit a line break.` | 60 | 0 | 0 | 0 |

**An empty box was not the alternative.** It also produces no line breaks, but on two of the five
Mandarin recordings the model then dropped nearly all punctuation — 1.0 and 0.0 marks per hundred
characters, a wall of text. The prose clause earns its place; only its paragraph half was harmful.
This is the same finding as the seeding decision above, from the other direction.

**Naming the failure made it worse.** A variant saying *"never in the middle of a sentence"* and
*"a pause is punctuation, never a line break"* produced more mid-sentence breaks than the wording
it was meant to fix, not fewer. That is the effect already recorded in `ContextEncoder.footer`,
where an instruction illustrating the rule with a concrete wrong value made substitution worse:
naming the wrong answer appears to prime it. The wording that shipped does not mention the failure,
it just declines to ask for the behaviour.

**Cost on the near-miss suite: none that this suite can see.** The reworded example and the 0.6.0
default were each run at three passes on the same day — 38 matched out of 48 both times, regressed
3 against 2. Both movements are inside the per-pass range the runner prints, and both runs are
committed as scorecards (`2026-09-06-prose-unbroken.json`, `2026-09-06-prose-paragraphs-control.json`)
so the grading can be re-checked without re-billing the suite.

The control run is also the re-measurement owed since the box began being seeded: it is the request
a 0.6.0 install actually makes, rather than the empty-box request every older row above describes.

**Existing installs do not pick this up on their own.** The example box stores a *copy* of the
preset's text and is seeded once, when the key has never been written, so an install that already
has the old sentence keeps sending it. Pressing **Prose** again in Settings replaces it. That is
the same property that makes the box editable at all, and it is why this is a fix somebody has to
accept rather than one that arrives silently.

### 2026-08-30 — the writing styles, and a third control run

`prompt/dictation-style.md` (the renamed `sample.md`) and `prompt/dictation-style/` were added, and
Expand Down
Loading
Loading