Skip to content

fix: carry digit context into the following word - #26

Merged
helloiamnotth3devnicetomeetyou merged 3 commits into
iamallyouneed:mainfrom
snowykr:fix/preceding-digit-context
Oct 24, 2026
Merged

helloiamnotth3devnicetomeetyou merged 3 commits into
iamallyouneed:mainfrom
snowykr:fix/preceding-digit-context

Conversation

@snowykr

@snowykr snowykr commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #24.

Repair

The session clears width.after_digit before the first letter can copy it into word_after_digit. It also fails to produce pending numeric context when a digit follows a nonempty word. Restore that producer/consumer handoff, without changing the classifier's numeric policy:

  • A letter consumes pending numeric adjacency once, at new-word entry.
  • A passed-through digit records the boundary after the preceding word is finished. The existing before_digit interval and confidence policy remain intact.
  • Deleting an owned suffix to empty restores its known digit adjacency; a subsequent app-owned Delete invalidates it.
  • Activation, commits/completion, internal splits, new Japanese symbol words and committed Japanese candidate prefixes cannot leak the original adjacency to unrelated text.
  • Japanese phrase/full-width routing is unchanged. No token exceptions, classifier tuning, learning-policy changes or new configuration.

This repair includes current main after #25 (869768bf08eeb80f38234865e6420d8c837821b8). Session and changelog conflicts are reconciled so both visible-reading correction provenance and numeric-context resets are retained. The premerge review also found that Shift+Space returned before clearing pending numeric adjacency; this is now reset in the shared Shift+Space path, with a regression covering both full-width insertion and half-width passthrough. #23's shape-evidence policy is unchanged.

Maintainer integration verification

The final integrated head is 8f3b662ded2b1b77b2bf2a370bb8602cc551abae, including merged #25 and the Shift+Space boundary repair. On Windows:

  • The new shared-session regression fails before the Shift+Space reset and passes afterward, covering full-width insertion and half-width passthrough after Japanese text.
  • The independent recording-engine and real-model review probes pass after the repair. Space and Shift+Space now agree on the following word's nonnumeric context.
  • cargo test --workspace and cargo test --workspace --all-features: pass, including accuracy/golden, correction learning, numeric context, Windows TIP and FFI tests.
  • cargo fmt --all -- --check, default/all-feature all-target workspace Clippy with -D warnings, changelog selftest, and git diff --check: pass.
  • The original measurements below remain evidence for the original implementation; combined corpus benchmarks and physical-key/installed-IME testing were not rerun. The disclosed numeric-path accuracy and allocation tradeoffs remain.
  • Issue question: should reversing a committed correction retain shape evidence? #23 is deliberately unchanged. CI for this updated head must pass before merge.

Original implementation gains and tradeoffs

This is not a zero-regression accuracy/performance claim. Restoring previously lost context activates existing numeric-scoring paths.

A fixed synthetic matrix of 38 Latin identifier/unit controls and 5 Korean counters, two language profiles and Enter/Space gives 172 observations per revision:

  • Correct: 142 → 152; no newly wrong observations in this set; 20 remain wrong.
  • 10cm: 10츠 → 10cm, enko and enkoja.
  • 10fps: 10렌 → 10fps, enko and enkoja.
  • 10sus: 10すs → 10년, enkoja. The original five enko counter controls remain correct.
  • These are selected controls, not representative identifier frequency or overall linguistic accuracy.

Existing enko InputSession replay (--fold 0/3 --corpus 3000) covers 7,050 lines. Exhaustive error listings show no newly wrong or fixed final sentences; all six buckets retain final accuracy. English remains 100%; Korean sentences remain 97.867%.

However, three partial previews become incorrect and two improve. One new mismatch is cm being kept Latin while it is an unfinished Korean 층 (cmd) prefix; two others occur within longer Korean text following a number. This is observable typing behavior, not a harmless metadata change. Existing numeric policy is not retuned to hide it.

Five alternating full-replay benchmark pairs also show a cost:

Bucket p95 median µs, before → after Allocated bytes/key, before → after
English sentences 7.791 → 7.750 439.788 → 439.838
Korean sentences 12.000 → 12.417 1134.453 → 1170.714
Korean comments 12.041 → 12.250 1057.732 → 1073.952
Chat scenarios 8.958 → 9.292 1185.147 → 1254.931
Developer scenarios 14.875 → 14.959 1811.986 → 1819.456
Writing scenarios 10.458 → 10.833 1235.078 → 1268.025

For Korean sentences, the p95 ranges were 11.916–12.041 / 12.375–12.792 µs: the increase is not described as overlapping-range noise. Allocated bytes/key rise about 3.2% for Korean and 5.9% for chat. These are local synthetic replay timings, not physical-key or end-to-end native latency.

The existing nonnumeric NoopClient preview/completion benchmark was also run in five alternating pairs per profile: zero-history p95 medians enko 31.125 → 30.958 µs, enkoja 63.375 → 62.958 µs. This does not cancel or measure away the numeric-path cost above, and no general speedup is claimed.

Original implementation verification

  • Baseline-red recording-engine handoff and owned-delete tests; corrected fixture suite is 71 pass / 2 expected failures before the product fix.
  • Patched: 73 session + 70 Japanese tests pass.
  • Workspace excluding Windows-only tacit-tip locally: 345 default / 486 all-feature tests pass, with 1 / 2 existing ignored tests respectively.
  • Expanded C ABI editor matrix: numeric units/counters/identifier, Enter/Space, known/unknown context, surrounding text, marked range and UTF-16 caret. Baseline fails 10cm; patched default and Japanese-feature editions pass.
  • Rust fmt and default/all-feature all-target Clippy, -D warnings: pass.
  • macOS context/controller suites, 8 installer language-isolation checks, release Japanese FFI build, changelog validator/selftest and git diff --check: pass.
  • Lab: 16 tests pass. Existing classifier --corpus 3000 reports retain 127 enko / 156 enkoja non-timing metrics. Session replay changes are reported separately above.
  • Baseline and patch use separate build targets and the same public data. No personal learning store is used.

The Korean split regression uses an explicitly synthetic routing engine and requires an actual split before testing the retained suffix. An initial real-model fixture did not split on either revision and was corrected; that fixture failure is not counted as a product regression. Japanese split/partial-candidate invalidation was source-reviewed and existing Japanese numeric/conversion tests pass; not every forced-Japanese path has a direct recording-engine context assertion.

The existing switching tool rejects enkoja, so no Japanese corpus-switching replay is claimed. No installed-patched-IME, physical-key, or local Windows/Linux/Android runtime test is claimed. Cross-platform CI is additional evidence.

Full-file Prettier warnings on historical changelog formatting already exist on the baseline; the new Unreleased sections match pinned prettier@3.3.3 without unrelated history churn.

Independent architecture and critic reviews approve the scoped implementation with the above tradeoffs disclosed; runtime checks were performed by the integrating parent, not independently rerun by those reviewers.

No installation, personal-data reset, version bump or release is included. The installed app remains unchanged.

Consume pending numeric adjacency only when starting a word. Preserve owned suffix deletion and invalidate numeric origin across context, commit and split boundaries without changing classifier policy.

Fixes iamallyouneed#24

Signed-off-by: snowykr <snowykr22@gmail.com>
Signed-off-by: notth3dev <160973721+noth3dev@users.noreply.github.com>
Shift+Space can insert or pass through whitespace before the generic dispatch reset. Clear the pending digit context before either return so the next word is scored without stale numeric adjacency. Cover both whitespace widths after integrating the correction-learning repair.

Signed-off-by: notth3dev <160973721+noth3dev@users.noreply.github.com>
@helloiamnotth3devnicetomeetyou
helloiamnotth3devnicetomeetyou merged commit bb76b93 into iamallyouneed:main Oct 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: preceding-digit context is cleared before the first letter

3 participants