bug: Shift-overlap exception converts case-sensitive Latin identifiers to Korean/Japanese

Summary

The Shift-overlap exception introduced in #15 changes some intentional, case-sensitive Latin identifier inputs into Korean or Japanese. This reproduces with empty learning data against the exact upstream parent and merged revisions, not just against my earlier local candidate.

This is the inverse failure direction of #12: Korean input was incorrectly forced to English there; here previously preserved Latin identifiers are automatically converted. It is separate from #11's confirmation-stability fix. I am not requesting a wholesale rollback of #15 or claiming that all English capitalization is broken.

Revisions and conditions

Parent: acb013e542829d8c8565d8e458929322417bf134
Merge of fix: preserve confirmed readings and handle shifted Korean vowels #15: ce7bd934eb784990cc6db86ac76a33d7f2e2bbfc
Both revisions freshly built as release tacit-ffi with --features japanese.
Same public language packs for both builds.
Fresh temporary profile per input; learned, once, and shapes all empty.
languages: ["ko", "en", "ja"], liveConversion: false, returnConfirmsOnly: true.
Tables below use an empty preceding document context and Enter confirmation.
These are synthetic character-event replays through the actual C FFI, not physical keyboard timing measurements or an installed-application test of the merged build.

Separating the code change from local learning or IME state

I performed an additional check specifically to distinguish this report from a personal dictionary, settings, external-pack, or local input-method problem:

FFI layer: the paired reproductions above create new temporary profiles with an empty learning store. They do not copy a personal dictionary. The parent and merge use the same inputs and configuration.
Direct model layer: I built a separate integration-test executable for each exact revision with tacit-core --all-features, including the embedded English/Korean/Japanese models. Each executable ran with a new empty HOME and working directory, empty XDG configuration/data roots, and no inherited application environment. The empty directory remained empty afterward.
That test directly constructs HeuristicEngine::for_languages(...) and calls decide(...). It does not construct a LearningEngine, load settings, initialize external packs with init_from_dir, use the C FFI or InputSession, or access macOS input-method/browser/native-client APIs. Model availability was asserted before the checks.

The direct, embedded-only checks independently reproduce:

Parent: cmdWL, rmWL, and dmlWL remain Latin.
Merge: the same inputs become 층찌, 그찌, and 의찌, in both partial and final direct decisions, with enko and enkoja.
With Japanese enabled, the direct parent decision keeps yukiRI; the direct merged decision returns ゆきり.
Direct Hello and CMDWL controls remain unchanged.

Thus these counterexamples reproduce without personal learning, user settings, external installed language packs, or the local IME/FFI/session path. turnsRI is supported by the paired FFI evidence; I am not extending the direct-model claim to that input without a matching direct assertion.

This isolates a source/model decision change rather than merely observing a symptom in my installed app. It does not certify behavior on every OS/compiler or establish real-world frequency: the runtime executions were on one macOS host, and no independent Windows/Linux/Android runtime reproduction is claimed. No personal learning data was inspected, copied, cleared, or modified for this investigation.

Expected and actual

For these fixtures, the intended text is the exact case-sensitive Latin identifier. The parent preserves it. The merge changes both the completed preview and the insertion on Enter:

Input	Parent preview / Enter	Merged preview / Enter
`cmdWL`	`cmdWL`	`층찌`
`rmWL`	`rmWL`	`그찌`
`dmlWL`	`dmlWL`	`의찌`
`turnsRI`	`turnsRI`	`셔군꺄`
`yukiRI`	`yukiRI`	`ゆきり`

These identifiers are deliberately constructed negative controls matching the new exception, not claims that they are common English dictionary words. No real-world frequency or overall regression rate is inferred from them. The same characters can have different intended readings; the concern is the loss of the previous capitalization protection under the stated English-preservation goal, not a demand that every arbitrary Latin key sequence always remain English.

The first four Korean outputs also reproduce with only English and Korean enabled. The Japanese result is particularly relevant: evidence intended to permit a Korean Shift-overlap reading also opens competition to Japanese.

The change happens before confirmation

For cmdWL, the final two key responses are (M = marked text):

After cmdW:
  parent: 1 | M층ㅉ
  merged: 1 | M층ㅉ

After the final uppercase L:
  parent: 1 | McmdWL
  merged: 1 | M층찌

Enter:
  parent: 1 | IcmdWL
  merged: 1 | I층찌

Thus the new completed-preview preservation on Enter is not the cause of the initial divergence. The already-changed reading is visible immediately after L.

Boundary controls and positive cases

In clean, no-context Enter replays:

Both revisions preserve CMDWL, CmdWL, cmdWLL, cmdWlL, Hello, API, getURL, isRTL, and forceRTL in enko/enkoja/enja. This is a selective gate regression, not a universal capitalization failure.
With Korean disabled (languages: ["en", "ja"]), both revisions preserve all five identifiers in the main table.
The merged code does improve Korean inputs such as akfgkfRK → 말할까 and smfWLr → 늘찍.
The bug: confirming Korean composition changes visible 연습햇 to 연습got #11 reproduction is fixed: dustmqgot keeps its visible 연습햇 on Enter rather than changing it to 연습got. That improvement should be retained.

I recorded 648 parent/merge observations across 18 inputs, three language profiles, three host-prefix fixtures, and Enter/Space. 648 is a coverage count, not 648 successful conversions or independent words. The claims and table above are specifically the verified clean, no-context Enter cases; I am not claiming identical outcomes across every context and confirmation method.

The related engine/Japanese/session suites and focused FFI confirmation test also passed locally (159 test functions in total). Their success does not cover the missing matching identifier negatives.

Mechanism

In HeuristicEngine::decide(), the new exception admits an otherwise protected uppercase vowel when:

Korean is enabled;
a lowercase ASCII character has already occurred;
the immediately preceding key is one of Q/W/E/R/T;
the current key maps to a vowel.

For cmdWL, W followed by L satisfies this pattern after the lowercase prefix. The hard English decision is bypassed, and ordinary model scoring selects Korean. This predicate does not itself distinguish an intentional identifier suffix from delayed Shift release.

Subsequent scoring still includes enabled Japanese candidates. Consequently, an exception justified by Korean composition can also change yukiRI into Japanese. Disabling Korean prevents admission through this gate, even when Japanese remains enabled, consistent with the control results.

The new confirmation policy is a separate concern. The report is about automatic language selection, not evidence that the stored raw keys were destroyed or that explicit cycling/learning overrides stopped working.

Reproducer

In separate checkouts of the parent and merge, build:

cargo build --release -p tacit-ffi --features japanese

Save the script below as repro.py and run it against each resulting library:

# English/Korean: sufficient for cmdWL, rmWL, dmlWL, turnsRI.
python3 repro.py /path/to/target/release/libtacit_ffi.dylib

# Also test Japanese; requires the normal Japanese pack to be available.
python3 repro.py /path/to/target/release/libtacit_ffi.dylib --japanese

The library filename shown is for macOS, where these replays were run. The script gives the engine an explicit temporary settings/learning directory and does not edit an existing user's learning profile. The U+0001 character separates FFI operations: M marks composition and I inserts text.

Standalone Python/ctypes reproducer, checked against both revisions

import ctypes
import json
import pathlib
import sys
import tempfile

lib = ctypes.CDLL(str(pathlib.Path(sys.argv[1]).resolve()))
lib.ts_new.argtypes = [ctypes.c_char_p]
lib.ts_new.restype = ctypes.c_void_p
lib.ts_activate.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
lib.ts_key.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_uint32,
                       ctypes.c_char_p, ctypes.c_int64]
lib.ts_key.restype = ctypes.c_void_p
lib.ts_string_free.argtypes = [ctypes.c_void_p]
lib.ts_free.argtypes = [ctypes.c_void_p]
languages = ['ko', 'en', 'ja'] if '--japanese' in sys.argv[2:] else ['ko', 'en']

for keys in ['cmdWL', 'rmWL', 'dmlWL', 'turnsRI', 'yukiRI',
             'CMDWL', 'Hello', 'akfgkfRK', 'dustmqgot']:
    with tempfile.TemporaryDirectory() as directory:
        folder = pathlib.Path(directory)
        (folder / 'settings.json').write_text(json.dumps({
            'languages': languages, 'liveConversion': False,
            'returnConfirmsOnly': True,
        }), encoding='utf-8')
        (folder / 'lexicon.json').write_text(
            '{"learned":{},"once":{},"shapes":{}}', encoding='utf-8')
        handle = lib.ts_new(directory.encode())
        assert handle
        try:
            lib.ts_activate(handle, b'public-identifier-repro')
            def send(kind, char=0, before=None):
                pointer = lib.ts_key(handle, kind, char, before, 0)
                assert pointer
                try:
                    return ctypes.string_at(pointer).decode('utf-8')
                finally:
                    lib.ts_string_free(pointer)
            trace = [send(0, ord(c), b'' if i == 0 else None)
                     for i, c in enumerate(keys)]
            print(json.dumps({
                'keys': keys, 'last_two_key_replies': trace[-2:],
                'preview': trace[-1], 'enter': send(5),
            }, ensure_ascii=False))
        finally:
            lib.ts_free(handle)

Requested follow-up

Please add identifier controls that actually intersect the overlap exception, alongside the Korean positive cases. Hello/API/all-caps controls alone do not exercise this admission path.

The desired boundary is to retain the Korean improvements and #11 confirmation stability without introducing these unwanted automatic identifier conversions. In particular, Korean overlap evidence should not silently become permission for an unrelated Japanese selection. This calls for reviewing the eligibility/scoring boundary, not adding exceptions for the five strings or lowercasing raw input globally.

If accepting these conversions is an intentional tradeoff, please make that scope explicit and evaluate matching intent-labeled negatives rather than treating the existing English controls as comprehensive protection.

Related investigation: #12 (comment). Remaining Korean false negatives from that investigation are a separate question; this issue is narrowly about new identifier false positives in the merged change.

A clarification about how to interpret the yukiRI example:

This is an ambiguous boundary case, not proof that yukiRI must always remain Latin. The fixture explicitly labels it as an intended case-sensitive identifier. Ignoring case, yukiri is composable as ゆきり; that establishes a valid romaji reading, not the user's intent or that it is necessarily the right word in context. The uppercase RI is evidence worth considering for identifier intent, but is not conclusive. Ordinary hiragana entry does not require that capitalization, yet the character-event replay cannot establish why Shift/capitalization occurred.

The narrower, reproducible observation remains:

Enabled languages	Exact parent	#15 merge
English + Japanese	`yukiRI`	`yukiRI`
English + Korean + Japanese	`yukiRI`	`ゆきり`

The Korean Shift-overlap exception relaxes the capitalization guard, after which Japanese can win the ordinary model competition. Thus this example is particularly useful for reviewing the cross-language reach of a Korean-specific eligibility exception. Adding Korean legitimately changes the candidate set in an automatic multilingual IME, so the profile difference alone does not establish a defect either; the question is whether this particular policy interaction is intended and evaluated.

Please treat yukiRI as a policy/ambiguity control rather than the strongest standalone wrong-conversion example. The measured revision/profile differences and environment-isolation results remain valid; this clarification narrows the interpretation, not the observations. It does not support an unconditional uppercase-to-English rule, a yukiRI exception, or treating synthetically assigned identifier intent as measured real-world usage frequency.

No new runtime experiment is claimed in this comment; it revisits the evidence already reported above.

Confirmed independently on Windows after merging #18 (9c02a6d), using direct HeuristicEngine calls without a learning store. The matching identifier fixtures cmdWL, rmWL, and dmlWL still select Korean in all 12 combinations of enko/enkoja and partial/final scoring. The Korean positive controls akfgkfRK -> 말할까 and smfWLr -> 늘찍 pass in the same profiles and scoring modes.

The earlier Hello/API controls did not exercise the overlap admission path; that was a gap in our previous regression coverage. I have prepared a follow-up branch with a failing regression test for the matching identifier negatives and passing Korean positive controls. No classifier change has been applied yet.

The next change will focus on the capitalization/overlap eligibility boundary, including its cross-language reach, while preserving the confirmation behavior from #15/#18. A higher confidence threshold alone is not sufficient: the incorrect dmlWL decision is already approximately 0.99775 confident. We will keep yukiRI as an ambiguity/policy control, as clarified, rather than treating it as proof of unconditional Latin intent. No per-token exceptions or global lowercasing are planned.

This is synthetic core-level reproduction, not a claim about real-world frequency or installed IME/physical keyboard behavior. Keeping this issue open for the follow-up fix and verification.

Thanks for the detailed reproductions. I reproduced the identifier conversions and implemented a fix using the existing Korean vocabulary and inflection checks, without adding per-token exceptions or new dictionary entries.

The Shift-overlap exception now requires a clean Korean reading supported by that lexical evidence, and it no longer enables Japanese candidates. akfgkfRK → 말할까 remains supported. Unlisted overlap inputs such as smfWLr retain their original case; canonical Korean keys (smfWlr → 늘찍) and explicit Esc correction remain available. Lexical evidence determines eligibility for the exception, rather than proving the user's intent.

The matching identifier regressions, Korean controls, and session/FFI tests pass. The full all-features workspace suite passed with 491 tests, and the existing accuracy metrics remained unchanged.

This deliberately favors preserving case-sensitive identifiers when Korean lexical evidence is missing. Does this boundary address your concern, including the tradeoff for unlisted Korean spellings? I'll leave this issue open for your feedback and any further counterexamples.

상세한 재현과 분석 감사합니다. 식별자가 변환되는 문제를 재현했고, 개별 단어 예외나 새 사전 항목을 추가하지 않고 기존 한국어 어휘 데이터와 활용형 판별을 사용하는 수정을 구현했습니다.

이제 Shift 겹침 예외는 조합된 한글에 미완성 자모가 없고, 기존 어휘 근거가 있을 때만 허용합니다. 이 한국어 예외로 일본어 후보까지 허용하던 경로도 제한했습니다. akfgkfRK → 말할까는 유지됩니다. smfWLr처럼 어휘 근거가 없는 겹침 입력은 원래 대소문자를 보존하며, 정상 한국어 키 입력(smfWlr → 늘찍)이나 명시적인 Esc 교정은 계속 사용할 수 있습니다. 어휘 근거는 예외를 허용하는 기준이지, 사용자의 의도를 확정하는 근거는 아닙니다.

예외 조건에 실제로 해당하는 식별자 회귀 테스트, 한국어 대조군, 세션·FFI 테스트가 통과했습니다. 전체 기능을 켠 워크스페이스 테스트 491개가 통과했고, 기존 정확도 지표도 유지됐습니다.

한국어 어휘 근거가 없을 때는 대소문자를 구분하는 식별자 보존을 우선하는 절충입니다. 사전에 없는 한국어의 자동 변환이 줄어드는 점까지 포함해서, 이 기준이 제기하신 우려를 해소하는지 의견 부탁드립니다. 추가 반례나 의견을 받을 수 있도록 이슈는 열어두겠습니다.

수정 방향과 절충을 명확하게 설명해 주셔서 감사합니다. 공개된 fix/shift-overlap-identifiers 브랜치의 1b322f86d159e866c9b5c8c5f1aee9af22b673ac을 기준 버전 9c02a6d7ab68669cb02e00b971c7e1e10b4d680e과 비교하고 독립 검증했습니다.

#16의 식별자 보호를 위한 보수적인 경계로는 동의합니다. 예외 조건에 실제로 해당하는 식별자 보호, 한국어 예외를 통한 일본어 후보 허용 제한, 명시적인 학습·Esc 교정 유지가 확인됐습니다. yukiRI는 앞서 정리한 대로 모호한 정책 대조군으로 구분했습니다.

검증 범위는 다음과 같습니다.

macOS에서 cargo test --workspace --exclude tacit-tip --all-features: 477개 통과, 기존 ignored helper 2개. Windows 전용 패키지는 제외했습니다.
별도로 작성한 독립 테스트 3개 통과. 식별자 보존 테스트는 정확한 기준 버전에서 실패하고 수정 버전에서 통과했습니다.
임시 학습 파일을 사용해 한국어·영어·일본어의 명시적 학습을 확인했습니다. 실제 InputSession에서 smfWLr를 Esc로 두 번 한국어로 교정하면 이후 입력이 학습된 늘찍으로 표시·확정되는 것도 확인했습니다.
기존 추가 사례 모음으로 버전당 4,940개 관측을 비교했습니다. 빈 HOME·작업 디렉터리·XDG 설정 환경에서도 결과가 동일했습니다. 개인 학습 데이터나 설치된 입력기는 이 비교에 사용하지 않았습니다.

다만 일반적인 Shift 겹침 처리의 범위는 별도로 명시하면 좋겠습니다. 이전에 조사했던 추가 사례 모음을 재사용한 결과, 문맥 없는 최종 판별에서 enko와 enkoja 각각 다음과 같았습니다.

검사	기준 버전	수정 버전
일반 한국어의 정상 키 입력	85/85	85/85
해당 한국어 형태의 Shift 겹침 입력	66/119	33/119
모호한 대조군을 제외한 합성 식별자 원문 보존	18/25	24/25

남은 식별자 대조군 RT → ㄲㅆ는 이번 수정으로 새로 생긴 문제가 아니며, 이 Shift 겹침 예외의 범위 밖입니다.

한국어 쪽에서는 akfgkfRK → 말할까, dlfWLr → 일찍이 유지됐지만, sjfWLrgkek(널찍하다), smfTLsgkrp(늘씬하게)는 기준 버전의 한국어 결과 대신 원문을 보존합니다. 따라서 줄어드는 범위가 임의의 오타뿐 아니라, 기존 어휘·활용형 검사로 뒷받침되지 않는 일반적인 형태도 포함한다는 점이 중요합니다. 정상 키 입력과 명시적 교정은 계속 사용할 수 있었습니다.

이 수치는 관련 형태가 반복되는 수작업 사례 모음의 결과이지, 대표적인 사용 정확도나 새로운 독립 홀드아웃 평가가 아닙니다. 합성 식별자라는 의도 표지도 실제 사용자의 의도·사용 빈도를 증명하지 않습니다. 보고하신 전체 정확도 벤치마크를 별도로 재실행한 것은 아니며, 새 빌드의 설치된 IME·물리 키보드·Windows 런타임 검증도 주장하지 않습니다.

정리하면, 이번 수정을 #16의 보수적인 오변환 방지로 받아들이되, Shift 겹침 전반을 해결한 것으로 설명하지 않고 지원되지 않는 형태의 개선은 별도 과제로 구분하는 것이 적절하다고 생각합니다. 위 범위에서는 추가 차단 사유를 발견하지 못했습니다.

The fix is now available in v0.10.9.

Overlapping Shift input is treated as Korean only when supported by existing vocabulary or inflection evidence; otherwise, case-sensitive Latin identifiers stay as typed. The Korean exception no longer admits Japanese candidates. No token-specific exceptions were added.

I'm keeping this issue open for feedback. Could you try the new version with your original examples and let me know whether the behavior matches your expectations?

수정 사항을 v0.10.9에 배포했습니다.

Shift가 겹친 입력은 기존 한국어 어휘·활용형 근거가 있을 때만 한국어로 처리하고, 그 외에는 라틴 식별자의 대소문자를 그대로 보존합니다. 한국어 예외로 일본어 후보까지 허용하던 경로도 제한했습니다. 개별 문자열 예외를 하드코딩하지는 않았습니다.

피드백을 받을 수 있도록 이슈는 열어 두겠습니다. 새 버전에서 처음 제보해 주신 예시들을 다시 확인해 보시고, 의도하신 동작과 맞는지 알려 주실 수 있을까요?

공식 v0.10.10 일본어 포함 macOS 배포본으로 추가 검증했습니다. 개인 학습 데이터와 분리한 한·영·일 테스트 환경에서 CLI 판별 90건과 실제 입력기에 합성 키 이벤트를 보낸 네이티브 입력 테스트 37건이 모두 통과했습니다. CLI 비교는 enko/enkoja 두 프로필에서, 실제 입력기 테스트는 한·영·일 프로필에서 진행했습니다.

식별자의 대소문자 보존, 말할까·일찍 변환 유지, 어휘 근거가 없는 Shift 겹침 입력의 원문 보존이 설명하신 정책과 일치했습니다. Esc로 두 번 교정한 뒤 해당 한국어 읽기가 학습되는 것도 확인했습니다. 기존 확정 동작 개선도 검증한 사례에서는 유지됐습니다.

따라서 #16의 보수적인 식별자 보호 수정이 의도대로 동작하는 것을 확인했고, 이 범위에서는 이슈를 닫아도 괜찮다고 생각합니다. 앞서 공유한 한국어 Shift 겹침 지원 범위의 한계는 별도 개선 과제로 구분하는 데 동의합니다.

이번 검증은 macOS 네이티브 입력 창과 합성 키 이벤트 기준이며, 브라우저·모든 앱·사람의 실제 타이핑까지 검증했다는 의미는 아닙니다. 테스트 후에는 기존 앱과 입력 소스를 복원했고, 개인 설정·학습 파일의 해시가 유지된 것도 확인했습니다.

수정과 상세한 설명 감사합니다.

Navigation Menu

Repository navigation

bug: Shift-overlap exception converts case-sensitive Latin identifiers to Korean/Japanese #16

Summary

Revisions and conditions

Separating the code change from local learning or IME state

Expected and actual

The change happens before confirmation

Boundary controls and positive cases

Mechanism

Reproducer

Requested follow-up

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bug: Shift-overlap exception converts case-sensitive Latin identifiers to Korean/Japanese #16

Description

Summary

Revisions and conditions

Separating the code change from local learning or IME state

Expected and actual

The change happens before confirmation

Boundary controls and positive cases

Mechanism

Reproducer

Requested follow-up

Activity

snowykr commented on Oct 23, 2026

helloiamnotth3devnicetomeetyou commented on Oct 23, 2026

helloiamnotth3devnicetomeetyou commented on Oct 23, 2026

snowykr commented on Oct 23, 2026

helloiamnotth3devnicetomeetyou commented on Oct 23, 2026

snowykr commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions