Repository navigation
identity: a differing identifier is a contradiction, never a tolerated gap - #1743
Conversation
…d gap Found on Open Dental in the Windows lab: a row band padded with a static label let a same-name sibling with another PatNum and age verify. "15 47" vs "16 50" was a 4-char uncovered run, within UNCOVERED_RUN_CAP, and 2-char tokens were below every contradiction rule. - An unmatched recorded token carrying a digit (id, date, age, count), of any length, is contradicted when the live band shows an unexplained digit-bearing token: a differing value is a different record. - Absent outright (OCR dropout), it marks identity evidence short, so the OCR tier abstains; it is never a tolerated uncovered run. - Same rules in the hashed-template path. - The covered-band repair weighs whole words only: the word the dialog cuts is already checked as a prefix of the recorded word.
…; params keep short ids
Review blockers on this PR:
- a second copy of the recorded value elsewhere in the band ("15 patients
found") explained the recorded id, so a sibling's differing id went
uncharged; a digit-bearing recorded token now matches one observed copy of
its own, on both paths;
- param mode dropped a short id lying inside the example value ("15" in
"20150101"); a digit token is never dropped for being inside the value.
Should-fix: split and join never glue two digit-bearing tokens ("15" "47"
is not id 1547).
|
Independent adversarial review, round 1: blockers. Fixed in 0879718.
Should-fix also done: split and join never glue two digit-bearing tokens, so Each case is a regression test on both paths. A re-review follows once the full identity suite and the validation re-run finish. Noted, not changed:
|
One id split by OCR ("100" "512" vs "100512") and two values beside a
record whose id is 1547 ("15" "47") read alike; such a split or join now
matches but flags the fragments glyph-ambiguous, so the band abstains, as the
9th-reopening split tests require, instead of verifying.
…self Review blocker: the similarity fallback still dropped a PatNum near the example account number (547 vs 15470), so a sibling's differing PatNum contradicted nothing. A digit token now belongs to a parameter only when the value is inside it or it is the value read with OCR confusions (l5470).
Independent adversarial review: Request changesI diffed head Fixes from earlier rounds hold. Each of these refuses or abstains on both paths, and main verified each one:
In the fuzz, no wrong record verified here that main refused. The change is stricter than main on every path I tried. Blockers1. A copy of the recorded value elsewhere in the live band still stands in for the sibling's differing value. This happens when the recorded band did not include that copy. The one-copy rule only helps when the recorded band holds both copies. Every recorded digit token then matches, so the sibling's unexplained id is never charged. Both paths return
Suggested fix: when every recorded digit token matched but the live band has an unexplained digit-bearing token, set 2. The one-copy assignment takes the first canonical-equal live token, not the raw-equal one, so an identical re-read now refuses. The new
In the Should fix
|
…abstains; a 2-letter name replaces a longer one
Independent adversarial review (head
|
| fix | recorded → live | main | this PR (plaintext / hashed) |
|---|---|---|---|
| 1. extra copy of the recorded value | Smith John 15 47 Hint: enter values in multiple boxes. → ... 16 47 ... 15 patients found |
verified | abstain / abstain |
| 1. | Smith John 03/04/1980 Active → Smith John 03/05/1980 Active 03/04/1980 |
verified | abstain / abstain |
| 1. | Smith John 15 Active → Smith John 16 15 Active |
verified | abstain / abstain |
| 2. exact copy first (same record) | Price Bo 80 03/04/1980 → identical |
verified | verified / verified |
| 2. | Smith John 25 47 Active → Smith John 2S 25 47 Active |
verified | abstain / abstain |
| 2. | Lee Li 11 03/04/1980 Active → identical |
abstain | abstain / abstain |
| 3. 3-letter name → 2-letter | Okafor Ann 05/14/1988 Active → Okafor Bo 05/14/1988 Active |
verified | mismatch / mismatch |
The plaintext and hashed paths agreed on every pair and every fuzz case. Nothing crashed, including empty and punctuation-only bands. Nothing verified here that main refused.
Blockers
These are left open rather than introduced: main verifies each one too. Each is in a class this PR says it closes, so the claim does not hold yet.
1. A recorded word that OCR-folds to the record's differing id explains that id, so the extra-digit rule never fires. In the single-token pass, a recorded alphabetic token still marks every canonical-equal live token as explained, including digit tokens (Bo ≡ 80, Io ≡ 10, So ≡ 50). With a copy of the recorded id elsewhere in the live band, every recorded token matches and no live digit token is left unexplained.
| recorded | live (different record) | plaintext / hashed |
|---|---|---|
Price Bo 81 Active |
Price Bo 80 Active 81 found |
verified / verified |
Price Bo 81 03/04/1980 Active |
Price Bo 80 03/04/1980 Active 81 |
verified / verified |
Smith Io 15 Active |
Smith Io 10 Active 15 |
verified / verified |
Suggested fix: a live digit-bearing token counts as explained only by a digit-bearing recorded token (or by a split or join that already abstains), in _match_tokens, _match_tokens_template and both _extra_digit_token checks.
2. The name-replacement rule still misses a 1-letter replacement and the reverse direction (a 1–2 letter recorded name replaced by a longer token that is not name-shaped). Fix 3 counts unexplained 2-letter live tokens for recorded names of 3 or more letters only. A recorded name of fewer than 3 letters goes through the same-length short-token count, and a lowercase live token is not name-shaped. So the replacement is charged only by the absent-name cap, which tolerates it.
| recorded | live (different record) | plaintext / hashed |
|---|---|---|
Okafor Ann 05/14/1988 Active |
Okafor J 05/14/1988 Active |
verified / verified |
Price Al 02/16/1963 76 Active |
Price J. 02/16/1963 76 Active |
verified / verified |
nguyen al 74 03/04/1980 pending |
nguyen joan 74 03/04/1980 pending |
verified / verified |
Okafor A. 05/14/1988 Active already returns mismatch here while Okafor J ... verifies, so the boundary is arbitrary. Suggested fix: while an alphabetic, name-plausible recorded token is unmatched and any unexplained alphabetic live token exists (any length, any case), set identity_evidence_short, so the band abstains. Keep the existing mismatch rules as they are, and make the same change in _band_match_template.
Should fix (fail closed, not blocking)
- An OCR split of a 2-digit token now returns
mismatchwhere main verified. For example,Smith John 15 47→Smith John 1 5 47andSmith John 15 47→Smith John 15 4 7. The join pass skips concatenations shorter thanMIN_BLOCK, so the fragments stay unexplained digit tokens and the new digit rule calls them a different value. Let the join pass try short digit concatenations; the glue rule then makes them abstain. - A truncated read of a 3-letter name now returns
mismatchwhere main verified. For example,Okafor Ann ...→Okafor An .... Abstain is the better result when the unexplained 2-letter token is a prefix of the unmatched recorded token. - A replacement name explained by a copy of a recorded word still verifies. For example,
Okafor Ann 05/14/1988 OK→Okafor OK 05/14/1988 OK. The rule for alphabetic tokens marks every copy, so the replacement is explained. This is contrived and low priority. - False-stop costs to note in the PR body: any extra digit token in the live band now abstains, for example a clock (
... Active 10:42) or a pager (Page 2).
…-shaped tokens need a recorded value
70 runs per build on the Windows 11 x64 laptop, aardwolf from the Mac, rdp_avc_real's protocol: 40 saves, 20 false stops (decoy 10, 1440x900 10), 10 stops on the look-alike, 0 wrong records, on both builds, with identical stop messages. The false stops come from a postcondition on an auto-selected row and a refused session size, not from identity checks.
… RDP) (#1798) Four OpenEnv environments on two Windows line-of-business apps, under the same task, `verify.yaml` and reward contract as the browser environments. The reward is verify-run's verdict on the app's own database, read outside the agent's session. | Environment | App | The agent observes | |---|---|---| | `win-patient-chart` | Patient Chart (WinForms) | Screen and UI Automation elements | | `win-patient-chart-rdp` | Patient Chart (WinForms) | Remote Desktop pixels only (FreeRDP 3, H.264/AVC444) | | `win-claims-desk` | Claims Desk (WPF) | Screen and UI Automation elements | | `win-claims-desk-rdp` | Claims Desk (WPF) | Remote Desktop pixels only (FreeRDP 3, H.264/AVC444) | - **Apps.** Both are MIT-licensed and backed by SQLite. - **Windows.** Not included. The environments run on a Windows machine you supply; `provision/setup_guest.ps1` prepares it. - **Tasks.** 24 public tasks per app: - 12 feasible. They include a first save that is silently lost, misleading banners and reordered lists. - 12 with a write fault the screen doesn't show: drop, wrong record, duplicate, truncate or stale. The correct outcome is to report failure. - Every target has a look-alike record. Task ids are opaque. - **Reset.** One call stops the app, restores the database, checks its digest against the seed, sets the fault and starts the app. An episode that would start from leftover state is refused. ## Run ```bash pip install 'openadapt-flow[openenv]' pywinrm python -m openadapt_flow.rl_env.server environments/openenv/win-patient-chart --port 8000 ``` See `environments/openenv/_windows/README.md` for guest setup, surfaces and validation. ## Results All four environments were measured on Windows 11 Pro x64: | Environment | Solutions earn 1 | Faults refuted | Random | Screen-trusting | State leaks | Reset median | |---|---:|---:|---:|---:|---|---:| | `win-patient-chart` | 24/24 | 12/12 | 0.0 | 0.5 | none | 7.3 s | | `win-claims-desk` | 24/24 | 12/12 | 0.0 | 0.5 | none | 6.6 s | | `win-patient-chart-rdp` | 24/24 | 12/12 | 0.0 | 0.5 | none | 3.3 s | | `win-claims-desk-rdp` | 24/24 | 12/12 | 0.0 | 0.5 | none | 3.2 s | The details and every episode are in `environments/openenv/_windows/results/windows11-x64-2026-11-04/`. That folder also has a 70-run Remote Desktop false-stop comparison before and after #1743 and #1773: 20 of 70 on both builds, with the same causes. ## Also changed - **`win_agent`.** A read-only `/uia/tree` operation, with `WindowsBackend.window_tree` as its client. - **`xfreerdp`.** The client passes `-decorations`, so frames line up with the remote desktop. - **`rl_env.tasks`.** `AppSpec` gains `kind: desktop`. ## Tests `tests/test_rl_env_desktop.py` runs the full contract against a simulated guest. It covers every public task, planted faults, state leaks, the key allowlist, failed resets and attempted writes.
|
Round-five independent review requested at |
|
From: Grok, To: coordinator APPROVE at 43ff310, round five, with one required text fix before merge (see below). Probes: I ran a plaintext-versus-hashed identity probe on main and on this head, checking both the plaintext path and the hashed-template path.
On the false-abort question: the private round-five validation shows a small rise in false aborts on one frozen corpus, while wrong-patient accepts stay at zero on every corpus. Nothing newly verifies in the fuzz, and the look-back finds no flips among committed verified checks. My judgment is that this is an intended and justified fail-closed cost, not a regression. The rise is what closing the round 1–4 wrong-record verifies costs, and every one of those closures is a real wrong-patient accept on main. The new stops are mismatch or abstain refusals, never a silent wrong action. Required text fix: the PR body's validation line says "no change in false aborts". That is no longer accurate after round five. Please reword it, for example: "a small rise in false aborts on one frozen corpus, a justified fail-closed cost; details in the private evidence repo". Should-fix, non-blocking (follow-ups):
Tests: test_identity, test_identity_identifier_contradiction and test_identity_template, on macOS with Python 3.12 and -n 4: 117 passed. Not tested: Windows (the round-five report covers it; its 2 failures also fail on main and are tracked in #1842), and live OCR on the lab VM. |
Wrong-patient fix, found by running OpenAdapt on a real Windows back-office app in the lab.
The hole.
Smith John 15 47 Hint: enter values in multiple boxes., verified against liveSmith John 16 50 Hint: enter values in multiple boxes.. That is a same-name record with a different number and age, and it verified on both the plaintext and the hashed-template paths.15 47counted as a 4-character uncovered run, withinUNCOVERED_RUN_CAP. Its 2-character tokens were below every contradiction rule.Change.
identity._contradictedandidentity_template._contradicted_template: an unmatched recorded token that carries a digit (id, date, age, count), of any length, is contradicted when the live band shows an unexplained digit-bearing token.band_match/_band_match_template: an unmatched digit-bearing recorded token with nothing in its place setsidentity_evidence_short, so the OCR tier abstains. It is never a tolerated uncovered run.repair/self_repair._uncovered_band_matches: the word a dialog cuts (1984-) is already checked as a prefix of the recorded word, so the template now weighs whole words only.Tests.
tests/test_identity_identifier_contradiction.py(20 tests; 12 fail on main). Cases:Validation (details in the private evidence repo):
Draft until the full identity suite and an independent adversarial review are in. One open design question, whether static label text should count toward coverage at all, is separate from this fix.